In a previous article that you can read here, I wrote about the rationale, and guidelines, for choosing an appropriate sample size in usability testing. However, usability testing, or evaluative research, is not the only time you might be asked to defend, or provide a rationale for, sample size in design research.
As design researchers, we are routinely conducting qualitative research that is generative, or exploratory, in nature. When using methods such as mobile ethnography, workshops or contextual enquiry and making decisions about sample size, the rationale(s) that underpins recommendations for number of participants for usability testing does not automatically apply.
As with usability testing, in generative research we are not aiming for a representative sample. Our goal is typically to use this research to inform and describe an experience, which in turn provides the insight needed to make good design decisions. Importantly, the goal is not to predict uptake of a product or service, but rather what it needs to deliver in order to meet expectations. Consequently, our approach to recruitment is based on behaviours, attitudes and goals, rather than simple demographics.
However, as this type of research informs the early stages of product or service design and development, stakeholders can become even more nervous about sample size. So, what can we say to reduce their concerns and make them feel more comfortable about the sample size?
Some theories to consider
A good starting place is a consideration of grounded theory and theoretical sampling, concepts associated with the social sciences that represent research methodology and data collection approaches that inform theory. I won’t go into detail on these concepts, but their relevance to this discussion is that they provide theoretical underpinnings to the way in which we approach generative design research.
Theoretical sampling describes a process of iterative data collection, where a theory emerges and is refined, with each round of data collection and analysis providing the researcher with direction on what questions to explore next and who they should recruit to deliver the required insight. This process continues until “data saturation” is achieved, meaning no new information can be obtained.
So how does this relate to design research and sample size? With generative research we are typically starting with a question that needs answering but an open mind as to the answer(s). We recruit the participants that we think will provide the best insights, rather than those who are representative of an entire audience. We then iterate, further refining our questions and recruiting accordingly, until we reach a conclusion and feel confident that the right thing is being built.
So, how many participants?
But what about sample size? Well in grounded theory, and theoretical sampling, no maximum or minimum sample size is recommended or prescribed. In reality, there is no definitive formula or method we can employ to determine sample size for qualitative research. So, in the absence of a hard rule or equation, we instead consider factors such as time available, budget, research knowledge and expertise, nature and availability of participants and the scope of the question(s) we are asking.
Although I mentioned above that there is no definitive formula for determining sample size, that does not mean attempts have not been made to create one. Victor Yocco has provided a thoughtful argument and rationale for a potential formula that can be read here. Victor’s article provides an excellent starting point for someone who is starting out as a design researcher and does not work with, or have access to, more experienced researchers. However, in general, I prefer to avoid the temptation to quantify the qualitative and would pay more attention to the variables that are used as inputs for the equation rather than the equation itself.
So, where does all of that leave us in our goal to manage stakeholder concerns regarding sample sizes in generative design research? We can point to the well-established theoretical foundations that underpin the way that we approach design research. We can also argue with confidence that this type of research is about representation, rather than representativeness. Our goal is to describe and inform rather than predict, so thoughts about representative samples do not apply.
At U1 we always suggest a minimum of twelve participants across two workshops, or alternatively eight depth interviews as a starting point for any generative research project. Ultimately, in the absence of infinite budget and no timetable for delivery, we’ll draw on our knowledge of good research design, our experience as researchers, the nature of the questions being asked, the characteristics of participants we need to recruit and of course the available time and budget. Using these parameters we’ll plot the best possible course to deliver research and insights that deliver the best bang for buck in informing design decisions.