In my experience its very common for well-meaning clients, colleagues or stakeholders to feel uncertain about taking a design (or user) research approach. They might question whether the results will be statistically significant, whether the sampling is appropriate, or the results will be generalisable to a wider population.

I’ve created this blog post/page as a way to collect my own general thoughts about how to reassure people asking these kinds of questions, or potentially challenge people who are actively trying to undermine the approach. I’m hoping to update and refine it over time.

There is of course lots of other resources that cover similar topics, I’ve tried to link out to the most helpful ones.

Last updated 2024/06/14

In this post, I refer to ‘design research’ as a broad title that includes various methods that may also be referred to as user research, qualitative research, in-depth interviews, usability testing, prototype testing etc.

In summary design research

• Is a qualitative approach, not a quantitative one
• Iteratively tests designs with small sample sizes, learning and making changes with each round of research
• Pro-actively seeks out ‘extreme’ users to identify the issues they might face with a product or service, recognising that this often improves it for all
• Is about finding issues and resolving them, not finding consensus or irrefutable truth before taking action

Qual vs Quant

Design research is a qualitative approach, not a quantitative one. This means it prioritises depth of research rather than breadth. Whilst there are different kinds of design research (generative interviews vs usability testing of interfaces) they intentionally take the same approach of small sample sizes (often 5-8 each round). This is because the research is ‘formative’ in nature – that is it about learning enough to take the next best action. 

Small sample sizes are not only a benefit for speed, it also significantly de-risks the research approach. In the same way that you should test a survey with two or three people before sending to 10,000 in case you identify any unexpected issues and undermine all data collection. When doing qualitative research, the ability to test your recruitment, screening and research on a small group initially prevents you from making mistakes at scale, whilst also dealing with the inherent logistical difficulties of doing research at scale at the same time. 

Once a design team has iterated multiple times and a higher level of confidence is needed in the usability or suitability of a design, it is common to attempt a ‘summative’ research stage with a larger sample size. However, even in the summative stage the numbers still have to be achieved through a multi-stage process, considering different user types and geographies (where relevant) because of the in depth nature of qualitative studies.

For a more detailed explainer, see the article from Nielsen norman group (the publishers of the original 5-8 user study) on different approaches to qual and quant design research. 

Whilst this is described as ‘design research’ the methodology is based on, and has been developed alongside academic approaches such as grounded theory and action research. These approaches are described as ‘Pragmatist’ – which is recognised as an especially valid approach for e.g. patient-orientated research in healthcare because a pragmatist research approach prioritises ‘what works’ rather than finding statistically significant certainty.  

This is also the approach to medical device development recommended by ISO62366 and HE75 in Europe and USA respectively. Design research approaches can often be perceived as ‘quick and dirty’ but the wide acceptance of them academically and in regulated industries demonstrate that its a valid approach to ‘knowing things’ and de-risking product or service development.

Other helpful links:
https://dscout.com/people-nerds/statistical-significance-uxr-qual-quant