User Research | Customer Interviews
5 tips to determine when to stop researching and to start prototyping
Teaching and doing Design Thinking and user-centred design for almost 10 years, we have some evergreens of questions that we hear in almost every project. One of those evergreen questions is:
How much user interviews do I need to conduct? When is enough?
This question is an important one as the information from these interviews lay the foundation of any good user-centred project. Garbage in, garbage out.
The problem with that question is: we don’t really have a satisfying answer to that question because it sounds so vague. Our responses are usually along those lines:
“Well, it depends on the project and what you want to find out!”
“Your professional gut feeling will tell you when it’s enough”
“You know it when you get there.”
Although all these answers are essentially true, for a beginner in that field they are far from giving any orientation nor guideline. Well, up until now.
This article will give you the ultimate calculator of how many interviews are needed for any individual project you want to pursue. Unicorn power! Or at least we’ll try. Aim for the stars, right?
Enough with unicorn magic, here are some solid numbers in the pursue of a better answer. Here are the 5 rules of thump on how many interviews you should conduct:
Interview rule #1: Rule of 5
This is actually a scientifically backed number (hear, hear!). Tom Landauer and Jakob Nielsen published a study on usability testing and the probability of finding usability fails when testing with single users. The visual summary of their study is this:
Well, what’s it say?
- “The best results come from testing no more than 5 users and running as many small tests as you can afford.”
- “The most striking truth of the curve is that zero users give zero insights.
- “As soon as you collect data from a single test user, your insights shoot up and you have already learned almost a third of all there is to know about the usability of the design. The difference between zero and even a little bit of data is astounding.”
It basically says: “Get out of the building”. And it says: 5 is enough. But wait, before you stop reading now: This is a usability study, not a Design Thinking study. It’s about finding usability flaws, not user needs. So you can’t really adapt those findings 1:1 to User Research.
We will give it a try anyway, be bold and suggest this rule of thumb:
Go for 5 interviews for each user group or persona.
And just below the rule of thump, the disclaimer and “do not try at home warning”: the rule of 5 is a good estimate to get you started. Once you went through a couple of interviews though, ask yourself these questions to assess whether you should get going or whether you should jump into the next phase:
- How much new information did I learn with the last interviewee? (marginal benefit of interview)
- Do you have a feeling to predict some of the answers of an interviewee because you sense a pattern?
- Do you have the feeling that there is more to the topic but your questions cannot get to it?
Interview rule #2: Assess your marginal benefit of retrieving information
This is a more economic perspective that may guide you to an optimal number of interviews. For that matter, let’s quickly jump back to pole position: what is the purpose of user research?
Right, learning more about the users, understanding their needs, acquiring new information you were missing before the interview. BUT: interviewing is not the only way of acquiring such information, prototyping and testing are other ways that are directly baked into the Design Thinking process.
So the economical question here is: which method is less costly to bring new information and learning to your project? The simple answer is: go for the method with the higher marginal benefit.
The crux of that simple answer is: you cannot really assess the marginal benefit, you probably just have a gut feeling about it. And this is why our answer usually is: trust your gut instinct. Choose whatever method your gut tells you is the easier way to gain new knowledge.
Interview rule #3: Prevent analysis paralysis.
I personally love researching. You become another Sherlock, dive into unknown universes of users and get in touch of parts of society that you didn’t know they even existed. But with every interview you conduct, you collect plenty of information you need to condense at some point. And our brain can only grasp so many information.
Just like your computer’s working memory: once you store too much information there, it gets clocked. The system crashes. Blue screen. Start from scratch. That’s is the downside of being Sherlock (unless you are Sherlock himself!).
So there is a point of too many interviews. It’s determined by your team’s ability to process all the information. So whenever you have the feeling of getting lost and mixing up interviewee insights, that is a good sign of stopping research and start processing it.
From experience, it is better to chunk down the interviews, one chunck at a time, rather than finding out what that exact number of information overflow is for your team 🙂
Interview rule #4: Your boss will never be convinced by qualitative data.
Quite often, there is another, hidden question. It goes along those lines: How many interviews are enough to convince my boss it’s a good idea? The true answer is: too many.
The reality is that your boss most likely works in a system that trusts in quantitative data. Only. Trying to convince with qualitative numbers is a Sisyphean task and will most likely exceed rule #3, i.e. too many interviews to analyze.
The approach we observed to be much easier is to not convince your boss “it’s a great idea” but to convince your boss “it’s a great idea worth testing”. Usually, anecdotes and storytelling from your user research can be convincing enough for the latter.
The difference? It’s much riskier for your boss to go try to convince the rest of the company about that great new idea (also a risk in terms of reputation) with only a handful of data to back this.
For the latter options, it’s only a rather small investment in testing the idea without trying to convince anybody else. How to test then, you may ask? The answer is quite obvious, yet often overlooked: it’s called market research.
Yet, we don’t mean the “big survey” kind of research. There are ways to conduct micro-studies on a quantitative basis that can evaluate your hypothesis without spending too much money.
For us, quantitative and qualitative research needs to go hand in hand to yield great results. And to convince your boss. So don’t bother about the number of interviews for that sake.
The rule of rules #5: Experience.
You may have recognized the pattern of the 4 rules above: it’s experience. With experience, you will fine-tune your gut feeling and get a good grasp around how many interviews are needed and when to move on.
Good judgement comes from experience. And experience? Well that comes from poor judgement.
Well, that’s that, 5 tips on determining when to stop researching. Hopefully all these words were a little bit more concrete than my prefered as-vague-as-possible answer of the old days of “it depends”.
Well, at least this answer has now numbers in it. Numbers sell, they say 🙂
🐎 Yay, you came all the way to the end. If you have the feeling, this was not a complete waste of time — fabulous! Then you might like our newsletter as well (german only so far, truly sorry 😬):