How to Run a Concept Test

Without Leading Participants or "Validating" Ideas

👋🏻Hi, this is Nikki with a 🔒subscriber-only 🔒 article from User Research Academy. In every article, I cover in-depth topics on how to conduct user research, grow in your career, and fall in love with the craft of user research again.

Really exciting news! I have two six-month mentorship spots open for this winter! This is your chance to have me on-call, working alongside you, in your back pocket for any struggle or question you have, and giving you all my knowledge through personalized feedback on your work. Let’s crush your career goals in 2024 — apply here!

---

Concept testing can sometimes feel like a mystery. There were quite a few times in my career when teams put an idea in front of me and asked me to test it with users (well, to validate or ask for preference, which we don’t do, of course 😁). And for some time, I felt incredibly stuck when my teams made these requests.

The ideas weren’t solid enough to conduct a usability test.

They weren’t basic enough for me to conduct generative research.

These concepts were in the in-between (or the upside-down, if you will 😂).

Whenever a team came to me with these concepts, I cringed. I had no idea how to get them the information they needed without leading the participants and asking biased questions. I knew I shouldn’t be asking things like:

• “Do you like this idea?”

• “Would you use something like this?”

• “How would you make this better?”

• “Is this going in the right direction?”

But I wasn’t sure how else to engage with participants.

So, I tried to conduct usability tests on concepts. That ended as a major failure. The ideas were too early to test. Participants got lost and confused because there wasn’t a flow. And, to be honest, I had no idea what I was trying to test. My goals for those research projects were vague, and the findings were unhelpful.

We understood that people had a hard time navigating a loosely defined concept, but we still had no idea whether or not we were heading in the right direction. Each of those reports ended up disappointing not only my teams but also me.

It was by chance that I heard about the concept of concept testing (very meta, I know). At first, I wasn’t sold. How were we meant to evaluate concepts in an unbiased way? And how were we meant to investigate their reactions without relying too heavily on future-based data?

But, after some more disappointing results and failed usability tests on ideas, I finally decided to give concept testing a whirl. Admittedly, I wasn’t very skilled at conducting those tests, but with some practice and guidance, I finally understood the importance of concept testing. And, from there, I never looked back.

What is Concept Testing, Anyway?

Concept testing is one of those elusive methods that, I believe, we don’t discuss nearly enough, as it can be an extremely powerful tool to use early in the discovery process. Because it can be such an “in-between” method, we often skip it, going straight from generative research to usability testing.

However, concept testing definitely has its place in our process. The way I define it is:

Concept testing is a way to engage with participants to more deeply understand a specific problem and their current process through a stimuli (concept). Through concept testing, we gather feedback that allows us to gauge how aligned we are (or not) with participants’ mental models regarding an idea.

Within the scope of this definition, we are looking for immediate reactions and perceptions from our participants. We are looking to see how participants respond to the idea and where there are gaps or confusion about what we’ve put in front of them.

This is the crux of the definition and often where I can see concept testing go wrong (and where I’ve done it incorrectly before).

Where Concept Testing Goes Wrong

As I mentioned, it took me a good amount of practice to hone my concept testing skills. Because it is a less-discussed methodology, I struggled with finding the proper resources on how to conduct a concept test and what exactly I was looking for as an outcome.

Unfortunately, I see concept testing used a lot for things like:

• Product/idea validation

• Preference testing

• A/B testing

• Asking about future-based behavior

When I first started concept testing, I made these mistakes. I wanted the concept test to tell me whether or not participants liked the concept, if they would use it or not, and if I tested multiple concepts, which one they preferred.

The problem with all of the above is that concept testing is still a qualitative method. And with qualitative methods, we can’t answer these types of questions. Qualitative user research isn’t set up for success to answer “whether or not” or “if” or “preference” questions.

Qualitative research involves uncovering reactions, perceptions, feelings, and mental models. Concept testing should be no different.

When I first started conducting concept tests, I asked many of those questions, and the results were more disappointing than the usability tests I had attempted to run on the concepts.

What did it mean if people liked or disliked a concept? What did preference mean when it came to the concepts? How would we know people would actually use the product in the future? Not only that, but usually, during these tests, there can be social desirability bias present, where participants will tell you what you want to hear.

So, when I delivered my results, my teams weren’t always sure what actions to take. We knew that people liked the concept and which they preferred, but there was so little depth to the answers and so little action within the data that the teams ended up feeling just as stuck as before.

From there, I changed how I thought and approached concept tests, ensuring that I got my teams the data they needed without asking participants questions that could skew our decision-making.

How to Conduct a Concept Test

Realizing that my current way of concept testing wasn’t helping teams, I shifted away from that mindset. I had to be much more intentional with exactly what I was testing, why I was using this approach, and the research outcomes. I also had to clarify what concept testing was and what it wasn’t for myself and my teams.