How To Run a Qualitative Usability Test

Asking Questions that Get You Good Data

👋🏻Hi, this is Nikki with a 🔒subscriber-only 🔒 article from User Research Academy. In every article, I cover in-depth topics on how to conduct user research, grow in your career, and fall in love with the craft of user research again.

---

For me, quantitative usability testing was always super straightforward. I put a high-fidelity design or live product in front of someone and asked them to do certain tasks, which I then measured through metrics like task success, time on task, and surveys like the Single Ease Questionnaire.

There was very little room for asking qualitative-based questions or for introducing bias. We were there to truly understand the effectiveness, efficiency, and satisfaction of what we put in front of the participants. Straightforward. Easy, dare I say. In fact, I could even set up an unmoderated test to get even more participants.

However, I felt uncomfortable when it came to qualitative usability testing. I never seemed to be able to strike the right balance and constantly felt like I was asking leading and biased questions. I hated the standard questions like:

• “What would you expect to see?”

• “What do you think of this?”

• “What would you change?”

• “Explore the interface and tell me what you would do.”

I hated those questions because they were so hypothetical and future-based. I felt like I was asking the participant to develop ideas and design the website or app. The data I got from those questions was skewed and unhelpful.

Very rarely, if ever, as a user, do I sit on a website and think, “What am I expecting to see?” I can’t remember the last time I went to a website or app to explore the interface. And, although sometimes I do have opinions on websites/apps, my opinions likely wouldn’t be helpful or actionable to teams trying to make changes.

“This is dumb” is not a very actionable quote.

So, how do you make qualitative usability testing useful and ensure you are asking the best questions for the best data? Let’s dive in.

What is Qualitative Usability Testing?

Before we get into forming the best questions, it’s important to define what qualitative usability testing is and to explore why it’s different from approaches like quantitative usability testing or concept testing.

Having the word usability in the approach is what makes things confusing. When we are usability testing, we are testing the usability of a product/service. And, to me, this makes total sense when looking at quantitative usability testing because you are directly measuring and observing the product's usability via effectiveness, efficiency, and satisfaction.

However, the usability part becomes difficult to understand regarding qualitative usability testing. With qualitative research, can we really test the usability of a product/service?

To me, qualitative usability testing isn’t actually about testing the usability of a product/service but a state in between concept testing and usability testing in which you are trying to get feedback on the direction of designs.

Versus Quantitative Usability Testing

Quantitative usability testing is all about observing and measuring a participant’s experience when interacting with the product in as much of a “real-world” context as possible. With this approach, you aren’t chatting to the participants and asking them what they think about the experience or designs.

Instead, you are one hundred percent focused on measuring the effectiveness, efficiency, and satisfaction of the product/service. You have the pre-determined tasks you need participants to complete, ask them to do so, and then move to the next scenario.

Ideally, for quantitative usability testing, your product/service is either a VERY high-fidelity prototype that allows people to make mistakes or a live product.

Versus Concept Testing

Concept testing is a way to engage with participants to understand a specific problem and their current process more deeply through a stimulus (concept). Through concept testing, we gather feedback that allows us to gauge how aligned we are (or not) with participants’ mental models regarding an idea.

You use a very low-fidelity sketch/prototype to help you stimulate the conversation with the participants to dive further into what and how they think about a certain problem. With this approach, you are still trying to understand the problem space better rather than looking at solutions.

Where Does this Leave Us?

Now that we’ve defined qualitative usability testing as neither quantitative nor concept testing, where does that leave us with the definition?

To be honest, I wish we could rename qualitative usability testing to something like a “qualitative feedback session.” The reason is that qualitative usability testing almost gives the wrong impression and is a misnomer for what we really get out of it. We can’t qualitatively test the usability, but we can gather feedback on people's experience.

So, to me, qualitative usability testing is about gathering qualitative feedback on the current mid-fidelity experience, not necessarily looking to measure the usability but rather understanding how someone perceives the product/service and their experience with it.

In a way, it is more like experience feedback sessions than usability testing — maybe we should start calling it that. 😁

Common Qualitative Usability Testing Goals

As you may know, I am obsessed with starting every project or approaching every method by understanding the goals and writing a research plan. By knowing your goals, you can make a much more informed decision on who you need to recruit and the best methodology for your study.

Every project and organization can be different, but there are certain models you can use for goals to understand if certain methods are better suited to your study than others. These goals are points, ideas, or areas you are trying to learn more about in your study. They set your study up for success and ensure you get the information you need to help your team make better decisions.

You can ask yourself the following questions when defining study goals:

• What is my team trying to understand better?

• What information does my team need to move forward?

• What decisions are we trying to make?

• What are we trying to learn by the end of the study?

Since not every study is exactly the same, your answers may vary, but here are a few examples that indicate a qualitative usability test is the right methodology for your study: