👋🏻Hi, this is Nikki with a 🔒subscriber-only 🔒 article from User Research Academy. In every article, I cover in-depth topics on how to conduct user research, grow in your career, and fall in love with the craft of user research again.
Surveys can feel like a simple methodology — write a few questions, send the survey to a bunch of people, and, voila, we have insights. If surveys were that easy, we’d all be swimming in deep and rich insights, and, likely, surveys would be the only method we need.
But that isn’t necessarily the case because surveys, while a great research method, can be difficult to assemble to get appropriate information. There have been many times when I’ve used a survey when I should have used another method, such as 1x1 interviews, diary studies, or looking into quantitative data.
I remember desperately trying to ask five open-ended questions in a survey to get qualitative information. That was back when I thought surveys could solve pretty much anything. I got my heart broken several times over trying to use surveys incorrectly (and for everything).
Now, I am very intentional and thoughtful when I choose surveys as a method
What are User Research Surveys (and How are they Different)
Imagine you are throwing a party for your friends, and you want to understand what your friends need to make the party enjoyable for them. You might need to understand the types of food to get for the party, the kind of music people listen to, the games they play, and a bit about different parties they’ve enjoyed in the past and what made them enjoyable. You are throwing out feelers to make the night successful. To make it a good experience for your guests.
A user research survey is similar. You are asking questions to understand, at scale, the pain points, habits, behaviors, and needs of your users. With a survey, you are looking to broaden your understanding of an audience, specifically around what they are doing and feeling.
Because of this “what” caveat, a great way to use surveys is to supplement with other “why” methodologies, such as 1x1 interviews. For example, surveys are fantastic at telling you, on a larger scale, what people are doing or what tools people are using, but they don’t go a great job telling you why a person is acting in a certain way or using a particular tool. Or how the experience is for someone beyond a rating.
it is super important to keep this in mind as you decide whether or not surveys are the right approach for your study.
How are they Different?
There are a lot of different surveys out there, so it can sometimes be confusing to delineate which one is best for what you need. Here are ways you can differentiate them:
Market Research Surveys: These surveys look at the big picture of your market or audience. They tend to have a much broader question or scope, such as looking at the entire audience to understand what type of party to throw, rather than getting into the nitty-gritty of people's experience.
Customer Satisfaction Surveys: These are quick check-ins post-purchase or interaction, seeing how the experience was for the user, typically surrounding their satisfaction. These focus a lot on post-experience, so people can tell you what actually went wrong and recall from a particular usage. This would be like asking people how satisfied or dissatisfied they were with the snacks at your party.
Feedback Forms: These are even quicker, often just a few questions or a rating scale about a specific concept or thing, like the new dip you tried out at your party. They're super focused and don't dive deep into the whole experience but rather look at a small slice or moment.
Where can they Fail?
Many companies rely solely on survey data to make data-driven decisions. They include comment boxes with the hope users will leave a message explaining their survey response— but most users don’t.
The truth is surveys don’t drive better decisions on their own. Too many companies sometimes use surveys to convince themselves they are customer-centric. But when you only use one method of user research or use only quantitative or qualitative data, you are missing other pieces of the puzzle that give you a more robust and empathetic understanding of your users.
It is super important to weigh surveys' pros and cons, such as:
Lack of Depth: Surveys may not provide the depth of insights that can be gained from qualitative methods like interviews or ethnographic studies. The predefined answers can limit the range of responses, potentially missing nuanced feedback.
Low Response Rates: Depending on the survey's design and distribution method, achieving a high response rate can be challenging, leading to sample bias if only certain users are more likely to respond.
Misinterpretation of Questions: Users might misunderstand survey questions, leading to inaccurate responses. The lack of interaction with respondents means there's no opportunity to clarify questions or probe deeper as you can in interviews.
Over-reliance on Self-reported Data: Surveys rely on users' perceptions and self-reported data, which can be biased or inaccurate due to memory recall issues or the desire to present oneself in a favorable light.
Surveys can be a really great tool when used properly. Let’s dive into how to know a survey is the right fit and, once you determined that, how to write an effective survey.
Where to Start with Your User Research Survey
Starting with Understanding
I often see a problem when people try to create a survey, and they don’t know the meaningful answers to put in as choices for participants. For example, when working at a hospitality start-up, I sent a survey to understand people’s top pain points with our software.
The issue? I had no idea what to list as the multiple-choice or ranking options. For half of those answers, I guessed what might be the pain points they were experiencing. This is obviously not ideal because, when it comes to surveys, people might just choose an easy answer that isn’t a one hundred percent fit or the one that is closest to what they are experiencing rather than the actual experience they are having.
What are Your Goals?
Defining and setting your goals (like with every other project) are so incredibly important for the success of your survey. They give you a super clear focus on what exact information you need from your survey and can help you define whether or not a survey is the right fit for your study. These goals will also help you develop questions that get you the information you need by the end of the study.
Creating goals is always the first step that I take because it honestly sets the entire project up for success, no matter what approach, method, or problem you are trying to solve.
I usually define goals for surveys into three distinct buckets:
To look at usability
To better understand your user base
To prioritize information
However, this is only the surface of creating goals and, sometimes, they can be tricky to create, especially if you aren’t used to writing them. So let’s go step-by-step into how you can determine your survey goals.
Define the overarching why. The best way to start this process is to understand what you are trying to achieve through the survey. This will help you determine if a survey is, in fact, the right method to use. In this step, you can ask yourself:
What are you trying to achieve with this survey?
How will this survey help you/the team?
What do you expect from the results (assumptions)?
What do you expect to do with the results?
How does this survey fit into any business/organizational goals?
Specify your needs. Next, defining what exactly you are trying to get from your survey is critical. I recommend drilling down on these questions to know whether a survey is the right approach for what you are trying to get. These questions will also help you develop screener questions.
What are you trying to learn about your users?
What information are you trying to get from your users? Does a survey make sense, given that information?
What kind of information do your users have to be able to give you?
What gaps does the information from this survey aim to fill?
List the criteria of your ideal participant. Once you better understand your needs and if a survey is the right fit, you can start thinking about who you want to recruit. I tend to do this before finalizing survey goals because it gives me another opportunity to ensure a survey is the best way to get the information I need from users. Read an in-depth guide on screener surveys here!
What are the questions your users have to answer to get you meaningful information?
What gaps in knowledge do you have that you need your participants to fill in?
What behaviors do you need to understand more?
What habits are you trying to target?
Write your goals. Now that you’ve gathered all this information, it is time to write your goals for the survey. As I mentioned above, I use certain models for writing survey goals that go into one of those three buckets:
Usability goals. These are all about understanding the usability of a product or service and often include UX metrics such as the SEQ, SUS, and UMUX-lite. You can read more about this process here! Here are some example goals:
Evaluate people’s perception of the ease of use of [product/service]
Identify the task- or product-based satisfaction of [product/service]
Understanding goals. These goals focus on understanding different behaviors, actions, tasks, or tools people use and the frequency of that information.
Identify behaviors and actions people are taking on [product/service]
Uncover the different tools participants are currently using to achieve their goals and what their experience is like with them
Uncover people’s perceptions/feelings regarding certain actions/tasks on [product/service]
Prioritization goals. Whenever you have a large amount of qualitative data, it is important to prioritize the data to allow your team to take action. Prioritization surveys, such as the opportunity gap survey, are perfect for this situation.
Prioritize qualitative insights based on the current level of importance and satisfaction
Understand the most impactful unmet needs/pain points to solve for
I highly recommend working with stakeholders on defining your goals to ensure everyone is aligned on the expected outcomes and information from the study!
Formulating Effective Survey Questions
Keep reading with a 7-day free trial
Subscribe to User Research Academy to keep reading this post and get 7 days of free access to the full post archives.