Seven steps to writing a screener survey
How to make sure you are always getting the best participants.
How to make sure you are always getting the best participants.
Screener surveys are an excellent tool for the beginning of each research study. Before you jump into recruiting, you can send out this survey to assess potential participant’s backgrounds, behaviors, habits, and other essential characteristics.
There are a few reasons we write screener surveys:
Find and talk to the most relevant participants for your research study
They help you guarantee you are getting high-value participants
Ensure return on investment for your research will be as high as possible
Avoid wasting time/money on suboptimal participants
Avoid wasting the time of others (ex: users who might not be the best fit)
Dodge awkward research sessions where the participant cannot meaningfully answer the questions you are asking
I learned about the importance of screener surveys the hard way: through first-hand awkward experiences with the wrong participants for my study. I was researching the process that occurs when people move to a new home or apartment. I wrote my screener survey, which was full of demographic information, and asking a few questions about whether they had moved recently.
We received a good number of responses quickly, and I was happy. They were matching the demographic data we wanted from our target participants. Nothing ticked in my head, saying, “how are we getting so many participants so quickly?” Not to spoil the plot, but it had to do with the screener survey. It was too focused on demographic data and not focused enough on behavior.
So, what happened? I had participants come in, and I asked them my default starting question, “tell me about the last time you moved.” They gave me an overview, and I set to dig into the details:
Me: “Okay, great, now walk me through the step-by-step process you went through.”
Participant: “Oh, I wasn’t involved in the process. My partner/real estate agent took care of that. You should probably talk to them.”
Me (internally): Sh*t.
Some of the people had lied on the screener survey. They responded saying they had moved in the past year. However, some had moved a few years ago. But they knew what I was looking for and the correct answer to participate for the incentive.
I had recruited the wrong participants because of my screener survey. Not all the participants were a bad fit, but an overwhelming majority ended up not being super helpful. I had to redo my screener survey and start recruiting again. That is when I learned, screener surveys:
Are not all about demographics
Should focus on behavior and habits
Need to elicit specific information
Can’t give away which answer is correct
After that experience, I took my screener surveys much more seriously and learned how to write them correctly.
Want a free 18-minute training on how to recruit the right participants? Sign-up here!
However, they can be tricky to write
Like most things in user research, we have to consider how we construct screener surveys seriously. Certain questions can backfire on us. You have to achieve a balance within a screener survey:
Get the right information to qualify the best participants
Make sure you are not asking obvious questions that would lead people to exaggerate their responses to participate
After a few years, I have done my best to achieve this balance. I always use the following steps before writing and sending out a screener survey.
The seven steps to writing an excellent screener survey
1. List the criteria of your ideal participants. Before you even think about recruiting, it is beneficial to list out all of the different criteria an exemplary participant would have. And think about the information you need from them. For instance, if I had genuinely thought about needing information on the step-by-step moving process, I would have (hopefully) included questions about whether or not they took part in this. It is beneficial to consider:
Who are your target personas or proto-personas
What behaviors do you want to understand more?
What habits are you trying to target?
What are the goals the user is trying to accomplish?
2. Write one question for each criterion you identified
For each ideal standard/behavior, write one screener question
For example, if it is essential someone has wanted to move for 60 days; write a question about this behavior
Make sure to write precise questions that ask for particular behavior patterns or timeframes:
Imprecise criteria: People who have wanted to move for 60 days
Precise criteria: People who have visited 3+ apartments in the past 60 days
3. Focus on how potential participants feel and HAVE behaved
Focus on their past behavior, as it is the best predictor of future behavior
Future question: Will you look at apartments in the next 60 days?
Past question: Have you looked at 3+ apartments in the past 60 days?
Ask for only the most basic demographic information that you need, since it makes the survey longer
4. Order your questions carefully (use logic!)
Ask questions that will quickly weed people out in the beginning
Prioritize the essential criteria you need the participant to align with before you ask specific questions
For example: Make sure you ask IF someone has thought about moving before asking them how often they have viewed apartments
Ask about location first if you need in-person interviews
Use a funnel approach
For example: Start with more broad questions, move into specifics, and then broaden out again with questions like demographic data
5. Avoid leading questions
Leading questions will influence people to answer in particular ways, and skew your data
Leading: How great was our coffee?
Fixed: How would you describe our coffee?
6. Include a mix of open-ended and closed questions to avoid obvious answers
Using a combination of open and closed questions helps to make sure you are getting the best participants. A 60/40 mix is safe, where 60% of questions are closed, and 40% are open-ended questions.
Open-ended questions help you see how a participant would answer in their own words, without priming them to respond in a specific way
Using closed questions in a non-obvious way can help you understand behavior patterns (ex: asking how many times someone viewed apartments in 3 months, and giving a range of answers)
Adding an open-ended question to your screener survey can give you clues as to how much insight a participant will provide in your actual study
One word responses, illogical rambling or cagey answers can all indicate that a participant may provide a low return on investment
7. Always include an open text field
Always make sure you include an open-text area in each multiple-choice question, as it is impossible for you to know you include all options someone might need
Ex: “other,” “not applicable,” “I don’t know,” or “none of the above”
Let’s go through an example:
We own a plant shop in Brooklyn, New York, where we sell groups of exotic plants. We have been posting our plants on social media, and people from around the United States are contacting us and asking if we have an online shop. We want to consider this as an option but aren’t sure what people would want or expect. We want to conduct user research to understand the following:
How people would like to purchase plants online
What do people expect of an online plant store
What do they think of other online plant stores
Get their opinion on a prototype of the online plant store
Our screener survey might include questions like:
Have you purchased plants online? (Yes/No)
If yes:
How often have you bought a plant online in the past six months? (Multiple choice)
How many plants have you purchased online in the past six months? (Multiple choice)
Where did you purchase from? (Short open text)
How was your experience with buying plants online? (Long open text)
If no:
Have you considered buying a plant online? (Yes/No)
If yes, why have you not purchased one online yet? (Open text)
If no, why have you never thought about it? (Open text)
Feel free to add more thoughts on how you would screen!
Interested in learning more?
Thanks for reading! If you enjoyed this:
Want to learn the foundational skills of being a user researcher? Check out my intro to user research course
Want to understand how an end-to-end project works and practice with data? Purchase my Execution to Planning course
Transitioning into the field? Check out this video with a bunch of helpful links in the description
Connect with me on LinkedIn, I post every weekday at 8:30am GMT/BST