Free, bonus article! Top questions to my UXCon Vienna presentation
My answers to measuring impact to make your research more strategic
👋 Hey, Nikki here! Welcome to this month’s ✨ free BONUS edition ✨ of User Research Academy’s Newsletter. I tackle reader questions about user research, impact, and accelerating your career. If you’re not a subscriber, here’s what you missed this month:
Episode 83: Membership Spotlight - Giulia Tumminelli on Transitioning from Product to UXR
Work With Me: Building a Research Plan for a Consultancy Client
Hello, Curious Human!
I recently had a talk at 2024 UXCon Vienna about articulating the strategic value of user research. You can download the slides below (please give credit if using/referencing the slides):
Some key takeaways from my talk:
Stakeholder interviews are integral to understanding what they need so that you can pitch impactful research
Two frameworks for writing and presenting your insights strategically
Several formulas for calculating how your research can impact the bottom line (a full article on that coming)
I couldn’t answer all the questions that came up, but the fantastic team sent me the captured questions, so I wanted to give some answers to those who asked great questions that needed more depth than I could reach within the 20-minute talk.
Let’s dive in.
Q: How can I calculate the ROI of the insights obtained from qualitative studies?
Calculating the ROI of qualitative insights may seem tricky because these insights don’t typically come with numerical data attached. However, they can lead to actions that result in measurable outcomes. Here’s how I quantify the value they bring:
1. Link insights to actionable changes
The first step is to identify the business actions directly influenced by the qualitative insights. Did the insights lead to a change in product design, marketing strategy, or customer experience? The more concrete the actions, the easier it will be to measure the impact.
Keep in mind that you might have to wait a little bit to tie your qualitative research to metrics because these things take time, and that is okay.
2. Measure the outcome of these actions
Use traditional business metrics to measure the outcome. For example:
Revenue impact: If the insight led to a product feature improvement, track how this affected customer acquisition, conversion, or retention.
Cost savings: If the insight reduces customer support tickets or operational inefficiencies, calculate the associated cost savings.
3. Use comparative metrics
Compare the “before” and “after” states of a key metric. For instance, if a design change based on qualitative research reduced the drop-off rate on a webpage from 50% to 30%, quantify the additional users retained and project their value in terms of sales or conversions.
B2B example:
Imagine a SaaS company that conducts qualitative interviews with enterprise customers and discovers that a particular dashboard feature is confusing. The company changes the feature, and over the next three months, they see a 15% reduction in churn. By calculating the lifetime value (LTV) of the retained customers and multiplying that by the number of customers saved, they can estimate the financial impact.
B2C example:
A retail company conducts user interviews and discovers that customers are abandoning their carts due to unclear shipping policies. They adjust the website copy and streamline shipping options. In the next quarter, they notice a 10% improvement in conversion rates. The revenue generated from these additional sales can be compared against the cost of conducting the research to determine ROI.
Q: How can we articulate the consequence of insights from exploratory/strategic research when the opportunity can’t yet be connected to KPIs or $$$?
Exploratory research often provides insights that influence long-term strategy, which can be hard to tie immediately to financial metrics. I’ve struggled with this myself. However, there are ways to communicate the strategic value effectively:
1. Link insights to company goals
Start by connecting the insight to broader company goals. For instance, does this insight contribute to innovation, product-market fit, or competitive differentiation? Even if the dollar impact isn’t immediate, you can articulate its role in driving the company toward key strategic objectives.
2. Frame in terms of potential impact
Use hypothetical scenarios. While you may not have data yet, you can model what success would look like if the opportunity were realized. For example, “If we can address this user pain point, it could lead to a 20% increase in adoption of this new product.”
3. Use risk mitigation language
Highlight what’s at stake if the company doesn’t act. Often, exploratory research helps mitigate future risks. Articulating the cost of inaction—such as market share loss or missed opportunities—can be just as compelling as presenting potential gains.
4. Work with others to understand the holistic consequence
One of the best things I’ve ever done is collaborating with other departments to understand, more holistically, what my insights might mean. I’ve gone to marketing, sales, account management, customer support, and other departments to share my findings and to understand how it might impact areas outside the immediate product. Make meetings with these colleagues to discuss how your findings could be relevant to their areas and goals as well.
B2B example:
For a B2B company exploring a new market, research reveals that customers are hesitant to adopt the product due to a lack of clear integration with their existing tools. While you can’t yet quantify the exact revenue lost, framing it as a potential barrier to entry for a $10 million market opportunity can emphasize the importance of addressing the issue.
B2C example:
For a B2C app targeting young users, exploratory research might reveal a growing trend in voice-activated commands. While it may be hard to quantify the financial impact immediately, you could frame it as a strategic investment to stay ahead of competitors who might capitalize on this trend if ignored.
Q: How do you avoid recommending approaches that increase revenue, but hurt your users or stakeholders?
Balancing the need to drive revenue while maintaining user satisfaction is critical, and can sometimes be a tricky thing to balance. Typically we can feel like our hands are tied — I remember a need for introducing a service fee to save the business that could have had an extremely negative impact on users. You can achieve this balance by following a user-centric approach that ensures business goals align with user needs.
1. Prioritize user feedback
Begin by deeply understanding your users’ pain points and motivations through research. Make sure any recommendations for revenue-generating changes align with these insights. And, if they don’t, understand what it is you are “giving up” when it comes to the user’s side.
2. Impact assessment
Before recommending any approach, conduct an impact assessment to weigh the benefits against potential drawbacks for users and stakeholders. Ask: “Will this drive short-term revenue but erode long-term trust?”
3. Create a balanced scorecard
Implement a balanced scorecard approach that includes metrics for both revenue and user satisfaction (ex: UMUX/SUS, churn rate). For every revenue-generating idea, evaluate its impact on these non-monetary metrics to ensure a sustainable approach.
B2B example:
A software company wants to increase revenue by adding premium features to its base product. However, qualitative feedback shows that customers are already frustrated with existing complexity. Rather than introducing more complexity for a revenue boost, the company decides to simplify the core product first and then introduce premium features based on user feedback, maintaining both revenue growth and customer satisfaction.
B2C example:
A streaming service considers increasing subscription prices to drive revenue. However, user surveys reveal dissatisfaction with the current content-to-price ratio. Instead of a blanket price hike, the company introduces flexible pricing tiers, ensuring that users feel they’re getting value at different price points. This approach sustains both revenue and loyalty.
Q: How do you create a safe space for both you and the stakeholders during a stakeholder interview on your work? I imagine it can be uncomfortable.
Creating a safe and productive space during stakeholder interviews is absolutely necessary especially when discussing potentially sensitive or uncomfortable topics. However, I’ve had a hard time in the past with trying to talk to stakeholders that were not only skeptical of my work, but also ones I had previously yelled at (you read that right) in the past.
After some years of holding these interviews, here’s how you can help to establish trust and openness in these interviews:
1. Set expectations early
At the beginning of the interview, set the stage by explaining the purpose of the discussion. Make it clear that the goal is to gather insights that will help everyone, not to assign blame or make anyone uncomfortable. Mention that the session is meant to be collaborative and solution-focused.
2. Use empathetic listening
Demonstrate empathy by actively listening to the stakeholder’s concerns. Reflect on their statements to show you understand their perspective. Avoid interrupting, and allow space for them to express their thoughts freely.
The same techniques that we apply in research sessions with participants are relevant here — make them feel heard and understood and remind them you are there to support them.
3. Ask open-ended, non-confrontational questions
Structure your questions to be open-ended, focusing on the stakeholder’s experiences and pain points rather than probing for specific failures. For example, instead of asking, “Why didn’t this project meet its goals?” try, “What were some of the challenges you faced during the project?”
4. Acknowledge vulnerability
Show vulnerability yourself by acknowledging that some topics may be difficult to discuss. For example, you might say, “I know discussing challenges can sometimes feel uncomfortable, but understanding them helps us work together to improve.”
5. Ensure confidentiality
If needed, assure the stakeholder that the feedback will be handled with care, and sensitive comments will remain confidential. This builds trust and encourages more open communication.
B2B example:
In a B2B setting, imagine you are interviewing a key decision-maker about why user research findings haven’t been fully implemented. You could start by asking, “Can you walk me through your thought process during the last implementation phase?” This framing shifts the focus to understanding the context rather than assigning blame.
Creating a safe space here allows the stakeholder to feel comfortable sharing internal challenges, like budget constraints or team dynamics, that may have impacted the decision-making process.
B2C example:
For a consumer-focused product team, you might interview a stakeholder in the marketing department about the challenges of aligning marketing efforts with product research. A question like, “What can I do to make research findings more actionable for your team?” opens up a constructive conversation. This approach can reveal gaps in communication or resources that you can work to resolve together, creating a sense of collaboration.
Q: We usually conduct qualitative studies. Is it appropriate to convert qualitative findings into quantitative insights (to calculate ROI)?
Yes, qualitative insights can and should be converted into quantitative data to help calculate ROI when necessary. While qualitative research provides depth and understanding, quantifying these insights helps communicate the value in business terms, making it easier to measure impact.
1. Identify themes and patterns:
After conducting qualitative research, identify recurring themes. For instance, if multiple users mention confusion with a specific feature, this insight could form the basis for quantification.
2. Quantify the frequency
Estimate how widespread the issue might be across your entire user base by quantifying how often these themes occur in your qualitative research. You could say, “50% of the users we interviewed expressed confusion about this feature.” This gives you a starting point to project the impact.
3. Estimate the business impact
If the insight suggests that an issue may cause users to abandon a product, you can estimate the revenue lost as a result. For example, if 30% of interviewed users indicated confusion with checkout options and your total user base experiences a similar issue, you can calculate potential lost sales based on conversion rates.
4. Run experiments to evaluate insights
You can also use A/B testing or surveys to turn qualitative insights into quantifiable data. For example, after redesigning a confusing feature based on qualitative feedback, you can run a survey to ask users whether the change improved their experience, and compare this to conversion metrics.
B2B example:
A B2B SaaS company conducts interviews and discovers that several clients find a reporting feature confusing. You estimate that this confusion affects about 20% of clients. If your average client contract is worth $100,000 annually and those 20% are at risk of churn due to dissatisfaction, you could project a potential $2 million revenue risk. This provides a clear ROI incentive to improve the feature based on qualitative insights.
B2C example:
A clothing e-commerce site finds through user interviews that customers often abandon their shopping carts due to unclear return policies. By conducting a survey or looking at session recordings, you can estimate that 10% of users drop off for this reason. If this represents a potential $50,000 per month in lost sales, clarifying the return policy could recover a significant portion of that revenue, providing a measurable ROI.
Q: What tooling/methods do you recommend for these measurable insights? Currently, we don’t have the numbers, making it challenging to give an indication of revenue.
When you don’t have existing data to quantify insights, there are several tools and methods you can use to gather the necessary numbers. These tools will help you turn qualitative insights into measurable outcomes and link them to business metrics such as revenue, customer retention, or cost savings.
*I have no relationship to the tools mentioned, they are simply tools I’ve used in the past.*
1. Set Up analytics tools
Tools like Google Analytics, Mixpanel, or Amplitude allow you to track user behavior, conversions, and drop-off points in real time. By understanding where users are having issues, you can quantify the size of the problem and estimate its impact on revenue.
2. A/B testing platforms
Platforms like Optimizely, VWO, or Google Optimize enable you to test different versions of a feature or product change to see which one performs better. These tests can give you concrete data about how a qualitative insight impacts metrics like conversion rate or retention.
3. Surveys and feedback tools
Tools like SurveyMonkey, Typeform, or Qualtrics help gather customer feedback at scale. You can survey users before and after making a change, asking them to rate their experience and satisfaction. This data can then be used to estimate the business impact of improvements based on qualitative feedback.
4. Heatmaps and session recordings
Tools like Hotjar and Crazy Egg provide heatmaps and session recordings that show how users are interacting with your website or app. This helps quantify how widespread an issue is and links qualitative insights (e.g., “users are confused by the layout”) to hard data (e.g., “30% of users never click past the first step of checkout”).
5. Revenue impact calculators
Many businesses use custom-built calculators to estimate revenue impact from product changes. These calculators typically factor in conversion rates, average order values, and user retention rates. You can input your qualitative findings (e.g., “20% of users mentioned frustration with a feature”) to see how much revenue might be at stake if the issue persists.
Please check all tools with your legal team prior to setting them up or trialing them!
B2B example:
A B2B marketing platform could use Mixpanel to track how users interact with their reporting dashboard. After identifying issues through user interviews, you can track how many users abandon the process before completing a report. You could also set up a targeted survey asking users to rate their satisfaction with the changes you implemented based on qualitative insights, helping you tie these improvements back to revenue or retention.
B2C example:
An e-commerce retailer could use Google Analytics combined with Hotjar to analyze how users navigate the site. If you notice a high drop-off on certain pages, you can introduce qualitative feedback via on-page surveys (“What stopped you from completing your purchase?”). Once you identify the issue, run A/B tests to validate any changes you make and use analytics to measure their direct impact on sales.
Those were all the questions — thanks to everyone who attended the talk and conference — I can’t wait for next year’s lineup!
Which of these questions resonated the most with you? Anything that I missed that you would add? Or anything in particular that you think you might try? Share in the comments!
📚 Additional resources to explore
Stay curious,
Nikki