Interview Sample Size For Win/Loss Analysis

How many win/loss interviews do you need to determine why you lose (and win) deals?

Googling this question, I’ve found a wide range of answers. From “It depends,” to “8 per product and competitor,” to “20-30 interviews per year.”

To my eyes, this is a road that’s already been travelled by Product Managers and UX Researchers. PMs interview customers to find insight into their needs and requirements. And UXRs interview and observe users to uncover problems and insights.

It required some debate, but UX researchers like James Lewis and Robert Virzi ultimately convinced the UXR community that using the binomial probability formula produces the best sample size estimates.

Detecting Uncommon Issues Requires A Larger Sample Of Win/Loss Interviews

Not every buyer or user of your software has the same needs. A more common need or issue will be found in a smaller sample than an uncommon one. Because more buyers have that need, it’s more likely there will be at least one buyer with it in any sample.

Sample size estimates based on the binomial probability formula take this issue of prevalence into account. Rather than a one-size-fits-all answer to the question of win/loss interview sample size, the formula reveals this fundamental truth. Detecting uncommon issues will require a larger sample.

The sample size estimates in the table above show how this cost/benefit tradeoff works.

You’ll need to analyze 9 sales opportunities to have an 85% chance of finding a problem or perception that affects 20% of buyers in lost deals. If you’re willing to limit your Win/Loss Analysis (defined here) to the more common issues that affect more buyers, sample size can be reduced. For example, to identify issues that affect more than 30% of buyers in lost deals, sample size can be reduced to 5.

The second parameter in the binomial model is the chance of observing the behavior. For the table above, I used 85%. You can use the calculator here to get sample size estimates for 95% or 75%, depending on how certain you need to be of finding the issue at least once.

We Recommend A Minimum Of 20 Win/Loss Interviews

In our win/loss analysis service, we begin with a baseline of at least 20 buyer interviews.

We want to have an 85% chance of finding problems and perceptions that affect 20% of buyers in lost deals, as well as the more common issues. By using the binomial probability formula we find our target should be 9 lost deal interviews.

And, we want to have the same level of sensitivity on the won deals, so 9 won deal interviews are required.

Then we use buyer surveys to close the loop on the improvements that have been made, and continuously track buyer perceptions and win/loss data.

We could make 18 interviews our baseline, but rounding up to 20 gives us margin when an interview doesn’t hit the bullseye.

Before you decide that 20+ win/loss interviews is more than you need, consider these two factors:

• • the cost-benefit of missed observations, and
  • how you’ll prioritize improvements to the issues you find.

Consider The Cost-Benefit Tradeoff

First, consider the cost-benefit of missed observations.

Losing a qualified opportunity is expensive—hard costs, soft costs, and opportunity costs. By the time a prospect becomes a Sales Accepted Opportunity (SAO) you’ve invested substantial financial and people resources. So it can be worthwhile to find and fix problems in sales execution that affect a small percentage of opportunities during the sales process.

This makes a good reason to interview a larger sample of twenty or more buyers.

And it’s a good reason to continuously collect win/loss feedback from buyers.

A Larger Interview Sample Will Make It Easier To Prioritize Next Steps

More substantial improvements to product and sales enablement need to be prioritized in a backlog. By generating repeated observations, a larger sample of at least 20 buyers makes it easier to decide which issues should be fixed first. The repeated observations are directional, indicating which issues are more common.

 

One Buying Decision At A Time, Please

The sales wins and losses you select for interviews must be comparable. They must include the same buying decision—the same product, from the same time period.

Mixing together deals for two products is comparing apples and oranges. Each product requires a separate set of interviews.

The same is true for close date. All the selected deals in a sample should come from the same 6-month lookback period because products and competitors evolve much too fast to rely on old data. Make sure you don’t combine into one Win/Loss Analysis several deals from last quarter and a few from three or four quarters ago. This is a fundamental flaw in the subscription model for win/loss buyer interviews.

The deals you analyze should share the same essential attributes such as:

• • product or use case
  • market segment (ie, midmarket vs enterprise)
  • sales model (assisted vs unassisted)
  • new accounts vs expansion sales

It’s common to include opportunities which Closed/Lost at different stages in the sales process, but we will occasionally select deals from just one stage. For example, if we’ve found a dropoff in conversion after a stage that includes a demo or workshop, we’ll screen out deals that were lost at a different stage in the sales process.

The Bottom Line: Interview At Least 20 Buyers

For a once a year, baseline set of win/loss interviews you should interview as many buyers as possible.

With 20 interviews, you will get the repeated observations you need to prioritize resource-intensive improvements to product, sales enablement, and pricing or packaging.

And, with 30 win/loss interviews, you’ll find the “long tail” issues that affect just 12% of buyers, but are still highly valuable because they strongly influence whether an opportunity is won or lost.

Gathering win/loss feedback continuously is even better. As new deals are Closed/Won and Closed/Lost, we survey buyers to increase the probability we’ll find uncommon issues causing losses.

 

Related Resources

Architecting A Win/Loss Program To Meet Your Goals

Win/Loss programs have a lifecycle of their own. The Win/Loss Lifecycle is a useful framework for thinking about your options and deciding what kind of program you need.

The Demand Gen Priority Matrix

We’re fans of the Demand Gen Priority Matrix because it requires no changes to value proposition or any other aspect of a vendor’s offering to quickly improve pipeline coverage and quality.