The insights industry exists to help businesses make better decisions: the bigger the decision, the more important it is to get right. Historically, the answer has been a large sample quantitative study with the understanding that more people mean less risk.  

REAL Insight’s journey into the insights industry is unique. Our long history with in-store intercepts gives us a rare appreciation for authentic feedback from real people within the right scope for the research. Because of our inception as an in-context intercept company, we were free from many of the industry norms rampant with methods like surveys and focus groups. As a result, we learned what sample sizes worked for the custom approaches we developed.  

The saying goes that there is comfort in numbers, but numbers don’t necessarily get you to the right business decisions. Leveraging more reliable inputs from authentic participants’ in-context behavior versus stated behavior from quantitative panel participants shouldn’t be viewed as apples to apples when it comes to appropriate sample sizing. 

Sample Size is Not a Magic Number

The right sample size isn’t a magic number. While larger numbers are appropriate for some scenarios, they aren’t entirely necessary to identify trends and determine outliers. Large sample sizes often control for flawed methodologies, data quality, and subpar respondents. Ensuring that participants are authentically the right people and testing in-context decreases the pressure on sample size.  

A couple of years ago, we were brought in to get in-store directional feedback on a new product shortly after launch.  After observing and talking with about 50 shoppers in-store, we left with a long list of first moment concerns including shelf breakthrough, concept understanding and appeal, price/value, and fit with intended usage occasion.  Uncovering issues when testing in-context isn’t unusual for us, but this product had so many concerns that our optimization recommendations looked more like evolutionary recommendations.  This is especially inconvenient when a product has already launched. 

Upon seeing our insights and recommendations, we were told they couldn’t be accurate because they had conducted a 500-person “in-context” test with a well-known company that said the product performed very well at-shelf and should be a smashing success.  We stood by our conviction in our findings, and the truth didn’t take long to emerge.  Within a few months, the product was discounted and discontinued within the first year.   

Break the Habit

Habits are very hard to break.  All of us probably experience this regularly as an obstacle to innovation and positive behavior change for our businesses or the world.  The solution to overcoming this feeling of safety via a clearly defined assurance—such as sample size—won’t come easily.  In researching this topic, we came across an article from 2004 that was banging the same drum, yet the industry demands for quant assurance are as prevalent now as ever. 

At REAL Insight, our willingness to focus on what is necessary for a study comes more easily because our history and methods are novel.  We could invent because we didn’t have the legacy of solutions that require reinvention. At the end of the day, our industry is here to guide companies to make the right decisions with the limited resources available. We can do better.

Join us in challenging the status quo.

1 comment

Leave a Reply

%d bloggers like this: