Written by Watching video footage of panel-recruited respondents “shopping” in a recent conference presentation reminded me of the unfortunate truth that for all the benefits of operational efficiency, there is a significant cost of respondent re-use: data validity. I could tell the behavior shown wasn’t “real”, even though it was captured in a real store.
A few years ago, we were brought in for some post-launch in-store qualitative learning on a huge project launch for a major brand. We spent 4 days in-store shortly after launch, talked with about 60 shoppers, and left with major red-flags across each aspect of our Win At-Shelf Framework™. Shelf breakthrough was a problem, consumers didn’t understand the proposition, and therefore, it wasn’t desirable. Not a fun message to share back at that point in the process, but we shared it anyway. Our client immediately pushed back because they had done a N=400 shopper lab study prior to launch that provided the literal opposite findings and recommendations. It had projected Year 1 sales volume to be millions of units. To get to the bottom of which approach was right, we dug deeper into the respective methodologies. What we noticed with the lab research was there were no restrictions on frequency of research participation and that about half of the sample noticed this product as new while shopping. No new product will achieve anything close to 50% unaided recall, and this product’s category especially, made this highly, highly improbable. What likely happened is that repeat respondents were trained through participating regularly in this type of research to look for the new product. Gamification turned a normal shopping mindset into a completely abnormal one for the category. Ultimately, the innovation failed quickly. They couldn’t pivot fast enough. Our research was accurate and we only wished we had been brought in prior to launch when there would have still been time to help.
As our industry continues to look to use technology to scale up qualitative and behavioral research, the fundamental watch-out is who is participating in the research. For a technology company, getting a reliable panel of respondents who participate regularly in the process and are trained to complete activities smoothly resolves so much potential friction. However, it creates massive problems for the clients that hire them because the chance of that data being unreliable, even if captured “in-context,” is extremely high. Gamification and respondent training produces inaccurate behavioral data.
An estimated 95% of all REAL Insight respondents haven’t participated in shopper behavioral research before engaging with us. This ratio of real shoppers can only be found in the wild, and it makes ALL the difference. If you are struggling to understand why things didn’t work out the way you had expected, even after doing in-context research, this is why. We can help.