Back in January, I wrote about the starting point of any concept (i.e., the core idea) and the importance of quantitatively vetting ideas prior to concept development. As a quick review, the power of quantitative idea screening is that it allows you to learn which ones are the best, so you can focus your development efforts on them. In short, quantitative idea screening allows you to test fewer, better concepts – saving time and money while increasing success rates.
- It increases your efficiency: You will spend less time and money developing and testing concepts if you quantitatively vet your new product ideas prior to concept development.
- And it increases your effectiveness: You won’t leave good ideas behind and you’ll move forward with the ideas that have the best chance of succeeding as concepts. This means higher pass-through rates and, ultimately, a better – more successful – innovation process.
In this article, I’ll share an actual case study that clearly demonstrates how idea screening saves time and money. Perhaps more importantly, I’ll also share some data on the validation of idea screening – how idea screening scores correlate with later concept testing scores – proving that idea screening is an essential part of an effective innovation process.
Saving Time and Money
The client was spending about a quarter of a million dollars with us to test about 60 concepts across three waves per year (20 per wave). Including graphics and text and editing time, each of these concepts took about 10 hours to create for a total of 600 hours spent. Once the concepts were created, it took about 4 weeks to field, analyze and deliver results – across 3 waves, this adds up to 12 weeks total time for testing over a year. Here’s the breakdown:
On average, this client had a 20% pass-through rate for their concepts. This is a pretty typical rate for systems that do not vet their ideas prior to concept development. So at the end of one year, the old school approach yielded 12 validated concepts with the above costs.
Now idea statements are a lot easier to write than full concepts, each taking only about 10 minutes each. The cost to test ideas is significantly less than concepts and, of course, our overnight platform makes timing much faster. Here’s the breakdown in just three months of testing.
What’s great about this is that each of the 5 waves of idea screening allowed our client to iterate on previously tested ideas as well as testing new ones, leveraging the feedback from the KPIs as well as Idea Evaluator (our highlighter tool) to optimize and retest the ideas. This ability to quickly iterate ultimately yielded powerful ideas that were ready to succeed in concept testing. Because of the iteration and optimization of the ideas, the entire system becomes much more efficient – with dramatically higher pass through rates at the concept phase. Moving into the concept testing phase, only 20 concepts were tested as the idea testing identified these as having the highest potential out of over 100 novel ideas tested.—and 12 out of 20 concepts passed (a 60% pass through rate).
Here’s the final comparison, incorporating all costs and timing:
|Without Idea Testing||With Overnight Idea Testing|
|Pass Through Rate||20%||60%|
|Time (stimuli development)||600 hours||225 hours|
|Time (testing)||12 weeks||5 weeks|
|Total Development Time||1 year||3 months|
|Cost to test Ideas||$0||$85,000|
|Cost to test Concepts||$300,000||$80,000|
The cost savings, in terms of both time and money are hugely significant – and yielded much better results. In a world where budgets are tighter and speed to market may be the only remaining competitive advantage, idea testing delivered in a big way.
Validation – correlations and pass through rate.
Clients have often asked for proof that idea screening is valid – that it can be related to consumer reaction once those ideas are turned into concepts. I’ll admit that I have often taken this for granted as the typical non-quantitative ways ideas are vetted — as part of the ideation session or within a focus group — are replete with social biases (see my January article for more). But solid evidence that idea screening is a valid tool has been lacking – until now.
To quantitatively validate idea screening, we took about a year’s worth of concepts, brought them back to an idea-level stimulus, tested them via Ideas Overnight and then correlated the idea scores with the concept scores. We found that that idea-level measures correlated between +0.6 to +0.7 with concept measures – a very strong relationship given the difference in stimuli. This means that the core idea of your concept represents from 36-50% of the variation seen in concept scores.
Given that I’m so passionate about concept writing, I was also glad to see that those numbers weren’t any higher. This means that the rest of your concept (headline, insight, RTB) are still essential elements to the selling of your idea to consumers.