Sample Size of 1? Run! Understanding Proof Points vs Case Studies
One month ago I signed up to be an investor in peer to peer lending platform LendingClub.com. Not an investor in the actual company, but a “loaner” of money to strangers who have signed up to borrow money via the service. Why? The potential exists for 7% – 10% returns on my money, a rate that may outpace the stock market in the next 3 years.
I’m nervous that this is a relatively new investment vehicle, but also about the negative impact of defaulters on my portfolio – very nervous. I would guess I’m about as nervous as a CMO committing millions of ad dollars to new media vehicles. Was I swayed by the individual testimonials of other investors featured on the LendingClub site? No way. How could the experience of 1 of tens of thousands of investors possibly be what I expect to experience?
Then why do we expect digital marketers to be swayed by the experience of one brand, one campaign, one buy? Or, what we call, “case studies” . . . .
What did sway me in the end? A back testing engine enabling over 60 filters of hundreds of thousands of loans – the ability to analyze what variables impact people paying back their loans. Income, credit score, loan amount, state of residence, arrests, delinquencies, open credit lines – I could see exactly what combination of borrower variables resulted in higher yield for me.
Wouldn’t be great if we could do that for advertising? If we could isolate digital campaign variables to understand what combinations lead to the highest lifts in Branding & Engagement metrics? A Normative Database housing thousands of campaigns across every category – that would be great! Wait, we can?
Yes we can. We can analyze the presence of a human face, creative size, branding vs call to action, endemic vs non-endemic, low tenure vs high tenure brand and understand how KPI is effected by these variables. But does past performance reflect future performance? Not necessarily in the stock market, but in advertising it does.
We looked at a time comparison of 2013 AOL Interaction data vs YTD 2014 – then we looked at 2012 AOL Branding data vs 2013 to see if the previous period of normative data was predictive of the subsequent time period. Turns out it was.
To be sure, case studies are useful if the case study subject IS your particular client. They’re also helpful in dissecting specific creative strategy – getting us closer to the “why” behind a strong performance.
The next time you’re looking for prove the potential impact of a digital campaign or plan – remember strength in numbers.
1 - Moat, 2013/2014
2 – Dynamic Logic Marketnorms