Super Crunchers. How anything can be predicted.

Ian Ayres
2007
Great Britain
Publisher: John Murray
Hardback
£16.99
260 pages
ISBN 978-0-7195-6463-5

This is a wonderfully clear exposition of how businesses are able to use the large
databases available to them to both predict and understand consumer behaviour. He
argues that the revolution in data storage has made large database interrogation
possible for all manner of businesses. His central premise is that statistical analysis in
field after field is uncovering hidden relationships among widely disparate kinds of
information that traditional experts in a field would never consider. The database
decision making tools he utilises will be familiar to many researchers namely
randomised testing and regression but researchers will be surprised and entertained by
his case histories which cover fields as diverse as politics, education and medicine.

Ian Ayres is a double professor at Yale in Law and Management. He has produced a
book that makes the world of data mining easily accessible to the non specialist.
Using a wonderful range of interesting examples he shows, for example, that it is
possible to predict whether or not a wine vintage will be good or bad not by asking a
wine expert but by plugging the weather statistics into a very simple regression
equation. This book is for those of you who like me are fascinated to learn that for
every centimetre of winter rain the expected price of wine increases by 0.00117
dollars!!

Researchers will enjoy and learn from this book as he shows how a new breed of
number crunchers who he calls “Super Crunchers” have emerged in many fields,
including marketing sciences, and who seek to discover empirical correlations
between seemingly unrelated things. He argues that this type of analysis is much
better at predicting the market potential of untried and immature products than so
called experts. He makes a strong and welcome case for the central role of empirical
data in decision making. He argues that we are in an historic moment where intuitive
and experiential expertise is losing out to number crunching as businesses
increasingly rely on large databases that can be speedily interrogated.

He makes interesting connections to some of the most influential marketing books of

the last few years. He shows, for example, how the use of this type of analysis lets
sellers access the “long tail” identified by Chris Anderson, via the use of
“collaborative filters” such as those that makes recommendations to the consumer
when you buy a book from Amazon or identify the music you like to hear on
Pandora.com. Interestingly he sees these collaborative filters as examples of James
Surowiecki’s “Wisdom of Crowds” in which the data collected from people like you
can make accurate predictions of what you might like to buy. This is the wisdom of
crowds that goes beyond the conscious choices of individual members to see what
works at the unconscious level. In his world of Super Crunching companies that use
this approach may be able make more accurate predictions of how individuals will
actually behave than the individual. This goes to the heart of some of the key issues
that Wendy Gordon, among others, has identified as facing our profession.

His final position is that the rise of Super Crunching does not mean the end of
intuition or expertise but rather that decision makers of the future will increasingly
need to toggle between their intuitions and data-based decision making. Their
intuitions will guide them to ask new questions of the data while Super Crunching
will facilitate hypothesis testing. He envisages a Bayesian future in which, as we
toggle back and forth between intuition and statistical predictions, we update
predictions and intuitions over time as we get new information from the wealth of
available databases and the ease with which we can conduct online randomised tests.
The future, he argues, belongs to those who can inhabit both the world of intuition
and the world of data. That sounds like a rosy future for market research to me

I recommend this book to all who wish to see how the world is becoming a rich data
mine and who want to start liberating the value that is hidden in that mine. In short his
book is for those of you, like me, who enjoyed “Freakonomics” by Levitt and Dubner.
It will similarly entertain and educate. You’ll be fascinated by questions such as why
have pilots readily adopted flight support software but doctors been slow to utilise
evidence based medicine? “Pilots go down with their planes”.

Mike Cooke
Global Director: GfK NOP Online Centre of Excellence