his blog focuses on the science of designing surveys.

Looking at the best ways to

engage and communicate with survey respondents and how to generate effective
feedback from your surveys. It explores and identifies new ideas and techniques
surrounding online research. It looks at The Future of Market Research.

All Methodology Thoughts Gamification MR Buzz words Fun

Wednesday, 9 November 2016
How does a polling company find out how many followers the anti-research party has?
A conundrum for you...

Imagine I have set up a completely new political party and in my manifesto I tell
my followers not to trust the polls and to slam down the phone on any polling
company that tried to call and not answer any surveys.

My party is now effectively invisible to researchers!

How does a polling company work out how many followers this new party has?

Could this phenomenon go to explain the massive miss read in the US election polls?

...Trump painted polling companies as the enemy, it is no wonder some of his

followers as results might have refused to engage with them and as a result the
polls end up with a hole in their numbers.

This is the conundrum market researchers have to face up to if they want to get to
grips with political polling in the future.

We need to find a way of measuring the opinions of those that hide away from
expressing their opinions.

My mum spotted a nice solution, at her local church fete during the Brexit campagn
a stall was selling 2 varieties of rhubarb, "in" and "out" they sold 26 bundles on
in and 28 bundles of out! Job done.

Posted by Jonpuleston at 01:09

Feedback:
No comments:
Email This
BlogThis!
Share to Twitter
Share to Facebook
Share to Pinterest

Here is what just happened

I am writing this on the morning after the US election so I don't think I need to
explain the headline....

I am sure there will be more sophisticated explanations emerge over the following
day but this is my take to understand the result of the election.

In the run up to the US election out team have been conducting an ongoing series of
political polling experiments to understand what has been going on. We have not
been polling in a conventional sense, more experimenting with how to measure voting
sentiments and voting intention in an attempt to find a better way of predicting
the outcome of tight elections.

We have been asking a lot more indirect question about the difficulties people were
having in making up their mind, exploring more implicit measures of voter sentiment
to see how they stack up with declared measures, getting people to play games to
predict the outcome to reveal some of their hidden feelings and we have also
focused a lot of attention on asking open ended questions to measure the level of
passion in the arguments and look at what reasons people have been using to make
their choices.

Here is what I think has just happened from my perspective from the learnings from
all this research.

Trump messaging was far stronger than Hillarys from the get go. It was more
coherent. Make American great again. Close the borders. Drain the swamp, Hillary
Clinton is corrupt. We were seeing all this being echoed back over and over again
to us in the explanations gave us for why people wanted to vote for him.

On the other hand Clintons messaging was, extremely weak, in fact almost none of it
seemed to stick. Less than 10% I estimate of the reasons cited for voting for
Clinton were anything to do with liking her policies, it was nearly all to do with
stopping Trump winning. So many people caveated their choice with an explanation
that they were picking the lesser of 2 evils.

All the implicit candidate favourability measures we undertook showed us how much
Clinton grated on the American public. Flashing pictures of her face solicited up
to a -60% negative reaction, even worse that Trump who is known as being a
pantomime villain. Here face and perhaps more importantly her voice did not fit.
The majority of American public implicitly did not warm to her.

What Trump was up against in this campaign was not Clinton, but Trump himself and,
well let's not beat about the bush here, his personal sociopathic character traits.

The Trump sexual harassment scandal embodied all this and the misgivings so many
people were having about handing him power seriously pegged back Trumps latent
momentum in the final month of the campaign as all this news broke out.

But beneath all this, he was a lot of people's implicit preference, but they could
not express that in opinion polls or even to themselves to that matter due to the
outrage being voiced in the media about his behaviour, but his core message
resonated.

Then comes along the FBI email investigation a week or so before the election. This
was like pulling out literally a Trump card. What is did was give all the people
who latently liked his messaging but were suffering from cognitive dissonance over
his character, a strong emotional counter argument for prefering him it gave them
something that they could dress up as a lot more significant, it validated all his
messaging about his opponent too. It could not have been more perfectly timed.

We actually saw the change happening in real time as we conducted a large scale
research experiment the week before the FBI press release so on the Monday after
this we did a follow up piece of research to see what was happening and Clinton's
7 point lead we had seen the week before literally evaporated overnight.

The shy Trump supporters were released from the closet so to speak and at the same
time all the people who didn't like Trump or Clinton were given a strong reason not
to vote for either candidate or stay at home and not vote at all.

The last minute FBI volte-face was far too late in the day to undo any of this.
What we have learnt from all this that the public's opinion on how they will vote
is very emotional process, similar to the way my daughter makes up her mind about
what type of curry she wants to order in our local curry house. She tries on
several choices to see how she feel about them thinks she has made up her mind,
changes it, changes it again and then ditches them at the last minute when the
waiter is standing over her and decides with
her gut instinct.

In this case the US gut fancied trying something new, despite some serious
misgivings because the other dish did not seem too appetizing at all they decided
to roll the dice.

You can chide the pollsters if you like, but this type of emotionally charged
election is almost impossible to predict even a day or so out but certainly its
clear you cannot predict an election by simply asking people which way they intend
to vote.

Here are a few other closing thought about why the polls got it wrong....

1. Are the types of people that slam down the phone if a pollster calls and would
never think about doing an online survey also the same type of people who might be
more likely to have voted for Trump?

3. Clearly Hillary Clinton did not motivate here democratic base to vote in the
same way Trump rallied his supporters and so the polls which are often weighted by
past voting activity were delivering a miss read

2. Likewise did weighting polls by traditional political allegiances have any

relevance in this election

4. Male blue collar workers who voted for Trump in droves are the hardest group to
reach with research as they are working

5. There is some evidence in our research that Trump supporters were slower to
respond to our online poll invitations and so some short turn around polls might
have closed up before all the Trump supports had a chance to register their
opinion.

Posted by Jonpuleston at 01:09

Feedback:
No comments:
Email This
BlogThis!
Share to Twitter
Share to Facebook
Share to Pinterest

Monday, 17 October 2016

My name is Jon Puleston and I am addicted to information
From the moment I get up in the morning to the last thing at night I am immersed in
information gathering.

News was something I used to read once a day. Ever since having a smart phone my
propensity to consume news has slowly increased month by month - and with the ever
increasing proliferation of news aggregation apps, its becoming something I dip
into almost every spare moment during the day. The first thing I do when I wake
up in the morning and last thing I do at night before switching off my phone is
check the news . It had become a total addiction. In addition to news there is
social media, which I consume with equal levels of hunger, be it Facebook or
Twitter or Linkedin. When I run out of new information from these sources to
consume, I switch to doing things like looking through pictures on Instagram or
virtual shopping on ebay.

During the day I find myself checking out the news every time I have a break,
make a coffee, go to grab lunch, go to the toilet or in the middle of a dull
meeting. Whilst watching TV I find my hand reaching out so often to my smart phone
to double screen consume information, whilst I listen to the radio news while
eating breakfast I am scanning the same news on my mobile. When I stop at a
traffic light I stifle the instict to grab my phone. While I walk down the street
anywhere my head is down foraging like a hungry pig for news truffles.

The way I am using news information is changing too. I find myself getting so
emotionally involved, in the same way perhaps as you get hooked into soap operas if
you watch them every day.

I find I am far more drawn into politics than I have been in the past. The
campaigning and social nature of information content is so much more significant
than it once was.

Politics for me is replacing soap operas literally.

The major news stories like the Scottish referendum, the UK election, the EU
referendum, the US election have all completely hooked me in. I have literally
mainlined news content from these political stories.

and how I am using and processing this information goes way beyond objective
information gathering. What I am looking for so often are arguments and evidence
that support my point of view. I am highly practiced in SCHADENFREUDE, delighting
in reading about the downfall of political foes, people with opposite opinions
having their arguments skilfully met.

The more news information I consume the broader the range of topics I need to
consume to feed my habit

Take sport for example, I dont watch much football these days, but I do consume
vast amounts of football information. I track the action of most of the matches on
a Saturday afternoon via news updates that ping me whenever certain teams score. I
listen to football pundits on the radio talking about football for hours a week. I
ferociously consume transfer news gossip and read numerous sporting blogs.

For all the tut-tutting I am probably drawn into as much salacious celebrity gossip
as your average teenager - but at the same time, and paradoxically, Im equally
drawn into hard core science information some in depth article about a
breakthrough in quantum physics or some Kardashian family antics are all hoovered
up into my brain with little differentiation, hanging around for a short while
before fading into the soup of other meaningless information I have consumed.

Searching for distractions

I find myself getting more increasingly distracted too, doing all those stupid mind
challenges they post on Facebook, I am like a moth drawn to them like a light. I
have always been an obsessional problem solver and so struggle to ever pass a
problem without wanting to think about it.

Entertain me, feed me with news, tell me something I didnt know, titillate me,
arouse me, impress me, make me feel happy, make me feel outraged, tell me a secret,
fill the void, surprise me, confirm to me I am normal, summarise the complicated,
provide me with a cheat sheet to life, wow me, take me out of the now to a more
interesting place, feed me with information about new technology, pander to my
political sensibilities, align with my point of view, tell me something bad about
Donald trump, help me to feel any emotion, I dont really care which one it is.

Let me be clear this is a serious addiction

Akin to smoking cigarettes and taking heroin you chase the dragon of dopamine hits,
but the more you do it the less you get out of it. Perversely this forces you to
consume ever larger quantities of it to the point that you realise that its taking
over your thinking patterns and disrupting your life.

And its not just me whos hooked, if you are reading this you are likely to be
hooked too!

Its a global epidemic.

Snow my partner came across this New York times article recently and sent it too
me, its long but worth a read http://nymag.com/selectall/2016/09/andrew-sullivan-
technology-almost-killed-me.html

The comparison I make is with alcohol consumption in the middle ages. When beer
started to become freely available, with the development of brewery skills everyone
drank beer all day long without constraint, even young children drank it and it and
took a few centuries to understand the negative impact it could have on our lives
and to establish the social rules for its consumption.

We are at similar point in history I think with information consumption on our

mobile phones. It is rather getting out of control. We are all trying to fight it I
am sure, like for example banning phones from meal times and establishing social
rules for when its acceptable to whip out your phone, but I think there is still a
long way to go.

Investigating information addiction.

With this background I decided to do some research to examine how are we are
processing this sea of information we are consuming nowadays.

What type of information is cutting through and why?

How do brands actually cut though and compete?

If you are interested to find out more about this subject come to the MRS Customers
Exposed event on Thursday 27th when I will be exploring this topic and revealing
some of the results of this research.

https://www.mrs.org.uk/event/conferences/customers_exposed_2016/course/4742/id/1174
2

Mea Culpa

As Mark one of my best friends politely reminds me sometimes. I am both observing

the problems but have become part of the problem too. The information I am
delivering is rather adding and abetting to process which is on my conscious.
If you feel like you are suffering from this addiction and looking to make steps to
deal with it here is a website I found useful.

http://www.psychguides.com/guides/treatment-for-addiction-to-smartphones/

Posted by Jonpuleston at 02:04

Feedback:
2 comments:
Email This
BlogThis!
Share to Twitter
Share to Facebook
Share to Pinterest

Monday, 13 June 2016

Why do we rate everything 7? I blame teachers.
Have you ever thought it odd why when we score thing we have a tendency to
disproportionately use the top end of the scale? When you ask people to rate
something on a scale of 1 to 10 the average person in most countries tend to score
things 7.

Now if we lived in a logical world you would have thought average score when we
rate things on a scale of 1 to 10 should be 5.

So why do we over score average? Well forget acquiescence bias, I have decided to
blame it on our teachers!

For most of us the first exposure we get to scoring is at school, when we get our
work marked. I am sure nearly everyone can remember those anxious feelings as the
teacher handed out the homework in class and you opened your homework book to see
what mark you got.

In British schools we tended to get marks out of 10. If I got 8, 9 or 10 I was

happy, that was what I was aiming for. A mark of 6 and below was a disaster as far
as I was concerned and 7 seemed to be what you got most of the time. 7 was what I
perceived as average.

7 being use as average is irrational of course. Children across the spectrum of a

class should be as likely to get a score below 5 as above 5 if the process of
marking was done totally rationally.

Yet when I think back I am not sure in my whole early school life did I ever get a
mark less than 5 even in English and god have you seen my spelling!

The whole processes gets corrupted by the natural eagerness of of teachers to

encourage us , so everything gets shifted up a few notches from reality to make us
feel good.

As you progress through schooling system they start to use another marking system
called grades and that is even more irrational as it has a built in scale heavily
weighted to the positive. Instead of being graded A,B,C,D,E,F,G,H,I,J,K They used
this more euphemistic version: A+, A, A-, B+, B, B-, C+, C, C-, D, E. Where a B
could actually mean you really got a mark 5 off the top score.

I think that teachers in their efforts not to disappoint us all have totally f%*@d
up our natural internal scoring mechanismAnd its us researchers are the ones that
have to deal with the consequences of all this later in life.
These experiences of being marked have anchored a scoring systems in our psyche
that is near impossible to shake clear of in later life when we dish out marks
ourselves in surveys. It badly scars how we mark. If you ask people to rate
brands on a scale of 1 to 10 they nearly all get 7. If you ask people to rate ads
they get an average of 7 if you ask people to rate films 80% of score are between 3
and 4.5 stars = 7.

We so often mark things as 7 where in truth we dont mean it, or rather when we
cannot be bothered to think about it or when an experience is un-contextualized.
This means we have to do a lot more work to actually find out the truth sometimes.

Posted by Jonpuleston at 08:46

Feedback:
6 comments:
Email This
BlogThis!
Share to Twitter
Share to Facebook
Share to Pinterest

Saturday, 11 June 2016

Overfit
Has this happened to you?

You are running a piece of research and you look at the results from the first 50
respondents and it looks like a really good story is emerging. You are seeing
some big differences in how some of the people are answering some of the questions
start to come up with theories as to why. You get exited and you build a whole
story about what the data is saying and all seems to make perfect sense.

Seeing and spotting trends in data is what we all love to do, in fact that is
largely what our brains are set up to do, to spot difference and try an interpret
them. It all too easy to come up with ready answers to explain why say men over 35
would prefer this brand of shampoo or why high income groups like cheese more than
low income groups....

You then go away and wait for the full sample to answer the survey and when you get
the results back back the data differences you saw initially have all evaporated.
The patterns you were seeing were in fact noise that you were treating as signal.
When the noise is statistically accounted for you are left with a sea of dull
homogeneous data with little or no stories to pull out.

Welcome to with world of overfit!

The term literally means "over fitting" theories on data that was not statistically
robust enough to validate these theories.

... And it's incredibly dangerous. Particularly in circumstances where you are
researching niche sample groups that are difficult to reach and you end up with a
completed survey with not enough sample. This is a particular problem in the world
of health care and BtoB resesarch where samples are hard to access.

It's difficult for us to get our heads round just how random random chance is, even
with large numbers.

So what does random look like?

Toss it 50 times and very rarely would you get exactly 25 heads and 25 tails. It
will happen only one in 10 occasions roughly.
In fact with 50 coin tosses there is a 60% chance that there will be be more than
20% difference - so the differences in the data looking like this chart below would
in fact be the expected norm.

If you had 20 question in the survey you would expect at least 1 of them by random
chance to have difference of 50% or more which looks like this....

Here is a summary of the data difference you would expect to see in a survey of 20
questions sampled to 50 people.

Here also is similar data for samples of 100 (sorry not done this for larger
samples it a bit of a pain to work out!)

How to be confident you data is reliable?

Simple trick is to divide it in 2 and seeing if both halves say that same thing.
Then do it again 20 times and see how many times its the same. If its the same 19
times out of 20 this is the definition of 95% certainty. The number of times out of
20 times by 5 will determine exactly how reliable your data is. You can go a step
further and divide the data in 4 and then look how often is the answer the same.
If all 4 cells give the same answer you are sitting on some quite robust data.

Posted by Jonpuleston at 06:15

Feedback:
No comments:
Email This
BlogThis!
Share to Twitter
Share to Facebook
Share to Pinterest

Saturday, 14 May 2016

The science of visual communication
In my job I conduct a large amount of research, and but also create plenty of
presentations. To help design good research, we have access to hundreds of
published research on research papers. Yet when it comes to designing presentations
or using any form of visuals, we have to rely largely on gut instinct and
experience to evaluate what works best. There are plenty of well-established
working practices and graphic design experts who are exceptionally good at what
they do, but very little research to help us to understand the impact of different
graphic design techniques, certainly in the market research arena.

Perhaps one of the reasons is that that graphic designers and market researchers
dont encounter each other very often.
A joint quest: researcher and graphic designer
Last year part of the Guardian's digital graphics unit responsible for creating
some of the most famous infographics circulated online, formed their own company,
the Graphic Digital Agency and happened to move into the same offices as our
research team in Westminster and we got talking about infographic design and the
lack of research to understand how it works. I was curious to know what they knew
about the science of design and I found out they were as curious as me. So we
though, using our experience in conducting research on research and their skills in
graphic design to produce the source material this represented a very good
opportunity for us to work together to do some experimentation. We sent out on a
joint quest to try and learn more about how visuals really work.

We ended up conducting over 70 experiments and tested over 500 visuals, icons,
charts, presentation and infographics on over 10,000 respondents in five countries,
one of the most extensive pieces of primary research I think we have ever
conducted. The complete findings have been published across two ESOMAR papers: The
quest to design the perfect icon, Puleston J & Sazuki S ESOMAR (2014) & Exploring
the use of visuals in the delivery of research data, Puleston J, Frost A, Stuart T,
ESOMAR (2014) . But I thought I would publish a summary of what we have learnt on
my blog site.

Read more
Posted by Jonpuleston at 01:17
Feedback:
No comments:
Email This
BlogThis!
Share to Twitter
Share to Facebook
Share to Pinterest

Tuesday, 10 May 2016

Researching the different words used by Women & Men
We have recently conducted a small piece of research to explore the differences in
the language used by men and women when they describe themselves and other people
including.

There are some quite surprising difference especially in the words men and women
use on their CV.

This link below is to a survey quiz to highlight some of the most popular words we
have identified with the biggest difference is gender usage. Please feel free to
circulate this link.