Beruflich Dokumente
Kultur Dokumente
Only
Poll That
Matters...
HOW THINGS WENT WRONG IN
CALGARY, AND HOW TO FIX IT.
Justin Ling
Its amazing how different all of the polling results
are--not an exact science.
As long as there have been polls, there have been You can change a variable, observe the results, and
questions raised about those polls. Present day, adjust accordingly. The art? Pollsters have had to
the industry and its interlopers are wrestling with justify their work to the public and to the politi-
how to reach a population that have hung up their cians they gauge. Theyve had to defend themselves
phones and moved online, but turn the clock back in the court of public opinion after catastrophic
and the issues feel very familiar. In 1949, Gallups mistakes, but rarely get a chance to talk up their
survey was the underpinning for the infamous work when they get it right. That, unfortunately
Dewey Defeats Truman headline. In 1982, the for pollsters, is the nature of the game.
industry and its media barnacles contemplated the
so-called Bradley Effect: Are white voters more At the same time, its hard to imagine public trust
likely to say theyll vote for a black candidate than in the opinion poll being any lower than it is today,
to actually do so? In the early 00s, the question and their skepticism is not altogether unreasonable.
became one of landlines versus cellphones, as the
public mused whether opinion polling had finally
heard its swan song.
I approached it much like I approach reporting a story: I spent hours trawling through polling reports
and media stories regarding the Calgary municipal election. I spoke with more than a dozen academics,
journalists, political staffers, and pollsters in Toronto, Montreal, Ottawa, and Calgary about their thoughts
on what went wrong in Calgary, and about polling in general.
I conducted all of those interviews on a strictly background, not-for-attribution basis. In this report, I do
not identify anyone who spoke with, nor do I identify those who did not (of which there were several, for
reasons that will become evident.) I did this in order to ensure that they would speak freely, candidly, and
so this report could become a helpful roadmap, not an airing of grievances.
While they may not be named, their comments, insight, and perceptions have heavily informed this report,
which has tried to stitch together what was, in many ways, a consensus.
This report is an assessment of what Mainstreet did wrong in that fateful municipal election in terms of
what Im placed to talk about: Its media relations strategy and its public-facing communications. Im not
here to assess what went wrong with Mainstreets polling itself thats someone elses job. Ive also not
been hired to implement any of these recommendations or suggestions, Ive just been hired to write the
report.
In writing the report, I did so with complete independence. While I have obviously spoken with the team
at Mainstreet to get their thoughts on what went wrong in Calgary, they had no role in telling me what to
write or, more crucially, what not to write.
As a journalist, I rarely report on opinion polling, and this report wont change that. While I am being
paid to produce this report, I have no financial stake in Mainstreet and have no other obvious conflicts of
interest that prevent me from producing this report or from continuing to report on the various goings-on
in Canada and abroad.
-Justin Ling
Three years later, Nenshi ran virtually unopposed. Calgarians are in a foul mood. Their city is
He won nearly three-quarters of the vote. Not struggling with high unemployment, heaps of
exactly a nail-biter. It was during this mandate that vacant downtown office space due to oilpatch
he won the World Mayor Prize. He was named the layoffs that has pushed the municipal tax bur-
best mayor of any major city. den to remaining businesses, and a string of
devastating energy project cancellations that
So that brings us to the 2017 race. are limiting future opportunities including
the Energy East pipeline last week. Theyve had
Seven years after being elected mayor, Nenshi re- it with anti-business governments, and Nenshi
mained a popular national figure and a vocal advo- just happens to be the first politician up for
cate for Canadas municipalities. Speculation and re-election. Hes seen as running a fat and inef-
rumour that he would make the jump to the federal ficient city hall that, like its senior government
Liberal Party or, perhaps, any of the other federal counterparts, raises taxes and spends without
or provincial parties, barring a few proved mis- restraint. The big ideas and high ideals Nen-
placed. He, by all accounts, just likes being mayor. shi the first Muslim mayor of a large North
American city stood for in booming times
But not everyone was Team Nenshi. In the lead-up arent so vital when everyone is tightening their
to the 2017 election, grumblings became apparent belt.
over Nenshis leadership. The centre-right base in
the city began looking for someone who could take
Some of this columnizing can be explained by a
need to explain the polling numbers emerging, Little more than a week later, Mainstreet had a new
absolutely, but many of the gripes and complaints poll. This one had the lead widening: Smith was
about Nenshi reported in these columns have been now 17 points ahead of Nenshi. With just 13% of
percolating for some time. I point this out not to voters undecided, Mainstreet felt confident enough
take aim at the mayor, and not just to wax nostal- to lead their report with the Dewey-Defeats-Tru-
gic, but in order to set up why one might suspect man title of: Calgarys Next Mayor is Smith,
that the erstwhile Worlds Best Mayor may have obviously, was the answer to that question.
been in trouble.
In a final poll, on October 13, Mainstreet found a
When Mainstreet Research went into the field in tightening of the race. They reported Smiths lead
March, 2017 some seven months before election had narrowed to just 13 points, when factoring out
day they found that Nenshis approval rating the undecideds. Their finding? Smith: 52%, Nenshi:
was, indeed, dropping. Only 56% of Calgarians 39%.
approved of his job performance, down nearly 10
points since January. A month later, when Main- The actual results, as anyone familiar with the race
street asked again, it was 52%. knows, were wildly different. In the only poll that
truly matters, as politicians always call it, Nenshi
Throughout the Summer earned just over 51% of the vote on election day,
and Fall, when Mainstreet besting Smiths 44% by nearly eight points.
asked voters their opin-
ions about building a new Not only did Mainstreet get the winner wrong, it
arena for the citys NHL had misjudged Nenshis support by some 12 points.
franchise (a controversial Their polling wasnt just bad, it was catastrophical-
topic in the city on which ly bad. Not to belabour the point, but a mistake of
Nenshi took a fervent, this magnitude regardless of the reason is hard
steadfast position) their enough to recover from, in and of itself. But the
polling found it to be far way in which Mainstreet communicated those poll-
from a slam-dunk issue, ing results, both before and after election day, added
with voters expressing un- kerosene to the fire.
derstandable reservations
about the deal. In Albertas newspapers, on television, and on radio,
Mainstreet broadcast absolute, unwavering confi-
Calgary, by all accounts, was a city tuned in to city dence that their numbers were correct before the
goings-on and was growing increasingly bearish on vote, in some cases ignoring the obvious warning
the direction of the city. signs that should have given them pause. On social
media, Mainstreet bickered and argued with on s
So when Mainstreet published its first real campaign variety of the issues, even after election day. In their
poll, on September 30, its results fit that trend. Nen- polling reports, Mainstreet prognosticated on Nen-
shis approval was a pathetic 36%. He stood some 9 shis apparent political failure, using their data as the
points behind his nearest rival, Bill Smith. backdrop to explain the current political situation in
Calgary.
That was a massive fall from his 62% approval
rating that Mainstreet registered in March, 2016. It was that front-facing communication strategy
Whats more, he was trailing behind a political no- or, really, lack thereof that turned a failure into a
body, in Smith: A one-time president of the prov- disaster.
inces Progressive Conservative Party, a firefighter,
and a lawyer.
What Mainstreet
Didnt Get Wrong
This report focuses pretty intently on where things Mainstreet also fielded allegations that its methodol-
went off the rails and where mistakes were made. ogy itself the very core of its work was wrong.
But its useful, before that, to identify some of the This argument can be a very technical one, and a
misperceptions that filtered into this whole affair partner report to this one is tasked with delving into
and which percolated on social media and in the those technicalities, but its worth addressing the
traditional press. top-line of this criticism.
First off, there really is no basis for the claim that First off, there almost certainly seemed to be sample
Mainstreet was in the bag for either the Smith cam- issues with the Mainstreet polling in Calgary (and
paign; the Calgary Flames, whose public dispute that issue may, it seems, be unique to Calgary.)
with the mayor grabbed headlines; or anyone else. Some, even those very well versed on polling,
Mainstreet had no political dog in the race. sometimes use the word methodology to mean
sample. Theyre not entirely wrong, per se, or at
Of course, users on Twitter and elsewhere were least not in a way that is meaningful in the public
quick to point fingers over Mainstreets wonky debate, but it can be misleading and wound up
results and its confidence in the state of the race as being the basis for several unnecessary arguments
evidence that the firm was secretly working hand- instigated by Mainstreet.
in-glove with Nenshis opponents. That notion was
certainly fed by Mainstreets bravado but I think Lets accept that Main-
they mistook the companys attempts to justify streets sample was, in
their unexpected results for an attempt to get Smith fact, off, and lets hone
elected. in on the methodology,
the process whereby
There was also a line of argument that Mainstreet Mainstreet actually
was owned and operated in one way or another conducts its polling.
by Postmedia and that the newspaper chain had a
vested interest in getting Smith elected. While the Public perception of
news chain and the polling firm obviously have polling can, at times, be
an arrangement, the perception that there was an facile. There still exists
organized effort to distort the results by both the a relatively binary belief that live-agent polling is
citys largest newspapers and Mainstreet Research god, robocalls are junk, and there is little gray space.
is, of course, absurd. But it is a reflected on how
bad public trust has gotten for both the mainstream In fact, every type of poll has its benefits and draw-
media and the polling industry. backs, and its generally unfair to merely throw a
single type of poll into the bin. An ideal campaign
Unfortunately, Im not sure Mainstreets efforts would have a mix of live-agent polling, Interac-
to counter that false narrative really took. As the tive Voice Response (IVR) calls, and online panels.
pile-on of critics began, Mainstreet got too far into But one of the benefits of IVR and online panels is
the muck in trying to argue and bicker away the that they tend to be lower-cost and are more able
conspiracy theory, sometimes conflating legitimate to enter the field for smaller races, like the one in
criticism with those invented allegations. Calgary.
To that end, asserting that we should do away with commentary. For those armchair pollsters who be-
IVR altogether is wrong. A post-election CBC lieved that Mainstreet was faulty simply because of
story report that Edmonton mayor pans automated how it made its dials, it was not.
polls, citing a scrum with Edmonton mayor Don
Iveson where he told reporters We should stop There were other, smaller, allegations made against
reporting things that are off by 10 or 15 per cent Mainstreet that made during the campaign, such as
and distort or oversample or undersample certain one suggesting that Mainstreet did not publicize its
groups. (Its not clear whether Iveson really sin- survey questions (it does.) Those were, generally,
gled-out IVR polls, or whether that was simply honest mistakes.
CBCs interpretation.)
Either way, a big concern for Mainstreet, going
Either way, Mainstreets problem was not that it forward, should be around how it responds to mis-
used IVR polls, and few (informed) commentators takes, allegations, and criticism. The hardest things
really suggested as much, although that quickly to address are the things that arent true at all. More
became part of the narrative. on that later.
There were, however, potential aspects of the But I include all this not to pull punches on Main-
Mainstreet methodology for example, whether street, but to note that the maelstrom over the race
Calgarys 18-34-year-old voters simply refused to was muddled and clouded with some inaccuracies
answer the automated polls, or whether Mainstreets that whipped up some undue criticism of the com-
likely voter model was sufficiently tuned. pany. It also explains some of the reason why Main-
streets tone became so frustrated and terse.
So for those critics who alluded to these potential
problems, they were certainly offering informed
In the context of the And, in the end, Mainstreet got it right. They put
poll, which showed Rachel Notley and the NDP at 44% three days be-
the incumbent may- fore the election. Notley came in just shy of 41%, a
or down by over ten difference that can likely be explained by last-min-
points, Mainstreet ute vote shifts.
was playing down-
right coy, at least But, of course, it is the gamblers fallacy to believe
compared to how that just because you made one right call that all
they would go on to of your future calls will be equally correct. Even
communicate their if you, like Mainstreet, had made right-call-after-
next set of results. right-call, it doesnt (necessarily) establish that your
next one will be on the nose. Polling, of course,
Without getting into the very serious red flags that isnt gambling, but it is still a science that can be
shouldve shot up based on this and subsequent re- acted upon by an array of unforeseen variables.
ports (well get into that later in this report) I think
its worth digging into the political projections that Which brings us back to Calgary, and Mainstreets
came out of Mainstreets research reports. first poll. Maggis analysis of the municipal land-
scape may have been fed by numbers that, looking
Pollster-as-pundit is not a new or unusual thing back, appear to have been severely skewed, but his
in the Canadian political sphere. Especially as the analysis nevertheless came pretty near the mark. It
political debate becomes increasingly dogged with was, as with any good punditry, informed by more
partisan encampments, pollsters should be bastions than one variable.
of non-partisan perspective, relying on the data in
front of them to draw conclusions on whos up, It was the October 13 poll that really set things into
whos down, and what it all means. tailspin, and put Mainstreet up against the boards, as
it were.
Mainstreet does exactly that. Take the following
passage from their polling report for the 2015 Al-
In that second, and penultimate, poll, Maggi makes only independent pollster in the field. And Mainstreet
some bold claims. points to that lack of data as a reason for why they
continued to trust their own numbers.
The Nenshi campaign could not see the change in
landscape and pivot successfully to issues that average But, of course, there were other polls in the field. Just
Calgarians care about, he writes in the report. Maggi not ones that Mainstreet chose to listen to.
sweeps through a conclusion that property taxes and a
sluggish economy are to blame for much of the, appar-
ent, growing resentment of Nenshi, but he hones in
on the idea that it is Nenshi himself who has sunk his
chances.
The reality is, Mainstreet was not as confident as they The Canadian Municipal Election Survey used Forum
appeared. After the first poll, amid concerns that their Research to recruit respondents for an online panel,
sample may be under-representing Nenshis base of which had a long lead, from late September, right up
support, they took steps to validate their results. They until the week before the election. That poll, similarly,
increased sample size. They later confirmed their had Nenshi winning by 21 points, after factoring out
phone results with online panels. undecided voters.
Unfortunately, looking back, it appears as though those Both LRT on the Green and the Canadian Municipal
attempts to double-check the data may have only en- Election Survey published their results with undecided
trenched the initial mistakes. voters included, and those results which had Nen-
shi up by 15 points and 17 points, respectively were
In a vacuum, Mainstreets process was pretty reason- reported widely in the media.
able: Their results were odd, they checked them, the
results remained the same. Even though there were It does seem that both over-represented Nenshis
incredibly odd results (Mainstreet showed that cen- support. (Or, of course, they were entirely accurate and
tre-left Nenshi performed worse with women than voters merely changed their minds in the last week.)
men and was incredibly weak in the 18-34 demograph-
ic) the process supported them. In a normal election, So why not listen to those polls, both of which showed
Mainstreet would be able to double-check its results consistent support for the incumbent mayor, especially
against Leger, Angus Reid, Ipsos, Forum a range and in contrast to Mainstreet in his core demo-
of Canadian pollsters but in Calgary, they were the graphics?
The Asking Canadians poll was not a normal
horserace poll. It was, in essence, a push poll. While Even in its third polling report, the one that showed
some use that phrase disparagingly, its not neces- the tightening race, Mainstreet reminded readers
sarily a bad word. LRT on the Green asked four that it had successfully predicted two Calgary bye-
questions before asking about their preference for lections in the past year.
mayor, all four designed to test the publics attitude
towards a new LRT line and to prod their opinions While this swagger has served Mainstreet relatively
in support of it. The final question, over their pre- well thus far, it is a high-risk, high-reward strate-
ferred mayor, came after those questions, meaning gy. Hanging your business model on infallibility in
it could have heavily favour Nenshi, a supporter of foresight is a recipe for headaches, if not outright
the line, as the favoured candidate. disaster. It seems, from reading Maggis NHL-based
analysis, Mainstreet had come to believe that it was
The Canadian Municipal Election Survey, mean- not just the best indicator of the Calgary race, but
while, was simply an unknown. While Forum the only indicator.
Research is a well-established firm, the survey, an
academic project, had never published an election Its that over-confidence that also led them to in-
poll before. They had only existed for a matter of clude those the Calgary Foothills byelection that
months before conducting the poll. Their method- turned up in the third report. Yes, Mainstreet called
ology, to recruit participants over the phone and the winner of that race correctly, but not the results.
invite them to an online panel, remains a question The firm predicted (albeit nearly two weeks before
mark in terms of accuracy, although there appears election day) that the Wildrose Party would win by
to be growing consensus that it is as legitimate as a hair, with 29% of the vote. On election day, they
many other methods. garnered nearly ten points higher. The poll also had
the Alberta Liberal Party overperforming two-to-
There was reason to be skeptical of both those polls, one. Now, it is entirely possible that Mainstreet was
but disregarding them entirely was a mistake. They right, and the Wildrose merely opened their lead
should have been given appropriate weight, given in the final two weeks. But it is equally as valid to
their shortcomings, but not ignored. assert that Mainstreet had failed to gauge the Wil-
drose support in the city by over-estimating support
Perhaps, in a normal election, Mainstreet would for the Liberals. All this to say is that concern over
have at least tipped its hat to those polls. Instead, picking the winner appears to have, in some cases,
they delved into a bunker mentality, one where trumped importance of getting the data right.
they had to defend their results to the questioning
masses. In so doing, they regularly took aim at the Some at Mainstreet might think this characteriza-
LRT on the Green poll, openly bashing their re- tion is unfair. But, fact is, Mainstreets perception
sults. doesnt matter. What matters is how the public sees
their brand. And the reality is that the perception of
In many ways, Mainstreets over-confidence was a Mainstreet as unreliable now exists, and its a per-
gamble. Mainstreet sees its value in the market as ception that culminated (but did not begin) at the
its accuracy and foresight as many pollsters do. apex of the Calgary municipal election.
But Mainstreet has also tried to carve out a niche for
itself as a scrappy, agile polling firm that can jump
into the field throughout the race, even races where
few others seem interested, and create trendlines.
They particularly see value in being the last pollster
to come out with results, a position that often allows
them to make the boldest claims: Majority, instead
of minority; upset, instead of status quo.
Especially as the political debate
becomes increasingly dogged with
partisan encampments, pollsters
should be bastions of non-
partisan perspective, relying on
the data in front of them to draw
conclusions on whos up, whos
down, and what it all means.
This isnt to say that Maggi and What was supposed to come off as
Valentin should answer journalists confidence came off as arrogance,
calls and read from a script. Theres what was intended to be competi-
nothing a journalist hates more than tive sounded combative, what was
a source, much less a pundit, who position as defensive ended up com-
only sticks to their talking points. ing off aggressive.
On the other end, speaking off-
the-cuff, like Maggi and Valentin Someone suggested the Canadian
did, was completely self-defeating. They frequent- Municipal Election Survey shouldnt be brushed off.
ly veered from what they knew into what they Maggis response? Every pollster in Canada agrees
thought. They speculated, called out individuals, with me. When LRT on the Green published its
and prolonged a bad story with a complete lack of results, Maggi jumps in: Buying panel responses
strategy or thought. doesnt make you a pollster. When Nenshis own
pollster needled Mainstreet, Maggi couldnt resist
Take Maggis public statement on the election responding. That furthered the narrative that it was
failure, posted to the company website. It, on one Nenshi vs. Mainstreet a perception that should
hand, was supposed to serve as a mea culpa. On the have never been allowed to bubble up.
other, he writes: A number of our previous polls
were attacked as inaccurate without context. It is All of this builds on Mainstreets reputation as
difficult to accept criticism at the best of times, but cocky, even making the company look downright
when it is based on false information, it is especially angry.
so.
Maggi is a passionate guy and he, like lots and lots
Pride is a hell of a thing. But sometimes you need to of business owners, have an automatic and under-
swallow it, stop trying to save face, and stop trying standable impulse to respond to naysayers online.
to dig. Given weve already established that Main- Like countless restaurant owners who cant let a
streets modus operandi is, above all else, to win; its bad Yelp review go, or like a long-shot political
crucial that it knows how to lose. candidate who feel the need to feed every troll with
a Twitter, Maggi wants to joust with anyone who
Again, it wasnt all bad. Especially after that first impugns his business.
poll, when Maggi spoke to the Herald, he admitted:
The fact that [Smith] is leading outright and by
Photo by Priscilla Du Preez on Unsplash
Any social media manager worth their salt will tell took quite seriously the idea that Maggi and Main-
you that this strategy never works. It is rarely help- street were taking down names.
ful to confront the haters. Social media, as often as
humanly possible, should be conducted with a great I dont believe the firm ever had any plans for ret-
big smile on your face (and, ideally, with full sen- ribution, but thats beside the point. The r-word
tences and proper punctuation.) may never have been used, but the perception of
that sentiment was very real. And Maggi did in-
It wasnt always bad. Sometimes, when Calgarian dicate his plans to get litigious. The fact that their
onlookers wondered how Mainstreet results were careless media strategy allowed for that belief, even
so outside the realm of the conventional wisdom, in a single person, is a sign that Mainstreets tone
Maggi would jump in to note the work his firm had was wildly off-base and inappropriate for a com-
done to double-check their results (work, by the pany trying to comport itself in the public, and
way, that the company mostly failed to advertise through the media.
publicly.)
That is encompassed so well in the decision to pub-
But mostly, it was bad. lish a scorecard, ranking the other pollsters on their
results. The card, published by Valentin but tweeted
One of the best pieces of media advice I can give is out enthusiastically by Maggi, asks voters to rank
this: When composing a tweet or Facebook post, how each polling firm did Mainstreet; Asking
writing a release, or speaking to a journalist, think Canadians, by way of LRT on the Green; and Fo-
of how your words, including your tone, would rum, by way of the Canadian Municipal Election
read to someone else. Put yourself in their Twitter Survey.
account, as it were.
What utility this served, beyond trying to humil-
Its simple advice, but it is apparently advice that iate an organization trying to promote a light-rail
Maggi, Mainstreet, and scores of others need to be line and a group of academics studying municipal
reminded of. I spoke to individuals who told me elections, is lost on me and, I suspect, most people
they honestly feared for their career if Smith had, who saw it.
indeed, won and Mainstreet had been right. They
How to Fix It
For all the mistakes made, there are easy solutions. Mainstreet, in Calgary, was victim of a conflagra-
tion of variables. They had no other independent polls to check their results on, they wound up in
the field in the height of acrimony and distrust aimed at pollsters. Members of the team had personal
issues to attend to.
But, even without those issues, this was a long-time coming. Every pollster in Canada should be
ready for the day they get it wrong, very wrong. Mainstreet, evidently, was not.
The following are some practical tips and guidelines that Mainstreet ought to adopt as soon as possi-
ble.
Do not try to opine on the motivations of your detractors. You are doing media
interviews in order to advance your own good work, not to tear down others even
if theyre tearing you down first. The schoolyard adage of ignore them and they will
go away is, surprisingly, a good rule of thumb for the corporate world.
But, fundamentally, many people who peddle inaccuracies are ignorant to the facts,
sometimes willfully. As weve come to realize, increasingly, sometimes people wish to
be entitled to their alternative facts. Arguing, educating, or correcting them is often
not worth the effort.
When companies screw up, they are usually faced with a choice: Door A, deny, deny,
deny and try to argue your way out of the scandal; Door B, go sword-and-shield style
communications, where the company buys a rainforest and paints over their dark
chapter or; Door C, fess up to the mistakes and grovel for forgiveness.
Many firms still go Door A, but increasingly opt for Door B. The right answer is Door
C.
If Mainstreet wants to be known as anything other than the firm that screwed up the
Calgary mayoral election, it needs to fully admit its failure, go public with the reasons
why it failed, show the public (and the media) where it will fix the mistakes, and never
stop reminding the public that it can and will be better. Pull the curtain back. These
reports, obviously, as a step on that process, but it cant be the last step.
The companys releases are certainly well-written and entertaining, but I do not think
they do enough to educate voters and journalists to the context the polls exist in.
Some pollsters have made a point to zoom out on their data, providing a birds-eye
view of the landscape that can influence public perception. Their reports are wonky,
detailed, and sometimes even present contrary viewpoints on what the data means.
That firm, and others like it, appear more interested with trying to explain the entire
ecosystem than with trying to, as we often say, explain a snapshot in time.
Opinion
That model doesnt work for everyone, but its worth
taking stock of. The fact that Mainstreet never detailed
research
its work to confirm its own wonky results in its official
releases its increased sample sizes and work to confirm
its IVR results with an online panel was a mistake, and it
was also a mistake not to identify the possible shortcomings literacy can
in their own data. It was a mistake not to include mention
of the other two polls in the field. obviously
At the end of the day, few people read polling reports. Its be vastly
mostly journalists, partisans, staffers, academics, and wonks.
Assuming that those who read the reports are either well- improved, and
versed already, interested in learning, or are in a position to
educate others on the poll. Pull the curtain back. its incumbent
Many journalists tasked with writing stories based on polls
on pollsters to
only have a tenuous grasp of what really differentiates an
IVR poll with a live-caller poll with online panels. Many
do that work.
may understand the concept behind a likely voter model,
but few could explain it any depth. Take the opportunity, in official releases, on the
Mainstreet website, and in telephone interviews, to help them. Opinion research
literacy can obviously be vastly improved, and its incumbent on pollsters to do that
work.
To that end, the boilerplate prefix The Mainstreet/Postmedia poll has margin of
error of +/- 2.53 percentage points, 19 times out of 20 isnt enough anymore. To
borrow a phrase from Homer Simpson, It doesnt mean anything, like Rama-Lama-
Ding-Dong or Give Peace a Chance.
What variables would give the 1 time out of 20? What would happen if the margin
of error were applied to the results? Are there any red flags in the result that should be
highlighted?
Many think that acknowledging your own shortcomings undercuts your work. It
doesnt. It strengthens it. As we learned in Calgary, if you dont point out your own
flaws, someone else will and theyre not going to be as kind about it. At least then,
you can have a discussion about the possible impact of those flaws, instead of bickering
about whether or not the flaws exist at all. You get to control the narrative.
Turn Mainstreetresearch.ca into a useful resource
As it stands, the company website is mostly used as a forum for the companys polling
results and the occasional blog post from Maggi. Those blog posts, while they could
use some additional editing, are actually a pretty effective tool to address criticism
without having to get into a Twitter fight with every interloper. To that end, it should
probably be used consistently, not once-every-six-months, as it is currently being
used.
To this end, theres nothing wrong with noting the good work of your competitors.
Voters are increasingly looking to sites like FiveThirtyEight and Real Clear Politics,
in the United States; and ThreeHundredEight and Too Close To Call, in Canada,
are becoming invaluable players in contextualizing opinion polls in the ecosystem in
which they live. To that end, pollsters, individually, dont really win anymore. A
high weighting on FiveThirtyEight means, in some ways, more than a single correct
poll.
This recommendation may be the one most likely to be ignored, but its worth
keeping in mind. The public only cares about who won the actual election, not the
polling race.
That said, and at the end of this laundry list of mistakes and stumbles,
Mainstreets sins were not that grave. They acted like data scientists:
They looked at their numbers, adjusted and tried to confirm their
findings, and stood by what they had. That, in a way, is what we expect
from our pollsters.
Maggi, specifically, poked eyes and rubbed feathers the wrong way in
such a fashion that shouldnt be repeated. It has left the company with a
black eye. But that will fade. At the end of the day, Maggi didnt kick a
dog.
The decision to launch a full review of its practises, both polling and
media, was a good one. But this is just the first step. Mainstreet, if it
wants to climb atop the heap of pollsters in Canada, needs to do better
and keep doing better.