Comparing the Pollsters and Pundits
Posted May 13, 2009 by Sacha Peter - Link
Category: Analysis
Comments (15)
So how did the predictors do?
Pollsters
The winner, by far and away, is Angus Reid Strategies. Congratulations to Hamish Marshall and the Angus Reid Strategies research team for clearly having the most accurate poll predicting the election result.
I will use the following as the 2009 election results: (based on initial count, morning of May 13th)
BC Liberals 46.0%
NDP 42.0%
Green 8.1%
Others 3.8%
| Pollster | Last Poll | LIB | %error | NDP | %error | GRN | %error | Total Error |
| Angus Reid Strategies | 6-May-09 | 44 | 2.0% | 42 | 0.0% | 10 | 1.9% | 3.9% |
| Ipsos-Reid | 7-May-09 | 47 | 1.0% | 39 | 3.0% | 11 | 2.9% | 6.9% |
| Innovative Research | 9-May-09 | 46 | 0.0% | 37 | 5.0% | 11 | 2.9% | 7.9% |
| Mustel | 6-May-09 | 47 | 1.0% | 38 | 4.0% | 12 | 3.9% | 8.9% |
| Environics | 2-May-09 | 47 | 1.0% | 36 | 6.0% | 13 | 4.9% | 11.9% |
Pundits (seat count)
Recall the predictions of the various pundits that put their reputations on the line, and staked some personal honour with the outcome of the election.
Because the judicial recount of Delta South may result in a change of results, I will calculate two scenarios. In either case, I have to call this victory for Jordan Bateman, Langley Township Councilllor, with an error of up to 2, and possibly an error of zero. I should point out that Jordan also managed to predict the 2008 federal election within an absolute error margin of 18 seats, which was much better than my error of 36 seats. So he’s on a two election winning streak (which also matches his own personal career as well!).
In the event that Vicki Huntington is declared the victor in Delta South, Jordan Bateman is tied with Trevor Loke (Green candidate in Surrey-Newton) and Vaughn Palmer (the main legislative reporter for the Vancouver Sun), all three with an error of 2.
Scenario 1:
BC Liberals 49
NDP 36
| Name | LIB | Error | NDP | Error | IND | Error | Total Error |
| Jordan Bateman | 49 | 0 | 36 | 0 | 0 | 0 | 0 |
| Trevor Loke | 49 | 0 | 35 | 1 | 1 | 1 | 2 |
| Vaughn Palmer | 47 | 2 | 37 | 1 | 1 | 1 | 4 |
| Nick Boragina | 52 | 3 | 33 | 3 | 0 | 0 | 6 |
| Milton Chan | 46 | 3 | 39 | 3 | 0 | 0 | 6 |
| Kennedy Stewart | 52 | 3 | 33 | 3 | 0 | 0 | 6 |
| Andy Shadrack | 46 | 3 | 39 | 3 | 1 | 1 | 7 |
| Sacha Peter | 45 | 4 | 39 | 3 | 1 | 1 | 8 |
| Freddy Hutter | 54 | 5 | 31 | 5 | 0 | 0 | 10 |
| Bernard von Schulmann | 55 | 6 | 30 | 6 | 0 | 0 | 12 |
| Keith Freeman | 42 | 7 | 43 | 7 | 0 | 0 | 14 |
| Eric Lanoix | 57 | 8 | 28 | 8 | 0 | 0 | 16 |
Scenario 2:
BC Liberals 48
NDP 36
Ind 1
| Name | LIB | Error | NDP | Error | IND | Error | Total Error |
| Jordan Bateman | 49 | 1 | 36 | 0 | 0 | 1 | 2 |
| Trevor Loke | 49 | 1 | 35 | 1 | 1 | 0 | 2 |
| Vaughn Palmer | 47 | 1 | 37 | 1 | 1 | 0 | 2 |
| Andy Shadrack | 46 | 2 | 39 | 3 | 1 | 0 | 5 |
| Milton Chan | 46 | 2 | 39 | 3 | 0 | 1 | 6 |
| Sacha Peter | 45 | 3 | 39 | 3 | 1 | 0 | 6 |
| Nick Boragina | 52 | 4 | 33 | 3 | 0 | 1 | 8 |
| Kennedy Stewart | 52 | 4 | 33 | 3 | 0 | 1 | 8 |
| Freddy Hutter | 54 | 6 | 31 | 5 | 0 | 1 | 12 |
| Bernard von Schulmann | 55 | 7 | 30 | 6 | 0 | 1 | 14 |
| Keith Freeman | 42 | 6 | 43 | 7 | 0 | 1 | 14 |
| Eric Lanoix | 57 | 9 | 28 | 8 | 0 | 1 | 18 |
Pundits (seats)
Five pundits predicted seats. The following is an analysis, with both Wally Oppal, and Vicki Huntington winning Delta South – either myself or Nick Boragina will have the lowest error for seat prediction:
| Pundit | Error (w/Wally) | Error (w/Vicki) |
| Nick Boragina | 7 | 8 |
| Milton Chan | 9 | 10 |
| Keith Freeman | 13 | 14 |
| Eric Lanoix | 10 | 11 |
| Sacha Peter | 8 | 7 |
Wasn’t Angus Reid ripped on by many of the readers of this site because they do online polling? Looks like phone polling is the inaccurate form of measurement now. I wonder if that has anything do to with households abandoning landlines and going cellphone only.
Thank you, thank you.
Of course, even blind squirrels find a nut once in a while.
Interesting… Ipsos is claiming victory.
http://www.ipsos-na.com/news/pressrelease.cfm?id=4388
“Ipsos Reid is the only pollster to hit the mark in both the general party election and the referendum.”
It is still possible that Charlie Wyse could lose. 23 is not insurmountable with absentee ballots. I think that those two races are the only ones in doubt, everything else is pretty much settled. In 2001, David Orcherton initially led Sheila Orr by 58 votes, but she ended up winning by 82 after a recount and absentee votes.
So it could be LIB-NDP-IND:
48-36-1
49-36
49-35-1
50-35
But I can’t blame Sacha for not making 2 more tables for the last 2 scenarios!
I think that Wally Oppal is likely going to win on the recount/absentee. People voting for Huntington absentee would have to write her name specifically, while Wally would win votes where people just write in the party. Since John Shavluk also ran in the riding as an independent, an absentee ballot with “independent” can’t be given to Vicki. In all likelihood: “BC Liberal” votes + “Wally Oppal” votes > “Vicki Huntington” votes.
My main error was under-estimating the NDP popular vote. I have faith in the complicated excel program I built to project riding by riding results, however, but I never thought I’d come in first for a provincial election! (my federal matrix is far more robust) There will be recounts however, and there seem to be at least 2 ridings won by the NDP that could go Liberal, and a Liberal riding that could go IND.
When everything is counted I’ll put the real popular vote though my matrix and see how it matches to real results.
Congrats to the winners. Given my performance, I wonder if I get the 1st draft pick next time :-). Seriously, I did not accurately predict the spread between the LIBs and the NDP. I predicted 7.3%, while the actual was 4%. Yikes, given the nature of Fist-Past-the-Post, a small error in popular vote prediction yields a big error in seat count. So that’s where my mistake comes from. I am smiling though because my result would have been a lot closer if I had gotten the popular vote right. Recall my last posting in (http://bc2009.com/2009/05/10/election-predictions-2009/#comments): “In fact a 3% spread in favour of the LIBs gives a most likely outcome of 47LIB/38NDP.” Well, the spread was actually 4% and my model predicted a 47/38 result with a 3% spread… As I said, this is a learning experience. Everyone ready for the N.S. election next month? :-)
EL
ARS did great overall, though they were somewhat off on the Interior results. I suspect that Internet polling would be more accurate in urban areas, which explains the discrepancy. Given urbanization trends, they’re definitely on the right track for the future.
Didn’t Ipsos run one or two online polls as well?
Something about the way that other pollsters work is seriously inflating support for Greens and other protest parties. The consistent story across so many federal and provincial elections is that almost everyone high-balls those parties at the expense of the traditional two parties (not just BCL/NDP, but also fed CON/LIB), and the small parties always come up short on e-day.
Interesting article on polling:
http://ukpollingreport.co.uk/blog/archives/2120
(Skipping the stuff specific to the UK, see the final two paragraphs)
The question: in a phone poll, should you prompt party choices or not?
Answer: it appears that it depends on whether the party is likely to win seats. Perennial no-seat losers get inflated when prompted, but seat-winning third parties get deflated if not prompted (UK LibDem support levels are about same as fed NDP here, though they sit between the two big parties in left-right).
It would seem, therefore, that prompting for Greens in BC phone polls tend to push up their numbers. Then again, you can imagine the righteous rage if pollsters were to only prompt for the NDP and BC Liberals.
Goes to show that polling’s more of an art than a science.
I did make a projection only using ARS and it was much closer. Liberals 46, NDP 38 Ind 1
http://bciconcoclast.blogspot.com/2009/04/what-if-angus-reid-strategy-numbers-are.html
Just wanted to point out that while Tim Ell’s calls are not listed (because I probably didn’t post them in time for the cutoff here), he is in fact the closest of all the pundits when it comes to seats and 4th or 5th on seat count.
He called Maple Ridge-Mission, Kamloops North Thompson, Saanich South and the Comox Valley incorrectly, and also Delta South if Wally’s 2-vote lead holds. But I think he deserves some credit.
Hello everyone, I am compiling my own ranking of the predictions. So far, I have only finished the riding-by-riding predictions. I am getting slightly different results from Sacha. My tables include Tim Ell’s results and he wins the riding-by-riding rankings by 2 seats! I also have Sacha and me tied. I assume Oppal got elected. Results will be updated if Elections BC changes it’s mind :-). Take a look if you like (http://www.reallifemath.ca/Site/Blog/Entries/2009/5/13_Ranking_of_Predictions_for_2009_B.C._General_Elections.html). Rest of rankings is coming soon. Cheers…
Eric – I’m not sure where you are getting my seat predictions from but they are available here – http://bc2009.com/2009/05/11/bc2009-election-prediction/
For example, you said my prediction for Burnaby-Deer Lake was BCL, but I had predicted NDP. My 8 seats wrong (with Wally) were:
Comox
Coquitlam-M
Delta South
Kamloops-NT
Saanich NATI
Saanich South
Van-Fairview
Van-Fraserview
My worst prediction was probably Comox Valley, although I gather others erred there, too. The NDP didn’t win as many of the close races as I thought– I think next time North Burnaby and Comox Valley will be considered safer from a BCL standpoint in any predictions.
I wasn’t at all surprised that the NDP smoked BCL in North Coast and Nelson-Creston. I think a lot of Rupertites who respected and voted for Herb Pond as mayor probably wondered why he was running for BC Liberals.
Figured Bennett would win E. Kootenay but the margin was surprising. Same with the North Island margin for the NDP’s Trevena. Krog won by a bit more than expected in Nanaimo..
Low turnout shall be my excuse ;)
-Keith
I’m sad to see my count wasnt included in there but nonetheless Heres was my predicition released last friday (seats)
Liberals 46
NDP 37
Green 1
Ind 1
My most innacurate predictions were in Van-False Creek were I predicted a Kettlewell victory but my most disaterous one was in West Vancouver-Cap. Sultan was Re-Elected as I expected but I expected all the other candidates (excluding libertarian) to score between 10-20% and in all reality all but platt scored less than 10%. As for Vicli well that is yet to be decided.
Apologies to everyone and especially to Sacha himself, my computations were incorrect and he did indeed predict only 8 ridings incorrectly (updates are posted on my site). Many thanks to Sacha for putting together an amazing blog on the election.