2002 Election Polls Review
FOR IMMEDIATE RELEASE: THURSDAY, DECEMBER 19, 2002
NATIONAL COUNCIL ON PUBLIC POLLS
POLLING REVIEW BOARD
ANALYSIS OF THE 2002 ELECTION POLLS
A review of the 159 Governor and U.S. Senate polls reported by the media in 2002 shows a very good performance for most polling organizations. The average candidate error for all polls was 2.4 percentage points. 84% of the polls differed from the election outcome by less than their theoretical margin of error.
Immediately after these elections there were reports in the media that the polls had been very seriously inaccurate, and that the future of polling was threatened by declining response rates. We note, however, that these accounts were based on the performance of just a few polls and that the conclusions reached in those accounts were based on too few cases.
Our analysis shows that the overall performance of the polls was nothing like those reports had suggested. So we conducted a review of every published poll we could find that was conducted during the last two weeks of the campaign and that was public before the election. We found more than 25 times the number of polls in those news stories.
Most organizations conducted polls in only one or two states. They accounted for 95 of the 159 polls. Five organizations conducted 64 polls. These organizations worked in three or more states. Mason-Dixon did the greatest number. They conducted 23 polls in 16 states. Only one of their polls had the wrong candidate winning. Zogby International did 17 polls in 12 states, and had 5 incorrect winners. Zogby also conducted partisan polls, which are not included here even though they were publicly available. Research 2000 did 13 polls in 9 states, and had 2 wrong winners. The Gallup Organization did 7 polls in 4 states with 1 mistake. And Quinnipiac College in Connecticut conducted 4 polls in as many states. None had the wrong candidate winning. The 95 polls from all other organizations had the wrong candidate winning in 12 of the contests. In only two of the 21 races where the wrong winner was named was the mistake due to a small lead in the poll for the losing candidate when the other candidate won a very close race.
|Polling Organization||Number Of Polls||Average Error On Candidate||Races With Wrong Winner|
The candidate error reported here is half the error on the margin between the top two candidates. For example, if a race was won by 55% to 45% the margin is 10 percentage points. If a poll reported a lead of only 47% to 43% with 10% undecided, the 4-point margin in the poll would be off by 6 percentage points. The candidate error in this case was counted as 3 points, half the error on the margin. No method of judging the error works perfectly. This approach to counting the error was used in order to avoid an arbitrary decision about how to allocate the 10% undecided. In all 159 polls the pollsters reported some percentage as undecided, leaving up to those using the poll to make their own interpretation of the outcome. Under the circumstances we decided to make no interpretation about the undecided and to judge the poll by the margin between the two leading candidates. Other evaluations of poll performance based on other methods can produce different conclusions. This analysis seeks only to gauge the accuracy of these polls final horse-race measurements. Other necessary components of good polling including rigorous methodology and a commitment to measure the full range of broader election issues and voter concerns are not part of this evaluation.
We examined two other characteristics of these polls. First, we looked to see if there was any relationship between the size of the winning margin in the election and the size of the error in the poll. There is no relationship. Elections won by small margins had errors that were just as large as those won by wide margins, and vice versa.
A few races stood out as problem races. These races may have continued changing right up to the election. It is not likely that all polls could be off the mark, although this is a possibility. These are the races with large polling errors: AL-Sen, CO-Sen., GA-Sen., PA-Gov., RI-Gov., VT-Gov.
We also looked at the margin of error (error due to sampling) for each poll and compared it to the candidate error. In 25 of the 159 polls (16%) the candidate error was larger than the statistical margin or error on the candidate. For the other 84% of the polls the candidate error did not exceed the margin of error. The margin of error should be taken into account when interpreting a poll. The lesson here is that one should only conclude that a poll is pointing to an election winner when the lead exceeds two times the margin of error. (The margin of error is for each candidate; therefore the error on the margin is double the margin of error on a candidate.) Even this is not infallible.
HOW POLLING ERRORS WERE COMPUTED
- This analysis is based on 159 publicly released polls which were either sponsored by media organizations or conducted by academic institutions. Both the Polling Report and Hotline were used as sources as a check for consistency to confirm accuracy and to obtain as complete a list as possible. Some polls were verified in the publication that carried them.
- Only final polls covering races for Governor and U.S. Senate were included. The last poll that appeared was considered final.
- Timeliness was an issue. Our standard was that all or most of the interviewing was conducted after October 20.
- The candidate error reported here is half the error on the margin between the top two candidates. The error was calculated by subtracting the margin between the top two candidates in a poll from the margin between the same candidates in the election.
For example, if a race was won by 55% to 45% the margin is 10 percentage points. If a poll reported a lead of only 47% to 43% with 10% undecided, the 4-point margin in the poll would be off by 6 percentage points. The candidate error in this case was counted as 3 points, half the error on the margin. No method of judging the error works perfectly. This approach to counting the error was used in order to avoid an arbitrary decision about how to allocate the 10% undecided. In all 157 polls the pollsters reported some percentage as undecided, leaving up to those using the poll to make their own interpretation of the outcome. Under the circumstances we decided to make no interpretation about the undecided and to judge to poll by the margin between the two leading candidates. The resulting difference was divided in half to get the error per candidate.
- Most percentages reported for polls were whole numbers. For the few that had decimals we rounded them to whole numbers. The difference between the top two candidates in each election also was rounded to two digits before polling errors were computed.
- Many states with elections for Governor or U.S. Senate did not have polls meeting the criteria of public release or timeliness. States with elections for Governor without such polls were: NV and OK. States without Senate polls were: AK, DE, KS, KY, MA, MS, OK, VA, WV and WY.
For more information about this and other polling issues, contact the NCPP Polling Review Board Members.