The past few weeks have been a rollercoaster ride for political poll junkies. Which polls do we believe? Without some filter, the wild swing of numbers is likely to give many amateur pundits agita. A review by the National Council on Public Polls of polls from the 2002 gubernatorial and senatorial races found that 22 of 159 polls (13.8%) called the wrong winner in the last two weeks of the elections. However, the week before the 2000 presidential election, state polls showed that the election would hinge on Florida, as it later did.

Polls are based on the principle that an unbiased random sample will accurately reflect the total population. Typical National Surveys are conducted by telephone with 1,000-1,500 people giving a 95% confidence level of a margin of error of only 2.5-3.5%. Non-response rates have been rising in recent years, possibly leading to more polls based on a ‘self-selected’ sample and introducing bias. However, a Pew Research study comparing a rigorous and standard survey (response rate of 60.6% vs. 36.0%) showed little difference among a wide variety of topics, though the higher response rate resulted in a larger sampling of white, educated, wealthier people.

After the raw data is collected, it is weighted in various ways to make it more representative of the entire population. Typically, more women are sampled, so census data is used to reduce the weight of their responses while simultaneously weighting up male responses. Similar weights are applied for other demographic quantities. Unfortunately, most polls do not reveal what weighting scheme was used so there is little oversight in the process. Polls will also attempt to make predictions based on “likely” voters. Often they ask a series of questions designed to eliminate those who may not vote. Often included are questions like “Did you vote in the last election?” and “Do you know where to vote in your district?” Thus younger voters may easily be excluded, or in such a heated election as this, other normally ambivalent people. A dangerous practice is to weight the data so that the ratio of Republicans, Democrats, and Independents is about the same as previous elections. Polls ask “what party do you identify yourself with?” a much more fluid and fluctuating quality than who you are registered with. This weighting may also not reflect reality if one party mobilizes an unusual number of voters.

Figure 1 presents a quantification of bias in polling organizations this year, assuming that all have some consistent bias due to methodology and that the underlying support for the candidates follows a smoothly changing function¹.

Question wording and order are probably one of the most serious factors affecting poll results. So reading the questionnaire and comparing responses to similar questions is of the utmost importance. To read more about polling methodology and several meta-analyses of the current race check the references below.

References:

¹ Poll of Polls.

² National Council on Public Polls

³ Jon Krosnick, Professor of Communication, Political Science, and Psychology, Stanford University.

⁴ Voss, D.S. A. Gelman and G. King 1995. A Review: Preelection Survey Methodology: Details From Eight Polling Organizations, 1988 and 1992. The Public Opinion Quarterly. 59(1): 98-132

⁵ Meta-analysis of state polls.

Related Articles:

• RUFP Précis
• Political Science