Liberty Maven

As an avid political junkie, and a statistician, I’m particularly fascinated by polling. The trends of the most recent polls, and the speculation on cause and effect (Did Bachmann’s debate performance really lift her to front-runner status overnight?) are intriguing, if not in any way gratifying.

My background lends itself to picking out oddities, or outliers, as well. When a poll released by CNN and Opinion Research Center (ORC) in late May showed political afterthought Rudy Giuliani leading the race for the GOP’s presidential nomination in 2012, I thought it was a bit bizarre. Here we had a candidate who failed to register a single victory in the party’s 2008 primaries, despite spending more than Mike Huckabee and Ron Paul – two candidates who soundly defeated him at the polls – combined. A candidate who had registered at 9% in two independent polls released in the previous month – a 9% number that was well outside ORC’s 4.5% margin of error. Were they saying that Giuliani had received at least a 3% bump from April to May? Based on nothing except, perhaps, the death of Osama bin Laden?

Fast forward to today, when a new poll of the 2012 GOP race was released, showing Rick Perry with a commanding lead at 27% as compared with Mitt Romney’s 14%. Perhaps most shocking in this most recent poll was a precipitous drop in declared support for Texas Congressman Ron Paul, who received just 6% as compared with 12% in a poll by the same company and news agency not even three weeks prior – a drop that falls outside the margin of error of 4.5% for the Republican sample. Did Ron Paul – who received a mixed bag of media attention, from his near-victory at the Ames Straw Poll to his jabs at FEMA on the eve of a hurricane to the stories of his being a non-story – really experience a statistically significant drop in support over these past three weeks? I had to check the cross-tabs.

The more I looked, the more suspicious I became. First of all, ORC decided to exclude cross-tabs on demographics and sample sections which were statistically insignificant – by my examination, it looks like anything with a margin of error greater than ± 8.5% was struck and replaced with “N/A”. This is an interesting methodology, and one that I’ve not seen any other major national pollster take – most report the cross-tabs without margin of error at all. However, having those MOE’s is vital to conduct the next analysis, since ORC apparently didn’t find it necessary to specify how they broke down their “Republican” sample into more detailed numbers.

At the top of Page 2 of the release, ORC claims to have a sample of 467 Republicans (registered? Likely voters? The world may never know.), to which they attribute a ± 4.5% margin of error. Using some quick statistical math (the binomial approximation to the normal distribution), I find that they are using a power of 0.62, which I used to perform subsequent calculations.

Scrolling down to the political party and political ideology cross-tabs (Page 8), we find a breakdown of the 467 polled, excluding the aforementioned Giuliani as well as former vice-presidential candidate Sarah Palin. In this sample, “Republican” is assigned a ± 6.0% margin of error, and “Independent” is assigned a ± 7.0% margin of error. Using the 0.62 power of sample attained above, this yields 264 Republicans and 194 Independents for a total of 458 – slightly less than the 467 actually polled. We’ll assign the remainder proportionally, and assume 270 Republicans and 197 Independents are in the sample.

Remember the ± 8.5% cutoff that ORC is using to determine which sections they will not report? In the sample of 467, again using the 0.62 power of sample, this equates to a sample of about 132 – that is, if the cross-section of respondents in a category falls below 132, the category will be reported as “N/A”.

Let’s look at the section of cross-tabs just below the political parties and political ideologies. Here we have a breakdown of geographic regions and household location demographics. Whoa! Only the “South” region registers as statistically significant to ORC, and only “Suburban” households. For now, we’ll take for granted that this sample of “Republicans” is reflective of a national voting populace. Using the ± 7.0% margin of error on the “South” region, we estimate that 194 of the 467 respondents lived in the “South” region (however ORC defined it). That’s 41.5% of the sample – again, we won’t dispute this selection of the sample.

In the “South” region, Ron Paul received 10% support among those polled. Given that it makes up such a notable portion of the total sample, this earns Ron Paul roughly 4.15% of the total vote in the sample, regardless of what respondents say in other regions. Yet, Paul receives just 6% of the total vote in the sample. Doing the quick calculation, this means that Paul must have received just 3.16% of the vote in the other 3 regions – West, Midwest, and Northeast – combined.

As far as differences between these regions are negated, that’s effectively a national poll with Paul receiving just above 3% of the vote. The last national poll in which he registered so low (without the presence of Giuliani)? An ABC News/Washington Post poll dated April 14-17, which had Paul at 2%, and a full 50% of respondents offering Other/None/Undecided.

Needless to say, this result falls well outside the expected range of Paul’s support across the rest of the country. In fact, the most recent Gallup poll (which had Paul at a 13%, admittedly on the high end of his polling range) showed that Ron Paul actually performed better in the West, Midwest, and Northeast than he did in the South:

No matter which poll you believe, one thing is clear: Both of them can’t be correct. However, based on my statistical experience, as well as my political savvy built over the past decade, I’d say that any poll which puts Paul around 3% anywhere almost discredits itself. After all, Paul’s supporters haven’t been the small, devoted group of internet spammers the media made them out to be, in light of their successes in the past few months.

One final thought: Any polling company which is willing to be hired to run an unabashedly biased headline such as: “Keep the EPA Strong – Let Them Do Their Job” is probably not the company you want to rely on for a fair and unbiased poll.