Another Literary Digest Fiasco in the Cards?

Posted on September 17, 2004

Jimmy Breslin’s latest column in Newsday makes a pretty spectacular assertion:

Anybody who believes these national political polls are giving you facts is a gullible fool…
The telephone polls do not include cellular phones. There are almost 169 million cell phones being used in America today – 168,900,019 as of Sept. 15, according to the cell phone institute in Washington.
There is no way to poll cell phone users, so it isn’t done.
Not one cell phone user has received a call on their cell phone asking them how they plan to vote as of today.

(emphasis mine)

Wow. If that’s true — and I should state up front that even people who geek out on this stuff are scrambling to find out if it is — it’s massively significant, because it implies that every telephone poll we’ve seen has been fundamentally flawed in its methodology, and therefore its results. It could even mean that some of the polling companies are setting themselves up for a replay of the infamous “Literary Digest” poll of 1936.

If you don’t know that story, the 1936 Literary Digest poll is the poll that essentially invented modern polling — not because it was a success, but because it was a spectacular failure. Here’s how it went down.

The national magazine Literary Digest ran a national survey every year asking people who they were going to vote for in the Presidential election. They conducted their survey by mailing postcards to a huge list of people — over 10 million — and more than 2 million people responded. (Modern polls usually only have 2,000 or so respondents at most.)

The theory the pollsters operated under was that, if you wanted accurate results, you had to get the largest possible cross-section of the voting public. And for a long time, it worked — the Literary Digest poll had picked the winner in every election they ran it for, from 1916 on. So in 1936, when the magazine announced that Republican Alf Landon would unseat incumbent Democrat Franklin Roosevelt, people paid attention.

Not everyone agreed with the Digest, though. One who didn’t was an upstart pollster who claimed his new methods, which he called “scientific polling”, were more accurate than the Digest’s — and that they predicted that FDR would crush Landon. Nobody took young George Gallup seriously, though.

Until election day, that is, when the votes were tallied and FDR crushed Alf Landon in one of the biggest landslides in U.S. history. The Literary Digest poll was so discredited that the magazine became the butt of jokes across the country, and actually shut down not long thereafter.

So what happened? What did the Digest do wrong? They made two major mistakes:

• First, they depended on a self-selected sample — people who chose to mail the postcards back. These were people who were more likely to be angry and anti-incumbent than the average voter. Gallup’s scientific sampling methods (which everyone uses today) didn’t have this problem.
• Second, they were based on a biased sample. Think about this — the Digest sent postcards to 10 million people. But where did it get the addresses for those people? The answer turned out to be that it got them from telephone books and automobile registrations. In 1936, seven years into the Great Depression, people who were poor or lower middle class — in other words, FDR voters — often didn’t own either a phone or a car. So they were pulling their list from a group of people that was mostly Republican to begin with.

So how is this relevant to Breslin’s assertion? If it’s true — if the pollsters are ignoring cell phones — there’s the potential for another colossally biased sample, just like the one from the Digest’s poll. There are millions of people today, mostly young people, whose only phone is a cell phone. If cell phones aren’t being polled, any telephone poll is going to omit these people — and introduce bias into its sample. (And this might help explain things like 2000, where the polls showed Bush ahead in the popular vote until Election Day, only to have Gore pull out a narrow win in the actual vote; those young voters who wouldn’t have shown up in the phone polls would have been disproportionately Gore voters.)

It’ll be interesting to see how the polling organizations respond to this. It’s certainly a serious charge.

UPDATE (9/18/2004): Here’s some follow-up material that fleshes out this issue further:

• In the comments, Sandy Smith asked the very pertinent question of where I exactly I came up with the figure of “millions” of people who have ditched their land line for a cell phone that I so breezily cited above. The only people who seem to have numbers for this are the industry association for cellphone companies, the Cellular Telecommunications & Internet Association — they’re cited, for example, in this AP story as counting 7.5 million Americans who have gone wireless-only. They have an interest in having that number be high, of course, so there may be some inflation in there, but even if you cut the number in half that’s still three million and change.
• The Electoral Vote Predictor, who asked his readers yesterday for information on whether Breslin’s charges had merit, has a long post today going into the ins and outs of telephone polling. Read it and you will know more about the subject than perhaps you ever wanted to. Executive summary: no, they don’t call cell phones; the impact of this in 2004 is less than Breslin thinks, though not small enough to discount completely; it will be more meaningful in the future than it is now.
• Pollster John Zogby has actually posted a response to Breslin on his company’s Web site. He stresses that the impact of cell phone-only households is more of a future problem for the polling industry than it is a current problem.