Another Literary Digest Fiasco in the Cards?
Jimmy Breslin’s latest column in Newsday makes a pretty spectacular assertion:
Anybody who believes these national political polls are giving you facts is a gullible fool…
The telephone polls do not include cellular phones. There are almost 169 million cell phones being used in America today – 168,900,019 as of Sept. 15, according to the cell phone institute in Washington.
There is no way to poll cell phone users, so it isn’t done.
Not one cell phone user has received a call on their cell phone asking them how they plan to vote as of today.
Wow. If that’s true — and I should state up front that even people who geek out on this stuff are scrambling to find out if it is — it’s massively significant, because it implies that every telephone poll we’ve seen has been fundamentally flawed in its methodology, and therefore its results. It could even mean that some of the polling companies are setting themselves up for a replay of the infamous “Literary Digest” poll of 1936.
If you don’t know that story, the 1936 Literary Digest poll is the poll that essentially invented modern polling — not because it was a success, but because it was a spectacular failure. Here’s how it went down.
The national magazine Literary Digest ran a national survey every year asking people who they were going to vote for in the Presidential election. They conducted their survey by mailing postcards to a huge list of people — over 10 million — and more than 2 million people responded. (Modern polls usually only have 2,000 or so respondents at most.)
The theory the pollsters operated under was that, if you wanted accurate results, you had to get the largest possible cross-section of the voting public. And for a long time, it worked — the Literary Digest poll had picked the winner in every election they ran it for, from 1916 on. So in 1936, when the magazine announced that Republican Alf Landon would unseat incumbent Democrat Franklin Roosevelt, people paid attention.
Not everyone agreed with the Digest, though. One who didn’t was an upstart pollster who claimed his new methods, which he called “scientific polling”, were more accurate than the Digest’s — and that they predicted that FDR would crush Landon. Nobody took young George Gallup seriously, though.
Until election day, that is, when the votes were tallied and FDR crushed Alf Landon in one of the biggest landslides in U.S. history. The Literary Digest poll was so discredited that the magazine became the butt of jokes across the country, and actually shut down not long thereafter.
So what happened? What did the Digest do wrong? They made two major mistakes:
- First, they depended on a self-selected sample — people who chose to mail the postcards back. These were people who were more likely to be angry and anti-incumbent than the average voter. Gallup’s scientific sampling methods (which everyone uses today) didn’t have this problem.
- Second, they were based on a biased sample. Think about this — the Digest sent postcards to 10 million people. But where did it get the addresses for those people? The answer turned out to be that it got them from telephone books and automobile registrations. In 1936, seven years into the Great Depression, people who were poor or lower middle class — in other words, FDR voters — often didn’t own either a phone or a car. So they were pulling their list from a group of people that was mostly Republican to begin with.
So how is this relevant to Breslin’s assertion? If it’s true — if the pollsters are ignoring cell phones — there’s the potential for another colossally biased sample, just like the one from the Digest’s poll. There are millions of people today, mostly young people, whose only phone is a cell phone. If cell phones aren’t being polled, any telephone poll is going to omit these people — and introduce bias into its sample. (And this might help explain things like 2000, where the polls showed Bush ahead in the popular vote until Election Day, only to have Gore pull out a narrow win in the actual vote; those young voters who wouldn’t have shown up in the phone polls would have been disproportionately Gore voters.)
It’ll be interesting to see how the polling organizations respond to this. It’s certainly a serious charge.
UPDATE (9/18/2004): Here’s some follow-up material that fleshes out this issue further:
- In the comments, Sandy Smith asked the very pertinent question of where I exactly I came up with the figure of “millions” of people who have ditched their land line for a cell phone that I so breezily cited above. The only people who seem to have numbers for this are the industry association for cellphone companies, the Cellular Telecommunications & Internet Association — they’re cited, for example, in this AP story as counting 7.5 million Americans who have gone wireless-only. They have an interest in having that number be high, of course, so there may be some inflation in there, but even if you cut the number in half that’s still three million and change.
- The Electoral Vote Predictor, who asked his readers yesterday for information on whether Breslin’s charges had merit, has a long post today going into the ins and outs of telephone polling. Read it and you will know more about the subject than perhaps you ever wanted to. Executive summary: no, they don’t call cell phones; the impact of this in 2004 is less than Breslin thinks, though not small enough to discount completely; it will be more meaningful in the future than it is now.
- Pollster John Zogby has actually posted a response to Breslin on his company’s Web site. He stresses that the impact of cell phone-only households is more of a future problem for the polling industry than it is a current problem.
September 17, 2004
Millions have cell as their only phone? Do you have a source for that? I find that kind of unlikely.
Especially unlikely if I consider that likely voters tend to be older and hence not likely (get it?) to have abandoned land lines.
Of course, my brother has, but he’s a geek, and nearly had to abandon it when service in his house got really bad. In most cities, it’s worse.
September 17, 2004
Here’s an article from Wired News that estimated the number of cell-only consumers in the US as high as 7.5 million as of a year ago:
The numbers are a little different depending on who you ask, but it seems clear that somewhere around 3-5% of the population has abandoned land-line telephony altogether at this point.
September 18, 2004
I keep going back and forth on this. I’m guessing that cell-phone-only types will be a) young and b) urban, which would mean they are more Blue State types, which would help Kerry.
But since blue-state types tend to be in places that go pretty solidly Democrat, they wouldn’t make as much of an electoral difference. Also, younger types tend not to be as good with the actually remembering to vote.
“Dude! We totally spaced on the election again! I thought it was Super Wednesday.” …which is pretty much the same reason we don’t have more liberal marijuana laws.
But it does sound like a flaw. I don’t know how you get around it until prices for cell service become as flat as land lines.
Which is about the time you’ll see me with a cell phone.
September 21, 2004
I’m wireless only. Make that 7.5 million plus one.
Actually, I know of a few others who are wireless only, my brother, my sister, a few co-workers….
September 22, 2004
I’m also wireless only…No point in having a landline since I live alone. I’m youngish, suburban, and planning to vote to Bush.
September 22, 2004
Ann, you’re planning to vote for Bush?!?
Oh, man… I’ve got less than 60 days to get you to see the light 🙂
September 23, 2004
Are you now secretly a little mad at yourself for trying to get me to vote in the first place? ;>
September 23, 2004
Nope. Everybody should vote, even if their vote is WRONG WRONG WRONG 😛