The margin of stupid

Every so often, I see a news story or tweet hyping the fact that a modest but non-negligible percent of Americans said some crazy or horrible thing in a survey. Here are two examples:

1. "A chilling study shows how hostile college students are toward free speech"

The most chilling findings, however, involved how students think repugnant speech should be dealt with...It gets even worse. Respondents were also asked if it would be acceptable for a student group to use violence to prevent that same controversial speaker from talking. Here, 19 percent said yes. 

Update: It turns out this particular survey was a badly designed piece of crap. I'm not particularly surprised...

2. "Millennials are just about as racist as their parents"

Racial slurs that have cropped up chants, e-mails and white boards on America's college campuses have some people worried about whether the nation's diverse and fawned-over millennial generation is not as racially tolerant as might be expected. 

 

So, from these two examples -- both of them in the Washington Post -- I'm supposed to believe that Millennials are a bunch of unreconstructed racists, except for the ones who go to college, who are a pack of intolerant leftists. 

It seems to me like there's something inherently suspicious about judging a group of people based on sentiments expressed by only 15 or 20 percent of those people. But beyond that, there's another problem here - the problem of whether we can really trust these surveys. 

Surveys give a false sense of precision, by reporting a "margin of error" (confidence interval). But that confidence interval comes purely from the fact that the sample is finite. It does not capture systematic error, like selection bias (Are the people who answer this survey representative of the population being sampled?). And it definitely doesn't capture the errors people themselves make when responding to surveys.

When I did happiness survey research with Miles Kimball, there was always the nagging question of whether people are really able to know how happy they are. Of course, the whole question of what "happiness" should mean is a difficult one, but presumably there are some neurochemical tests you could do to determine how good someone feels, at least relative to how they felt in the past. How well do survey responses reflect this "true" emotion? Do people in different countries have cultural pressures that make them respond differently? Do Americans feel the need to say they're happy all the time, while British people would be ashamed to admit happiness? And are people measuring their happiness relative to yesterday, or to their youth, or to how happy they think they ought to be?

These errors were things that we lumped into something we called "response style" (psychologists call it response bias). It's very very hard to observe response style. But I'd say we can make a pretty good guess that Americans - and possibly everyone - do a lot of random responding when it comes to these sorts of surveys.

For example, a 2014 survey reported that 26 percent of Americans said that the sun goes around the Earth. 

Now, maybe there are a bunch of pre-Copernican geocentrists out there in America (there certainly are the flat-earthers!). Or maybe people just don't think very hard about how they answer these questions. Maybe some people are confused by the questions. Maybe some are trolling. 

Whatever the cause, it seems like you can get 20 to 25 percent of Americans to say any ridiculous thing imaginable. "Do you think eating raccoon poop reduces the risk of brain cancer?" "23 percent of Americans say yes!" "Would you be willing to cut your toes off with a rotary saw if it meant your neighbor had to do the same?" "17 percent of Americans say they would!" Etc.

You can also see this just from looking at some of the crosstabs in the first survey above. 20 percent of Democrats and 22% of Republicans say it's OK to use violence to shut down speakers you don't like. This sounds kind of nuts, given the panic on the right over lefty violence against campus speakers. Why would Republicans even more likely than Democrats to condone this sort of violence? It makes no sense at all...unless you can get ~20 percent of Americans to say pretty much any ridiculous thing on a survey. 

I call this the margin of stupid. Unlike the margin of error, it's not even a roughly symmetric error -- because you can't have less than 0% of people give a certain answer on a survey, the margin of stupid always biases surveys toward showing some non-negligible amount of support for any crazy or stupid or horrible position. 

Whenever you read a survey like this, you must take the margin of stupid into account. Yes, there are Americans who believe crazy, stupid, and horrible things. But dammit, there aren't that many. Next time you see some poll breathlessly claiming that 21 percent of Americans support executing anyone whose name starts with "G", or that 18 percent of Millennials believe themselves to be the reincarnation of Kublai Khan, take it with a grain of salt. It's a lot easier to give a stupid answer on a survey than to actually truly hold a nuts belief.

Sadly, the margin of stupid also probably applies to voting.