Political Polling is Broken (And How Mushiness Can Fix It)

Advertisement: Statwing is an easy-to-use tool for analyzing survey results and other business data. Click through the visualization below to see the analysis in Statwing (and play around with the rest of the dataset).

Daniel Yankelovich

Daniel Yankelovich, the “founding father of public opinion research.”
Photo by Matthew Septimus, courtesy of strategy+business.

Daniel Yankelovich, the “founding father of public opinion research”, was bothered that political polling did not take into account the conviction with which respondents held a position.[1] Take the question “Should the U.S. ban assault weapons, Yes or No?” Someone who felt very strongly about the issue was given the same weight by a pollster as someone who barely cared. 

So in the 1970′s Yankelovich developed a set of questions that assessed how strong someone’s conviction was on an issue. He presented it to one of his clients, Time Magazine, who dubbed it the “Mushiness Index”. Says Yankelovich, “They were quite enthusiastic. And then they found, when the reporters and writers sat down to write the stories, they found … it slowed up the storytelling and, as a result, they just didn’t use it.”[2] Little has changed since.

We decided to run an experiment to see if the Mushiness Index could add anything to our understanding of contemporary political attitudes. We reached out to a couple hundred Americans via CrowdFlower, and asked them to take a survey we created on Kwiksurvey.[3] They were asked about their attitudes on increasing funding for border control, U.S. Postal Service Saturday delivery elimination, fracking, marijuana legalization, and Internet neutrality. For each of those questions we asked both their position and how important that issue was to them, on a 1 to 7 scale.[4],[5]

For three of those questions, respondents on one side of the issue tended to feel more strongly than respondents on the other side of the issue. For example, only 32% of our (unrepresentative) sample thought the United States should spend more on border security. But those who did feel the U.S. should spend more tended to feel much more strongly about it than those who did not.[6]

Click the image below to see the statistical test results, and to explore the rest of the dataset.

Raising Border Spending, Mushiness-weighting

The 57 respondents who answered “Yes” to whether or not border spending should be increased felt more strongly about the issue than the 118 respondents who answered “No”

We weighted responses by the respondents’ degree of conviction, counting a respondent more heavily if they said the issue was extremely important to them. When we did, we found 41% support instead of 32% support for higher border spending (see endnote for exact methodology).[7] Using mushiness-weighting, you might say that America is collectively 41% in favor of spending more on border security, even if only 32% of individual Americans are in favor of such. Applying the same method to the other questions we asked yields the following:

Mushiness-weighted support for political issues

Some issues have a larger disparity between supporter and opponent mushiness than others

In the polling world, a 9-point difference is important, and can be the difference between a headline reading “The majority of Americans agree that…” and a headline reading the opposite. And the disparity between the unweighted and mushiness-weighted result is itself an interesting datapoint, an indicator of the relative conviction of people on either side of the issue. When we ignore the conviction individuals have in their views, we miss a lot of the story.

What other lessons should we take from this? First, pollsters include questions regarding the strength of the conviction of those they’re polling. Second, we should take poll findings that don’t account for strength of conviction with a grain of salt. Lastly, everyday non-political surveys should also account for conviction—which can be as simple as allowing respondents to answer on a one to seven scale instead of with a binary Yes/No. Mushiness is an important part of how the world thinks and feels, so it’s time we start accounting for it.

 

Discussion on Hacker News
Statwing Front Page

Notes

[1] The “founding father of public opinion research” description comes from Bill Moyers. Courtesy of strategy+business.

[2] Hat tip to the excellent This American Life story “The Facts Don’t Matter“.

[3] The sample was admittedly unrepresentative (e.g., it leaned young and Democratic, among other things). Our goal here was only to demonstrate whether people on different sides of important issues have meaningfully different degrees of conviction about it, not to actually come up with unassailable polling data.

[4] Our survey questions:

  • “Should the United States increase government spending on security measures and enforcement at U.S. borders?” adapted from Gallup (68% support)
  • “As you may know, the U.S. Postal Service is anticipating billions of dollars in losses this year. Should the post office eliminate residential mail delivery on Saturdays to save money?” adapted from Gallup (63% support).
  • “A process known as hydraulic fracturing, sometimes called fracking, is used to drill for oil and natural gas in shale oil reserves. Do you favor or oppose the use of fracking to produce more oil and natural gas in this country?” adapted from Rasmussen (57% support).
  • “Do you think the use of marijuana should be made legal?” adapted from Gallup (50% support)
  • “Should internet service providers (like Comcast) be allowed to give different priority to different types of Internet traffic (for example, giving normal Internet traffic a higher priority than peer-to-peer filesharing traffic)?”
  • For each of the above we individually asked “How important to you personally is this issue?”

[5] We used one question to measure conviction, where the original Mushiness Index used four. It’s worth noting that asking someone how strongly they feel is problematic; it’d be better to examine revealed preference indicators like whether someone has written their representative about the issue or tried to convince a friend of their view.

[6] Again, 32% isn’t the actual number. Gallup’s more reliably sampled poll says that number is 68%.

[7] We normalized responses to “How important to you personally is this issue” by subtracting 1 and then dividing by 6 to get a 0 to 1 scale of how strongly the respondent felt (the math works out the same if you don’t divide, but that felt cleanest). The weights of the 57 “Yes” respondents summed to 28.1; those of the 121 “No” respondents summed to 39.7. Where 57/(57+121) yields 32% of the unweighted sample supporting higher spending, 28.1/(28.1+39.7) yields 41% of the mushiness-weighted sample supports higher spending.

Sorry, comments are closed for this post.