Democrats are Neurotic, Republicans are Close-minded, & Other Statistical Overstatements

We here at Statwing are curious folk, so we thought it’d be fun to analyze the relationship between politics and personality. Using CrowdFlower, we asked 2,000 Americans about their U.S. presidential election voting plans, as well as what psychologists call the Big 5 personality traits—Conscientiousness, Agreeableness, Neuroticism, Openness, and Extraversion. Then we took the results and pasted them into Statwing (methodology). Here’s what we found:

Controversial but Statistically Supported Overstatement #1

Obama voters are more neurotic then Romney voters.

Statwing’s output:

Scores for Neuroticism were on a 1 to 15 scale. The automatically generated histogram visualizations show how many respondents had each Neuroticism score.

This result isn’t just because Obama voters tend to be younger (Neuroticism is linked to youth). When you exclude folks younger than 40 from the analysis, the effect remains.

Controversial but Statistically Supported Overstatement #2

Romney voters are less open then Obama voters.

As in the previous example, Statwing automatically uses a t-test behind the scenes (details)

Turns out scoring relatively low on Openness is reliably associated with planning to vote for Romney. If you want to dig into this by looking at the fancy stats (p-values, confidence intervals, nonparametric tests, etc.), you can play around with the data yourself in Statwing. For what it’s worth, both of these findings are relatively consistent with existing studies of politics and the Big Five.

So. Statistically significant results! Fire up the talking heads! Orrrr wait a second. These results are statistically significant, but are they meaningful? Say you meet someone at a party who’s really neurotic, could you be assured that yelling, “How ‘bout that Obamacare!” would yield an approving high five from said neuroses-plagued partygoer? In other words, are all Obama voters neurotic and all Romney voters close-minded, or is there a lot of overlap between the two groups? To answer that, we need our new friend effect size.

Effect Size, the Beautiful but Unappreciated Stepsister of Statistical Significance

Statistical significance says, “Here’s how certain we are that something is going on here,” while effect size says, “and here’s how big a deal that something is.” It’s very common to have a statistically significant effect that isn’t terribly meaningful.

For example, let’s say you ask thousands of people who live in either Baltimore or Philadelphia how tall they are, and Philadelphians are statistically significantly taller than Baltimoreans (yup, that’s the actual word for someone from Baltimore). Then the question becomes how noticeable is the difference in heights? If Philadelphians are only 0.0001 inches taller on average—a small effect size—that’s not a very interesting finding. Alternatively, if they’re an average of 3 inches taller—a large effect size—that’s much more interesting.

Same statistical significance + different effect size = different meaning.

These curves look very similar. There’s actually a statistically significant difference, but the effect size is so trivial that you can’t really see it.


Here’s what a large effect size looks like. If this were actual data and you saw a really tall person on TV, it’d be a good guess that that person is not living in Baltimore.

 

Take a look at the screenshots for the Neuroticism and Openness analyses again, and look at effect size under the Statistical Relationship header. You’ll see a heart-warming statistical finding (not a phrase you hear every day). The relationships between personality traits and voter preference have small effect sizes. You can sort of see the differences in the curves visually, but it’s not that dramatic. So there’s not really much of a difference between Obama voters and Romney voters on these measures after all. One big American family!

Play Around in the Data

There’s a lot of fun questions to explore using this dataset. For example, do folks become agreeable or crotchety as they age? Are cat-lovers more neurotic than dog-lovers? If you really like controversy, try looking at which sex is the more conscientious, men or women?

Go ahead, geek out a bit on the dataset; nobody’s looking.

 

 

Technical Notes and Caveats

Our results were consistent with published research, but our process wasn’t publication-quality. For example, we weren’t horribly scientific about calculating who is a “Likely Voter.” There are defined ways of doing so that would have been too time-consuming to use.

Our survey is very biased towards the young Obama voters. But our doublecheck, logistic regression controlling for age, gave similar results to those presented above.

You could debate whether or not it’s appropriate to use parametric statistics for Likert-type scales. But even those who don’t believe its appropriate say it’s “common practice” to do so, so we follow that convention. Regardless, results of nonparametric tests are very similar for the above analyses (which you can see in Statwing by going to the advanced tab of an analysis and selecting “Show/Hide Ranked T-Test Results”).

Methodology

We surveyed 2,073 folks on Mechanical Turk via CrowdFlower, using a SurveyMonkey survey. CrowdFlower and Mechanical Turk are becoming commonly used tools for academic researchers looking to collect psychological survey data.

We asked each respondent to answer a 15-question survey that is known to reliably type personality traits in large sample sizes. The survey rates each personality trait—Conscientiousness, Agreeableness, Neuroticism, Openness, and Extraversion—on a 1 to 15 scale.

We also asked respondents whether they would definitely be voting for a major party candidate in 2012. For those who responded “Yes,” we asked which candidate they were planning on voting for, using a 1 (Definitely Obama) to 7 (Definitely Romney) scale. If a respondent responded with a 1 or 2 on that scale and indicated they were definitely planning on voting, then they were labeled as an Obama voter. If a respondent responded with a 6 or 7 on that scale and indicated they were definitely planning on voting, then they were labeled as an Romney voter.

We also asked people if they liked cats or dogs better. Dogs won.

Sorry, comments are closed for this post.