Support for same sex unions hasn't grown during Obama's first term

Journalists do the darnedest things with statistics.

There is a story titled Poll: Most Americans support same-sex unions on CBSNews.com today.

Mr. Reals comes up with that conclusion by lumping together the two categories, 1) Same sex couples should be allowed to marry; and 2) Same sex couples should be allowed civil unions, and noting that the sum of those proportions exceeds the proportion of respondents who say same sex couples should have no legal recognition.

If you look at things that way, it's hard to see what the story is. The following picture of a table Mr. Reals includes in his post indicates that those who oppose any legal recognition of same sex couples were outnumbered even back in 2004.

Anyone who actually paid attention in Intro Stats knows that the point estimates matter very little. What matters is whether the change can be explained by chance or not.

Given that all surveys mentioned in the table seem to have used simple random sampling, we can assume they were independent of each other. The CBS article mentions that the number of respondents to the current survey was 615. We are not given any information indicating the number of respondents to the March 2009 survey which was conducted right after President Obama took office. So, absent any further information, I am going to assume the sample sizes were the same in March 2009 and the most recent survey.

If you have some basic arithmetic ability, you can apply the formula for the pooled two-proportion z-test on this Wikipedia page.

In keeping with standard statistics (the kind that allow Mr. Reals to say things like The error due to sampling for results based on the entire sample could be plus or minus four percentage points. The margin of error for the sample of registered voters is four percentage points. The error for subgroups is higher), we have:

H₀: P₂₀₀₉ = P₂₀₁₂
H_A: P₂₀₀₉ < P₂₀₁₂

In words, the null hypothesis states that there is no difference between the proportion of the population supporting same sex unions marriage between 2009 and 2012. We are testing against the alternative that the proportion of the population supporting same sex unions increased during President Obama's first term.

If the difference between the two poll results is such that the chances of it being produced due to random sampling variation from a population with a given, constant proportion are low enough, we'll reject the null in favor of the alternative.

Now, by Mr. Reals' definition, the support for same sex unions was 60% in March 2009. In the most recent survey cited by Mr. Reals, the support was 62%. That is, the numerator for the test statistic, i.e. P₂₀₁₂ – P₂₀₀₉, is 0.02; 2 percentage points.

The standard error of the difference is:

Therefore, the test statistic is z^☆ = 1.44.

The right tail probability of the standard normal distribution for z^☆ = 1.44, i.e. the p-value of our one tailed test, is 7.5% which is greater than α = 5%.

That is, we failed to reject the null hypothesis that there is no difference in the proportion of support for same sex unions; i.e. there is no statistically significant difference between the support for same sex unions between March 2009 and now.

That doesn't mean there are no differences between then and now.

One subtle but interesting difference is in the number of respondents who are not in any of the categories mentioned in Mr. Reals' story. In 2006, the three categories added up to 97%. In the most recent poll, they added up to 95%. Considering that 90 million votes were cast in the 2010 midterm elections, that 2% represents a non-negligible number of voters, depending on their geographic distribution.

In the mean time, the important thing in the poll is that, over time, while the proportion of those who want some legal recognition of same sex unions has remained somewhat constant, the composition within that group has changed in favor of conferring marriage rights. However, 38% is still not "most," and, in this case, the Mr. Reals' desire to proclaim "Most Americans support same-sex unions" has caused him to neglect the more important story.