Order in Questions

Surveys are funny things and all kinds of subtle changes can affect results. Or not. Or not in the direction you expect. Take for example this test included in the ANES 2016 pilot study. Half of the respondents were randomly assigned to get this question:

Generally speaking, do you usually think of yourself as a Republican, a Democrat, an independent, or what? 

And half received this question:

Generally speaking, do you usually think of yourself as a Democrat, a Republican, an independent, or what?

See the difference?

One lists Republican first, the other Democrat first. Does it make a difference? You'd expect so, right? You'd expect offering Republican first would increase the odds, slightly, of that answer being given, and the same for offering Democrat first.

The results are puzzling.

	Democrat Asked First	Republican Asked First
Are you a …
Democrat	37.9%	38.9%
Republican	25.7%	20.9%
Independent	30.2%	33.2%
Something Else	6.2%	7.0%

As you can see, just eyeballing the data above, fewer respondents say "Republican" when that's asked first as a response alternative. That's a 5 percentage point difference. It's weird, especially as the answers of "Democrat" seem unaffected by the test (1 percentage point difference and, again, in the opposite direction I expected).

Asking Republican first seems to increase (or asking Democrat first seems to decrease) the number of those choosing to identify themselves as "Independent."

My working theory? I wish I had one. I thought maybe it was a "leaner" effect, that somehow those who say Independent but lean one way or the other shifted because of how the question was worded, but my quick-and-dirty analysis says that's not the case.

It's a small effect, true, but 5 percentage points in party identification can sometimes effect how a survey is weighted, which in turn can influence its final results.