Agreeable Disagreement
Giving the other side its due improves message testing.
One of my favorite tasks as a pollster is writing a he-said, she-said question testing our candidate’s or party’s message against an opposing argument. Especially early on in a campaign, they are among the most important questions we write. Not only are they the longest questions in the survey – and therefore the most expensive on the phone – but they help lay the foundation for the campaign in the early months. Later in the campaign, those questions may help figure out how to respond to a new challenge or best pursue a new opportunity.
Here’s an example from a 2011 special election survey:
Which of the following statements comes closer to your view about Medicare:
a) The Republican plan to change Medicare is a good idea. It ensures anyone 55 or older keeps the same plan, while saving Medicare for future generations. Without any action, Medicare will go bankrupt in the next decade, requiring major tax increases or political rationing.
b) The Republican plan to change Medicare is a bad idea. It introduces private companies into the Medicare system, and ends Medicare as we know it. Rather than cutting benefits for seniors, Republicans should end corporate tax breaks to balance the budget.
The result on that question was 43 percent to 43 percent. What did that split tell us? While Republicans had a vulnerability if we didn’t respond to attacks, we did have a simple statement that got us back to even, at least in that district. Republicans could hold serve on that issue and then pivot to the criticisms of then-President Barack Obama and Democrats that had been so effective in 2010.
Balanced Statements and Real-World Framing
It’s tempting to just throw everything you have at the other side, but there are technical and substantive reasons to keep a balance in what we test. The technical reason is straightforward: in a he-said, she-said question, giving a longer argument on one side cues the respondent into which side the interviewer/pollster wants to win, and creates an opportunity for a variation of respondent acquiescence1. In the case of unbalanced wording, respondents who tend to want to be agreeable might choose the answer they think is most likely to please the interviewer.
Suppose we’re asking a question about taxes along the lines of, “Which of the following comes closer to your view: the federal government should reduce taxes, allowing the American people to save and grow the economy, or the federal government should continue collecting the same level of taxes.” Even if actual opinion on tax rates is split2, voters who might be undecided in a neutral question are likely to lean toward the more loaded option in this version. It’s not always easy to come up with the rationale for the other side – although trying to balance the budget or provide more services works well in this example – but a balanced question has a rationale for both sides or omits the rationale for both sides.
The substantive reasons are not quite as obvious, but even more important. Asking a bad question that yields a “positive” result for a client – a false positive, as it were – risks encouraging the client to dive headfirst into dangerous waters. In 2013, Heritage Action asked the following question: “In order to get President Obama to agree to at least have a ‘time out’ on implementing the health care law and its full effects, would you approve or disapprove of a temporary slowdown in non-essential government operations, which still left all essential government services operating?” Maybe not surprisingly, they found “almost 60 percent of respondents” in ten “battleground districts” supportive of a “temporary slowdown in non-essential government operations.”
We asked the following question in a 600-respondent survey of ten swing Senate states3': “Some people say that ObamaCare is so bad that an effort to repeal it should be attached to a bill necessary to keep the government running, which could lead to a temporary government shutdown. Do you think it is a good idea or a bad idea for opponents of ObamaCare to risk shutting down the government temporarily in an effort to get rid of the law?” The swing state voters said the shutdown was a bad idea by a 59 to 31 percent margin4. Of course, the shutdown happened, and Republicans paid a price in the short term before rebounding quickly in a favorable environment. Republicans would be hard-pressed to take such a misstep now and have it go unpunished this fall.
Respondents have a tendency to agree with survey questions asked in a yes/no or agree/disagree format as discussed by Howard Schuman and Stanley Presser and cited here: https://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1134&context=sociologyfacpub.
Last year, Gallup found that 46 percent of Americans think the federal income tax they pay is fair and 50 percent think it is unfair: https://news.gallup.com/poll/659003/perceptions-fair-income-taxes-hold-near-record-low.aspx.
The swing Senate states were AK, AR, GA, IA, KY, LA, MI, NC, SD, and WV, which shows how much the world has changed since 2013.
We also fielded the survey with 300 respondents in lean-Republican districts (PVI of R+4 to R+9) and with 300 respondents in swing districts (PVI of D+2 to R+3) and found similar results: voters in lean-Republican districts said it was a bad idea by a 64 to 26 percent margin and voters in swing districts said it was a bad idea by a 62 to 28 percent margin.