Virtually every month, new polls appear that suggest that white evangelicals will support Donald Trump no matter what he does, that they hold virulently anti-immigrant or anti-Muslim views, or some similar theme. Are these polls to be trusted?
There’s no question that millions of self-identified white evangelicals do fit these kinds of descriptions (pollsters often only account for white evangelicals, by the way). But there’s also reason to be cautious about the reliability of polling of all kinds. The reasons for doubts include declining response rates and the way questions are framed. The eminent Princeton sociologist Robert Wuthnow explained these problems in his 2015 book Inventing American Religion: Polls, Surveys, and the Tenuous Quest for a Nation’s Faith:
National Journal’s Steven Shepard argued that the days of accurate polling were numbered due to the proliferation of polls and the attendant difficulties of reaching people. Many of the polling companies, he feared, were abandoning the rigorous survey research methods used in the past. Huffington Post survey consultant Mark Blumenthal described the reality of polling as “they’re all pretty ugly.” Noting the increasing difficulty of distinguishing accurate research from “cr*p,” he observed that the Pew Research Center, despite being regarded as doing “rigorous, expensive, high quality” surveys, was “getting a typical response rate of 9 percent.”
None of the polls and commentaries about polls dealt specifically with polling about religion. The prevalence of polls about candidates and issues during presidential campaigns suggested that much of the concern was directed at polls focusing on those topics. And yet, many of the political polls included questions about religion and polls specifically about religion suffered from the same low response rates as other polls. Concerns previously discussed behind closed doors in academic seminars started to be aired in public.
“When I was a young sociologist at Berkeley’s Survey Research Center,” [Baylor] sociologist of religion Rodney Stark wrote in the Wall Street Journal, “it was assumed that any survey that failed to interview at least 85 percent of those originally drawn into the sample was not to be trusted.” They were not to be trusted because nonrespondents were known to differ from respondents and, for those reasons, response rates should always be reported. However, he wrote, Pew was now reporting response rates of only 9 percent, and other firms were not disclosing their response rates at all. “Only one thing is really certain,” he concluded: “those who take part in any survey are not a random selection of the population.”
Wall Street Journal “Numbers Guy” Carl Bialik, whose essays frequently emphasized journalists’ misuses of polls, cautioned readers about the difficulties in interpreting religion polls as well. He was less concerned with low response rates than with question wording. “Questions remain,” he wrote, “about how to count population by religion, and how to define those who have no religion. Different surveys use different question wording and definitions, which, combined with the huge variety of beliefs and practices, complicate researchers’ work.”
As they had done in the past, religious organizations also questioned pollsters’ results. When Pew announced that 57 percent of evangelicals thought many religions can lead to eternal life, evangelical leaders questioned the poll’s validity, especially when journalists interpreted the poll as evidence that a majority of evangelicals now believed that anybody could get to heaven whether they believed in Jesus or not. To counter the misinformation, LifeWay Research, which conducted polls for Southern Baptists and evangelical organizations, asked instead whether a person could obtain eternal life through “religions other than Christianity,” to which only 31 percent of Protestant churchgoers agreed. “The LifeWay Research finding adds quantifiable data to growing criticisms that the Pew survey was flawed,” a columnist for Religion Today wrote, “in how it asked its questions and that poor wording caused the Pew’s counterintuitive conclusions.”
Still, journalists and op-ed columnists continued to find grist in polls for sweeping conclusions about vast changes in American religion, including evangelicalism. As a writer in the New York Times concluded, “Evangelicalism as we knew it in the 20th century is disintegrating.”
Unfortunately, evangelical leaders are also prone to use polling data in order to make dire predictions and summary judgments about evangelicals. So the next time you’re tempted to get stirred up about a poll, pause for a second. Polls almost always give us a snapshot about something going on with some self-described evangelicals. But polls rarely deliver what news accounts portray them to be. They don’t offer hard information of indisputable meaning about an entire religious group.
Sign up for the Thomas S. Kidd newsletter. It delivers weekly unique content only to subscribers.
