About Those Religion Surveys …

The latest edition of First Things magazine, soon to be available online, contains an important piece by Princeton sociologist of religion Robert Wuthnow, “In Polls We Trust.” Actually, it’s one of the most important pieces on American religion I’ve read in quite a while. Not for what it says about American religion, necessarily. Wuthnow’s piece is important because of what it says about the polls on which everyone, academics included, rely for insights on American religion.

It’s hard to overestimate the importance of these surveys. Scholars pore over the results to ascertain trends, and, on the basis of those trends, to evaluate the state of American institutions: churches, government, courts. For example, the much touted rise of the “Nones,” the percentage of Americans with no religious affiliation, has implications for our First Amendment jurisprudence. The fewer Americans who identify with institutional religion, the weaker we can expect First Amendment protections to get. Or so some scholars, myself included, have argued.

Of course, everything turns on the accuracy of the surveys. Most of us, not being statisticians, more or less take them on faith. If Wuthnow is right, though, our faith is misguided. He points out that many surveys of American religion have serious methodological flaws. For example, religion does not always lend itself to straightforward yes/no questions of the sort surveyors ask. In addition, pollsters sometimes fail to account for regional and racial variations.

Most important, response rates are very low. The typical response rate nowadays is about nine or 10 percent, and rarely exceeds 15 percent. “In other words,” Wuthnow writes, “upwards of 90 percent of the people who should have been included in a poll for it to be nationally representative are missing. They were either unreachable or refused to participate.” With such poor response rates, it’s hard to know what the polls reveal about religion in America. This problem is compounded by the fact that the media present the results as accurate representations of what Americans believe—a misimpression that the polling industry, now worth a billion dollars a year, is understandably reluctant to correct—and by the fact that most of us “are unlikely to wade through obscure methodological appendices to learn if the response rate was respectable or not.”

Consider the rise of the Nones, for example. Maybe we really are seeing an explosion in the number of Americans without a religious affiliation, as these surveys suggest. But maybe we aren’t. Maybe the number of Nones is actually much lower. Maybe the number is much higher. Wuthnow’s point is, it’s hard to know on the basis of flawed polls. Now, to be sure, there are other indications that organized religion is declining. Some churches keep membership records; these are harder numbers, and they show that some churches are experiencing declining memberships. Still, one has to be a little careful about declaring trends on the basis of limited information.

The inaccuracy of the polls is more than just an academic matter, because polls may actually help drive social change. It’s human nature to want to follow the crowd. If you think that Nones are the wave of the future, you’re more likely to call yourself one; if you think that Church is a dying institution, you’re more likely to leave. On the basis of these polls, pundits write stories about the new religious movement; advertisers and other cultural influencers take note of the polls and factor them into their work. Before you know it, the decline of religion and the rise of the Nones is a matter of conventional wisdom people take for granted. In other words, polls can have a disproportionate social impact, even if they are unreliable.

None of this is to say that organized religion isn’t in fact experiencing a decline; as I say, there are plenty of indications, other than these polls. But I wonder how major polling firms will respond to Wuthnow’s criticisms. At the very least, his essay suggests many of us should treat surveys on American religion with more caution than we do.

Mark Movsesian is the Frederick A. Whitney Professor of Contract Law and the Director of the Center for Law and Religion at St. John’s University School of Law. His previous blog posts can be found here.