What a four percent response rate actually does to a poll
Single-digit response rates are not a scandal. Hiding them is. Here is what they change, what they don't, and what we do about it.
Integrity Polling research staff · Methods
· 3 min read
Sample data. This article is a demonstration of our analysis format. The figures it discusses are illustrative unless a source is cited.
In 1997, Pew Research Center reported that about 36 percent of the households it called completed a telephone interview. By 2018, the figure it reported was 6 percent. The decline has not reversed. Most telephone and text-to-web polls today, including the ones we will field, finish with response rates in the low single digits.
This is usually presented as either a crisis or a non-issue. It is neither. It is a specific, measurable change in what a poll can and cannot do, and it has specific remedies with specific limits.
Telephone survey response rates, as reported by Pew Research Center
Approximate figures from Pew's published methodology reports. Verify against the original source before citing.
Pew Research Center, 'Response rates in telephone surveys have resumed their decline' (2019) and earlier methodology statements. Values are illustrative of the trend, not a reproduction of Pew's series.
What a low response rate does
A low response rate does not, by itself, make a poll wrong. It makes a poll fragile. The people who answer are a small, self-selected sliver of the people who were asked, and the poll is accurate only to the extent that the sliver resembles the whole, or can be weighted until it does.
Weighting fixes the part we can see. If 40 percent of respondents have a college degree and 35 percent of the population does, we weight the graduates down and the non-graduates up. Age, sex, race, region, and increasingly past vote are handled the same way.
Weighting cannot fix the part we cannot see. The person who answers a survey from an unknown number is more trusting, more civically engaged, more interested in politics, and more comfortable with institutions than the person who lets it ring out. None of those traits is on the Census, so none of them is a standard weighting target. If they correlate with how people vote, and in recent elections they have, the poll is biased in a direction that weighting on demographics does not touch.
What we do about it
We cannot raise the response rate much; nobody can. What we can do is reduce how much the response rate matters.
Weight on engagement, not just demographics. Every Integrity Polling survey carries a civic-engagement item (contacted an official, attended a meeting, volunteered in the past year) and we weight to an external benchmark for it. It is an imperfect proxy for the trust-and-interest skew, but it is a proxy, and it moves numbers in the direction the post-mortems say they should move.
Mix modes. People who will not answer a phone sometimes answer a text. People who will not click a text link sometimes sit on a probability panel. No single mode reaches everyone, and the biases of each mode are at least partly different. Our releases report the mode mix so you can see how much of the sample came from where.
Publish the response rate. Every release states it, computed to the AAPOR standard, in the methodology box. A pollster who reports a 4 percent response rate is telling you something true and uncomfortable. A pollster who does not report one is asking you not to wonder.
Grade the outcome. Response rates are an input. Error is the output. The Scorecard will show, over time, whether our weighting and mode mix are producing results within our stated margins. If they are not, the response rate is the first place we will look, and the first thing we will tell you.
The honest summary
A four percent response rate means a poll is leaning hard on its weighting model, and a weighting model is a set of assumptions about who did not answer. We publish ours. That does not make the assumptions right. It makes them checkable, which is the most any pollster working in this environment can honestly offer.