By R. Michael Alvarez
I recently had the opportunity to talk with Lonna Atkeson, Professor of Political Science and Regents’ Lecturer at the University of New Mexico. We discussed her opinions about improving survey methodology and her thoughts about how surveys are being used to study important applied questions. Lonna has written extensively about survey methodology, and has developed innovative ways to use surveys to improve election administration (her 2012 study of election administration is a wonderful example).
In the current issue of Political Analysis is the Symposium on Advances in Survey Methodology, which Lonna and I co-edited; in addition to the five research articles in the Symposium, we wrote an introduction that puts each of the research articles in context and talks about the current state of research in survey methodology. Also, Lonna and I are co-editing the Oxford Handbook on Polling and Polling Methods, which is in initial stages of development.
It’s well-known that response rates for traditional telephone surveying have declined dramatically. What’s the solution? ow can survey researchers produce quality data given low response rates with traditional telephone survey approaches?
What we’ve learned about response rates is they are not the be all or end all as an evaluative tool for the quality of the survey, which is a good thing because response rates are ubiquitously low! There is mounting evidence that response rates per se are not necessarily reflective of problems in nonresponse. Nonresponse error appears to be more related to the response rate interacting with the characteristic of the nonrespondent. Thus, if survey topic salience leads to response bias then nonresponse error becomes a problem, but in and of itself response rate is only indirect evidence of a potential problem. One potential solution to falling response rates is to use mixed mode surveys and find the best contact and response option for the respondent. As polling becomes more and more sophisticated, we need to consider best contact and response methods for different types of sample members. Survey researchers need to be able to predict the most likely response option for the individual and pursue that strategy.
Close up of a man smiling on the line through a headset. © cenix via iStockphoto.
Much of your recent work uses “mixed-mode” survey methods. What’s a “mixed-mode” survey? What are the strengths and weaknesses of this approach?
Mixed mode surveys use multiple methods to contact or receive information from respondents. Thus, mixed mode surveys involve both mixtures of data collection and communications with the respondent. For example, a mixed mode survey might contact sample members by phone or mail and then have them respond to a questionnaire over the Internet. Alternatively a mixed mode survey might allow for multiple forms of response. For example, sample frame members may be able to complete the interview over the phone, by mail, or on the web. Thus a respondent who does not respond over the Internet may in subsequent contact receive a phone call or a FTF visit or may be offered a choice of response mode on the initial contact.
When you see a poll or survey reported online or in the news media, how do you determine if the poll was conducted in a way that has produced reliable data? What indicates a high-quality poll?
This is a difficult question because all polls are not created equally and many reported polls might have problems with sampling, nonresponse bias, question wording, etc. The point being that there are many places where error creeps into your survey not just one and to evaluate a poll researchers like to think in terms of total survey error, but the tools for that evaluation are still in the development stage and is an area of opportunity for survey researchers and political methodologists. We also need to consider a total survey error approach in how survey context, which now varies tremendously, influences respondents and what that means for our models and inferences. This is an area for continued research. Nevertheless, the first criteria for examining a poll ought to be its transparency. Polling data should include information on who funded the poll, a copy of the instrument, a description of the sampling frame, and sampling design (e.g. probability, non-probability, the study size, estimates of sampling error for probability designs, information on any weighting of the data, and how and when the data were collected). These are basic criteria that are necessary to evaluate the quality of the poll.
Clearly, as our symposium on survey methodology in the current issue of Political Analysis discusses, survey methodology is at an important juncture. What’s the future of public opinion polling?
Survey research is a rapidly changing environment with new methods for respondent contacting and responding. Perhaps the biggest change in the most recent decade is the move away from predominantly interviewer driven data collection methods (e.g. phone, FTF) to respondent driven data collection methods (e.g. mail, Internet, CASI), the greater use of mixed mode surveys, and the introduction of professional respondents who participate over long periods of time in discontinuous panels. We are just beginning to figure out how all these pieces fit together and we need to come up with better tools to assess the quality of data we are obtaining. The future of polling and its importance in the discipline, in marketing, and in campaigns will continue, and as academics we need to be at the forefront of evaluating these changes and their impact on our data. We tend to brush over the quality of data in favor of massaging the data statistically or ignoring issues of quality and measurement altogether. I’m hoping the changing survey environment will bring more political scientists into an important interdisciplinary debate about public opinion as a methodology as opposed to the study of the frequencies of opinions. To this end, I have a new Oxford Handbook, along with my co-editor Mike Alvarez, on polling and polling methods that will take a closer look at many of these issues and be a helpful guide for current and future projects.
In your recent research on election administration, you use polling techniques as tools to evaluate elections. What have you learned from these studies, and based on your research what do you see are issues that we might want to pay close attention to in this fall’s midterm elections in the United States?
We’ve learned so much from our election administration work about designing polling places, training poll workers, mixed mode surveys and more generally evaluating the election process. In New Mexico, for example, we have been interviewing both poll workers and voters since 2006, giving us five election cycles, including 2014, that provide an overall picture of the current state of election administration and how it’s doing relative to past election cycles. Our multi-method approach provides continuous evaluation, review, and improvement to New Mexico elections. This fall I think there are many interesting questions. We are interested in some election reform questions about purging voter registration files, open primaries, the straight party ballot options and felon re-enfranchisement. We are also especially interested in how voters decide whether to vote early or on Election Day and on Election Day where they decide to vote if they are using voting convenience centers instead of precincts. This is an important policy question, but where we place vote centers might impact turnout or voter satisfaction or confidence. We are also very interested in election lines and their impact on voters. In 2012 we found that voters on average can fairly easily tolerate lines of about ½ an hour, but feel there are administrative problems when lines grow longer. We want to continue to drill down on this question and examine when lines deter voters or create poor experiences that reduce the quality of their vote experience.
Lonna Rae Atkeson is Professor of Political Science and Regents’ Lecturer at the University of New Mexico. She is a nationally recognized expert in the area of campaigns, elections, election administration, survey methodology, public opinion and political behavior and has written numerous articles, book chapters, monographs and technical reports on these topics. Her work has been supported by the National Science Foundation, the Pew Charitable Trusts, the JEHT Foundation, the Galisano Foundation, the Bernalillo County Clerk, and the New Mexico Secretary of State. She holds a BA in political science from the University of California, Riverside and a Ph.D. in political science from the University of Colorado, Boulder.
R. Michael Alvarez is a professor of Political Science at Caltech. His research and teaching focuses on elections, voting behavior, and election technologies. He is editor-in-chief of Political Analysis with Jonathan N. Katz.
Political Analysis chronicles the exciting developments in the field of political methodology, with contributions to empirical and methodological scholarship outside the diffuse borders of political science. It is published on behalf of The Society for Political Methodology and the Political Methodology Section of the American Political Science Association. Political Analysis is ranked #5 out of 157 journals in Political Science by 5-year impact factor, according to the 2012 ISI Journal Citation Reports. Like Political Analysis on Facebook and follow @PolAnalysis on Twitter.
Subscribe to the OUPblog via email or RSS.
Subscribe to only politics and political science articles on the OUPblog via email or RSS.
The post Improving survey methodology: a Q&A with Lonna Atkeson appeared first on OUPblog.
