JacketFlap connects you to the work of more than 200,000 authors, illustrators, publishers and other creators of books for Children and Young Adults. The site is updated daily with information about every book, author, illustrator, and publisher in the children's / young adult book industry. Members include published authors and illustrators, librarians, agents, editors, publicists, booksellers, publishers and fans. Join now (it's free).
Login or Register for free to create your own customized page of blog posts from your favorite blogs. You can also add blogs by clicking the "Add to MyJacketFlap" links next to the blog name in each post.
Blog Posts by Tag
In the past 7 days
Blog Posts by Date
Click days in this calendar to see posts by day or month
Viewing: Blog Posts Tagged with: Political Analysis, Most Recent at Top [Help]
Results 1 - 18 of 18
How to use this Page
You are viewing the most recent posts tagged with the words: Political Analysis in the JacketFlap blog reader. What is a tag? Think of a tag as a keyword or category label. Tags can both help you find posts on JacketFlap.com as well as provide an easy way for you to "remember" and classify posts for later recall. Try adding a tag yourself by clicking "Add a tag" below a post's header. Scroll down through the list of Recent Posts in the left column and click on a post title that sounds interesting. You can view all posts from a specific blog by clicking the Blog name in the right column, or you can click a 'More Posts from this Blog' link in any individual post.
Recent years have seen amazing growth in the development of new tools that can be used to make causal claims about complex social phenomenon. Social scientists have been at the forefront of developing many of these new tools, in particular ones that can give analysts the ability to make causal inferences in survey research.
Academic publishing is not as simple as it may appear. University presses such as Oxford and Fordham range from large to small; for-profit publishers such as Wiley and Elsevier must appeal to both academics and shareholders; start-ups such as Academia.edu and WriteLatex are fulfilling smaller services; and niche publishers, such as Hurst, offer tremendous depth and breadth of specific subject areas.
On 20 October 2015, the global mathematical community is celebrating World Statistics Day. In honour of this, we present here a reading list of OUP books and journal articles that have helped to advance the understanding of these mathematical concepts.
Every two days, humans produce more textual information than the combined output of humanity from the dawn of recorded history up through the year 2003. Much of this text is directly relevant to questions in political science. Governments, politicians, and average citizens regularly communicate their thoughts and opinions in writing, providing new data from which to understand the political world and suggesting new avenues of study in areas that were previously thought intractable.
Peer review is one of the foundations of science. To have research scrutinized, criticized, and evaluated by other experts in the field helps to make sure that a study is well-designed, appropriately analyzed, and well-documented. It helps to make sure that other scholars can readily understand, appreciate, and build upon that work.
Social scientists made important contributions towards improving the conduct and administration of elections. A paper recently published in Political Analysis continues that tradition, and introduces the use of web search data to the study of public administration and public policy.
Today’s data scientist must know how to write good code. Regardless of whether they are working with a commercial off-the-shelf statistical software package, R, python, or perl, all require the use of good coding practices. Large and complex datasets need lots of manipulation to wrangle them into shape for analytics, statistical estimation often is complex, and presentation of complicated results sometimes requires writing lots of code. To make sure that code is understandable to the author and others, good coding practices are essential.
Many who teach methodology, statistics, and data science, are increasingly teaching their students how to write good computer code. As a practical matter, if a professor requires that students turn in their code for a problem set, that code needs to be well-crafted to be legible to the instructor. But as increasing numbers of our students are writing and distributing their code and software tools to the public, professionally we need to do more to train students how to write good code. Finally, good code is critical for research replication and transparency — if you can’t understand someone’s code, it might be difficult or impossible to be able to reproduce their analysis.
When I first started teaching methods to graduate students, there was little in the methodological literature that I found useful for teaching graduate students good coding practices. But in 1995, my colleague Jonathan Nagler wrote out some great guidance on good methodological practices, in particular guidelines for good coding style. His piece is available online (“Coding Style and Good Computing Practices”), and his advice from 1995 is as relevant today as it was then. I use Jonathan’s guidelines in my graduate teaching.
Over the past few years, as Political Analysis has focused resources on research replication and transparency, it’s become clear that we need to develop better guidance for researchers and authors regarding how to write good code. One of the biggest issues that we run into when we review replication materials that are submitted to the journal is poor documentation and unclear code; and if we can’t figure out how the code works, I’m sure that our readers will have the same problem.
We’ve been thinking of developing some guidelines for documentation of replication materials, and standards for coding practices. As part of that research, I asked Jonathan if he would write an update of his 1995 essay, and for him to reflect some on how things might have evolved in terms of good computing practices since 1995. His thoughts are below, and I encourage readers to also read Jonathan’s original 1995 essay.
* * * * *
Coding style and good computing practices: it is easy to get the style right, harder to get good practice, by Jonathan Nagler, NYU
Many years ago I was prompted to write Coding Style and Good Computing Practices, an article laying out guidelines for coding style for political scientists. The article was reprinted in a symposium on replication in PS (September 1995, Vol. 28, No. 3, 488-492). According to Google Scholar, it has rarely been cited, but I’m convinced it has been read quite often because I’ve seem some idiosyncratic suggestions made in it in the code of other political scientists. Though re-reading the article I am reminded how many people have not read it, or just ignored it.
Here is a list of basic points reproduced from that article:
Labbooks: essential.
Command files: they should be kept.
Data-manipulation vs. data-analysis: these should be in distinct files.
Keep tasks compartmentalized (‘modularity’).
Know what the code is supposed to do before you start.
Don’t be too clever.
Variable names should mean something.
Use parentheses and white-space to make code readable.
Documentation: all code should include comments meaningful to others.
And I concluded with a list of rules:
Maintain a labbook from the beginning of a project to the end.
Code each variable so that it corresponds as closely as possible to a verbal description of the substantive hypothesis the variable will be used to test.
Errors in code should be corrected where they occur and the code re-run.
Separate tasks related to data-manipulation vs data-analysis into separate files.
Each program should perform only one task.
Do not try to be as clever as possible when coding. Try to write code that is as simple as possible.
Each section of a program should perform only one task.
Use a consistent style regarding lower and upper case letters.
Use variable names that have substantive meaning.
Use variable names that indicate direction where possible.
Use appropriate white-space in your programs, and do so in a consistent fashion to make them easy to read.
Include comments before each block of code describing the purpose of the code.
Include comments for any line of code if the meaning of the line will not be unambiguous to someone other than yourself.
Rewrite any code that is not clear.
Verify that missing data is handled correctly on any recode or creation of a new variable.
After creating each new variable or recoding any variable, produce frequencies or descriptive statistics of the new variable and examine them to be sure that you achieved what you intended.
When possible, automate things and avoid placing hard-wired values (those computed ‘by-hand’) in code.
Those are still very good rules, I would not change any of them. I would add one, and that is to put comments in any paper citing the piece of code that produced the figures or tables in the paper. In 20 years a lot of things have changed about how we do computing. It has gotten much easier to follow good computing practices. Github has made it easy to share code, maintain revision history, and publish code. And the set of people who seamlessly collaborate by sharing files over Dropbox or one of its competitors probably dwarfs the number of political scientists using Github. But to paraphrase a common computing aphorism (GIGO), sharing or publishing badly written code won’t make it easy for people to replicate or build on your work.
I was motivated to write that article because as I stated then, most political scientists aren’t trained as computer programmers. Nor were most political scientists trained to work in a laboratory. So the article covered both style of code, and computing practice to make sure that an entire research project could be reproduced by someone else. That means keeping track of where you got your data, how it was processed, etc.
Any computer code is a set of instructions that produces results when read by a machine, and we can evaluate the code based on the results it produces. But when we share code we expect it to be read by humans. Two pieces of code be functionally equivalent — they could produce identical results when read by a machine — even though one is easy to read and understand by a human; while the other is pretty much unintelligible to a human. If you expect people to use your code, you need to make the code easy to read. I try to ask every graduate student I am going to work with to read several chapters from Brian W. Kernighan and Rob Pike’s, The Practice of Programming (1999), especially the Preface, Chapters 1, 3, 5, 6, and the Epilogue.
It has turned out to be easier to write clean code than to maintain good computing practices overall that would lead to easy reproducibility of an entire research project. It is fairly easy to post a ‘replication’ dataset, and the code used to produce the figures and tables in a paper. But that doesn’t really tell someone everything they need to know to try to reproduce your work, or extend it to other data. They need to know how your data was generated. And those steps occur in the production of the replication dataset, not in the use of it.
Most research projects in political science pull in data from many sources. And many, many coding decisions are made along the way to a finished product. All of those decisions may be visible in the code; but keeping coherent lab-books is essential for sifting through all the lines of code of any large project. And ‘projects’ rarely stand-alone anymore. Work on one dataset is linked to many projects, often with over-lapping sets of co-authors.
At the beginning of a research project it’s important for everyone to agree where the code is, where the data is, and what the overall structure of the documentation is. That means decisions about whether documentation is grouped by project (which could mean by individual paper), or by dataset. And it means reaching some agreement on whether there is a master document that points to many smaller documents describing individual tasks, or whether the whole project description sits in a single document. None of this is exciting to work out, certainly not as exciting as doing the research. But it is essential. A good goal of doing all this is to make it as easy as possible to make the whole bundle of documentation and code public as soon as it is time to do so. It both saves time when it is time to release documentation, and imposes some good habits and structure along the way.
Heading image: Typing computer screen reflection by Almonroth. CC BY-SA 3.0 via Wikimedia Commons.
Introduction, from Michael Alvarez, co-editor of Political Analysis
Recently I asked Nathaniel Beck to write about his experiences with research replication. His essay, published on 24 August 2014 on the OUPblog, concluded with a brief discussion of a recent experience of his when he tried to obtain replication data from the authors of a recent study published in PNAS, on an experiment run on Facebook regarding social contagion. Since then the story of Neal’s efforts to obtain this replication material have taken a few interesting twists and turns, so I asked Neal to provide an update — because the lessons from his efforts to get the replication data from this PNAS study are useful for the continued discussion of research transparency in the social sciences.
After not hearing from Adam Kramer of Facebook, even after contacting PNAS, I persisted with both the editor of PNAS (Inder Verma, who was most kind) and with the NAS through “well connected” friends. (Getting replication data should not depend on knowing NAS members!). I was finally contacted by Adam Kramer, who offered that I could come out to Palo Alto to look at the replication data. Since Facebook did not offer to fly me out, I said no. I was then offered a chance to look at the replication files in the Facebook office 4 blocks from NYU, so I accepted. Let me stress that all dealings with Adam Kramer were highly cordial, and I assume that delays were due to Facebook higher ups who were dealing with the human subjects firestorm related to the Kramer piece.
When I got to the Facebook office I was asked to sign a standard non-disclosure agreement, which I dec. To my surprise this was not a problem, with the only consequence being that a security officer would have had to escort me to the bathroom. I then was put in a room with a Facebook secure notebook with the data and R-studio loaded; Adam Kramer was there to answer questions, and I was also joined by a security person and an external relations person. All were quite pleasant, and the security person and I could even discuss the disastrous season being suffered by Liverpool.
I was given a replication file which was a data frame which had approximately 700,000 rows (one for each respondent) and 7 columns containing the number of positive and negative words used by each respondent as well as the total word count of each respondent, percentages based on these numbers, experimental condition. and a variable which omitted some respondents for producing the tables. This is exactly the data frame that would have been put in an archive since it contained all the data needed to replicate the article. I also was given the R-code that produced every item in the article. I was allowed to do anything I wanted with that data, and I could copy the results into a file. That file was then checked by Facebook people and about two weeks later I received the entire file I created. All good, or at least as good as it is going to get.
Intel team inside Facebook data center. Intel Free Press. CC BY 2.0 via Wikimedia Commons.
The data frame I played with was based on aggregating user posts so each user had one row of data, regardless of the number of posts (and the data frame did not contain anything more than the total number of words posted). I can understand why Facebook did not want to give me the data frame, innocuous as it seemed; those who specialize in de-de-identifying private data and reverse engineering code are quite good these days, and I can surely understand Facebook’s reluctance to have this raw data out there. And I understand why they could not give me all the actual raw data, which included how feeds were changed and so forth; this is the secret sauce that they would not like reverse engineered.
I got what I wanted. I could see their code, could play with density plots to get a sense of words used, I could change the number of extreme points dropped, and I could have moved to a negative binomial instead of a Poisson. Satisfied, I left after about an hour; there are only so many things one can do with one experiment on two outcomes. I felt bad that Adam Kramer had to fly to New York, but I guess this is not so horrible. Had the data been more complicated I might have felt that I could not do everything I wanted, and running a replication with 3 other people in a room is not ideal (especially given my typing!).
My belief is that that PNAS and the authors could simply have had a different replication footnote. This would have said that the code used (about 5 lines of R, basically a call to a Poisson regression using GLM) is available at a dataverse. In addition, they could have noted that the GLM called used the data frame I described, with the summary statistics for that data frame. Readers could then see what was done, and I can see no reason for such a procedure to bother Facebook (though I do not speak for them). I also note a clear statement on a dataverse would have obviated the need for some discussion. Since bytes are cheap, the dataverse could also contain whatever policy statement Facebook has on replication data. This (IMHO) is much better than the “contact the authors for replication data” footnote that was published. It is obviously up to individual editors as to whether this is enough to satisfy replication standards, but at least it is better than the status quo.
What if I didn’t work four blocks from Astor Place? Fortunately I did not have to confront this horror. How many other offices does Facebook have? Would Adam Kramer have flown to Peoria? I batted this around, but I did most of the batting and the Facebook people mostly did no comment. So someone else will have to test this issue. But for me, the procedure worked. Obviously I am analyzing lots more proprietary data, and (IMHO) this is a good thing. So Facebook, et al., and journal editors and societies have many details to work out. But, based on this one experience, this can be done. So I close this with thanks to Adam Kramer (but do remind him that I have had auto-responders to email for quite while now).
On the more trivial issue of my own dataverse, I am happy to report that almost everything that was once on an a private ftp site is now on my Harvard dataverse. Some of this was already up because of various co-authors who always cared about replication. And on stuff that was not up, I was lucky to have a co-author like Jonathan Katz, who has many skills I do not possess (and is a bug on RCS and the like, which beats my “I have a few TB and the stuff is probably hidden there somewhere”). So everything is now on the dataverse, except for one data set that we were given for our 1995 APSR piece (and which Katz never had). Interestingly, I checked the original authors’ web sites (one no longer exists, one did not go back nearly that far) and failed to make contact with either author. Twenty years is a long time! So everyone should do both themselves and all of us a favor, and build the appropriate dataverse files contemporaneously with the work. Editors will demand this, but even with this coercion, this is just good practice. I was shocked (shocked) at how bad my own practice was.
Heading image: Wikimedia Foundation Servers-8055 24 by Victorgrigas. CC BY-SA 3.0 via Wikimedia Commons.
At the American Political Science Association meetings earlier this year, Gary King, Albert J. Weatherhead III University Professor at Harvard University, gave a presentation on Dataverse. Dataverse is an important tool that many researchers use to archive and share their research materials. As many readers of this blog may already know, the journal that I co-edit, Political Analysis, uses Dataverse to archive and disseminate the replication materials for the articles we publish in our journal. I asked Gary to write some remarks about Dataverse, based on his APSA presentation. His remarks are below.
* * * * *
An update on Dataverse
By Gary King
If you’re an academic researcher, odds are you’re not a professional archivist and so you probably have more interesting things to do when making data available than following the detailed protocols and procedures established over many years by the archiving community. That of course might be OK for any one of us but it is a terrible loss for all of us. The Dataverse Network Project offers a solution to this problem by eliminating transaction costs and changing the incentives to make data available by giving you substantial web visibility and academic citation credit for your data and scholarship (King, 2007). Dataverse Networks are installed at universities and other institutions around the world (e.g., here is the Dataverse network at Harvard’s IQSS), and represent the world’s largest collection of social science research data. In recent years, Dataverse has also been adopted by an increasingly diverse array of other fields and protocols and procedures are being built out to enable numerous fields of science, social science, and the humanities to work together.
With a few minutes of set-up time, you can add your own Dataverse to your homepage with a list of data sets or replication data sets you make available, with whatever levels of permission you want for the broader community, and a vast array of professional services (e.g., here’s my Dataverse on my homepage). People will be able to more easily find your data and homepage, explore your data and scholarship, find connections to other resources, download data in any format, and learn proper ways of citing your work. They will even be able to analyze your data while still on your web site with a vast array of statistical methods through the transparent and automated connection Dataverse has built to Zelig: Everyone’s Statistical Software, and through Zelig to R. The result is that your data will be professionally preserved and easier to access — effectively automating the tasks of professional archiving, including citing, sharing, analyzing, archiving, preserving, distributing, cataloging, translating, disseminating, naming, verifying, and replicating data.
Dataverse Network Diagram, by Institute for Quantitative Social Science. CC-BY-2.0 via Wikimedia Commons.
Dataverse is an active project with new developments in software, protocols, and community connections coming rapidly. A brand new version of the code, written from scratch, will be available in a few months. Through generous grants from the Sloan Foundation, we have been working hard on eliminating other types of transaction costs for capturing data for the research community. These include deep integration with scholarly journals so that it can be trivially easy for an editor to encourage or require data associated with publications to be made available. We presently offer journals three options:
Do it yourself. Authors publish data to their own dataverse, put the citation to their data in their final submitted paper. Journals verify compliance by having the copyeditor check for the existence of the citation.
Journal verification. Authors submit draft of replication data to Journal Dataverse. Journal reviews it, and approves it for release. Finally, the dataset is published with a formal data citation and back to the article. (See, for example, the Political Analysis Dataverse, with replication data back to 1999.)
Full automation: Seamless integration between journal submission system and Dataverse; Automatic Link created between article and data. The result is that it is easy for the journal and author and many errors are eliminated.
Full automation in our third option is where we are heading. Already today, in 400 scholarly journals in the Open Journal System, the author enters their data as part of submission of the final draft of the accepted paper for publication, and the citation, permanent links between the data and the article, and formal preservation is taken care of, all automatically. We are working on expanding this as an option for all of OJS’s 5,000+ journals, and to a wide array of other scholarly journal publishers. The result will be that we capture data with the least effort on anyone’s part, at exactly the point where it is easiest and most important to capture.
We are also working on extending Dataverse to cover new higher levels of security that are more prevalent in big data collections and those in public health, medicine, and other areas with informative data on human subjects. Yes, you can preserve data and make it available under appropriate protections, even if you have highly confidential, proprietary, or otherwise sensitive data. We are working on other privacy tools as well. We already have an extensive versioning system in Dataverse, but are planning to add support for continuously updated data such as streamed from sensors, tools for online fast data access, queries, visualization, analysis methods for when data cannot be moved because of size or privacy concerns, and ways to use the huge volume of web analytics to improve Dataverse and Zelig.
This post comes from the talk I gave at the American Political Association Meetings August 2014, using these slides. Many thanks to Mike Alvarez for inviting this post.
Featured image: Matrix code computer by Comfreak. CC0 via Pixabay.
Throughout my career, there have been many times when advice, support, and criticism were critical for my own professional development. Sometimes that assistance came from people who were formally tasked with providing advice; a good example is a Ph.D. advisor (in my case, John Aldrich of Duke University, who has been a fantastic advisor and mentor to a long list of very successful students). Sometimes that advice was less formal, coming from senior colleagues, other academics at conferences, and in many cases from peers. The lesson is professional advice and support — or to put it into a single term, mentoring — comes from many different sources and occurs in many different ways.
However, there is growing concern in political science that more mentoring is necessary, that there are scholars who are not getting the professional support and advice that they need to help them with career decisions, teaching, and the publication of their research. There are many good programs that have developed in recent years to help provide more mentoring in political methodology, for example the excellent “Visions in Methodology” program. And the Society for Political Methodology recently approved the foundation of a new professional award, to recognize excellent mentors. But more needs to be done to improve mentoring and mentoring opportunities in academia.
After the conference I sent Leslie, Tiffany, and Ashley some questions about mentoring by email. Their responses are informative and helpful, and should be read by anyone who is interested in mentoring.
R. Michael Alvarez: How have you benefited from being involved in mentoring relationships?
Tiffany D. Barnes: I have benefited in a number of ways from being involved in a mentoring relationship. Mentors have provided me with feedback on research at multiple different stages of the research process. They have provided me with professional advice about a number of things including applying for fellowships and grants, marketing my book manuscript to university presses, and navigating the negotiation process at my university. My mentoring relationships have broadened my network of scholars with similar research interests and/or professional goals, which in turn have resulted in a number of different opportunities (e.g. coauthors, and invitations to participate in conference panels/round tables, mini-conferences, and edited volumes/special journal issues). Equally important, my mentoring relationships have resulted in a number of valuable friendships that make working in the profession more enjoyable.
Leslie A. Schwindt-Bayer: As a mentee, I really benefited from getting guidance, feedback, and research assistance from many different formal and informal mentors over the years. As a mentor, I get to give that back which is a great opportunity.
Brett Ashley Leeds: I believe fundamentally that no one figures everything out on his or her own. I know for sure that I did not, and I have had (and continue to have) a variety of mentors throughout my career. As a mentee, what I really value is knowing that I have people who respect me enough to tell me when I am wrong and to help me improve. As a mentor, I not only learn a lot from thinking intently about my mentees’ work and articulating my opinions for them, but I also get great personal satisfaction from the relationships that evolve and from helping others to succeed. It feels good to pay forward what has been done for me.
R. Michael Alvarez: Why has the issue of mentoring become an important topic of conversation in academia, and in particular, in political science?
Tiffany D. Barnes: Although it is well established that mentoring is an important aspect of professional development, it has recently become an important topic of conversation because academics have become aware that not all scholars have the same opportunities to develop mentorship relationships nor do they derive the same benefits from mentor relationships. In particular, women and minorities may face more challenges when it comes to identifying mentors in the field and they may not reap the same benefits (e.g. opportunities to collaborate, sponsorship) from mentorship relationships as men do. In the long run, this “mentor gap” may have negative repercussions for the retention and career advancement of some otherwise talented scholars.
If a scholar feels they would benefit by mentoring, how can they seek out a mentor? What should they look for in an appropriate mentor?
Leslie A. Schwindt-Bayer: Mentoring relationships can be both informal and formal. Informal relationships often emerge when scholars ask for advice and support from colleagues in their department, subfields, or various disciplinary organizations. Formal relationships sometimes emerge organically or at the initiative of a mentee or mentor, but they also can be entered into through a number of mentoring programs in the discipline. For women, the Visions in Methodology program offers a mentoring program through which mentees can ask to be paired with a mentor. They usually ask the mentee to suggest someone they would like to be paired with and then check with the suggested mentor about interest and availability. The Midwest Women’s Caucus has a mentoring program for women in any subfield. They ask individuals interested in mentoring and being mentored to volunteer to participate and then pair them by interest. Other organizations and groups probably offer similar programs.
In seeking a mentor, either formally or informally, you should think about exactly what you want out of the relationship. Are you looking for someone to provide you with general guidance about the profession or are you seeking someone who is willing to read your work from time to time and talk through research challenges when you come across them? Are you in your first year out, feeling lost, and needing help getting back on track or are you close to tenure and looking for guidance on how to navigate the process? Do you want a mentor whose style is to give “pep talks” or “straight talk?” Knowing what you want out of the relationship will help you identify the right person for the job.
Tiffany D. Barnes: Scholars who want to find a mentor can look for a mentor by signing up for a formal mentor match or by identifying someone in the profession who shares similar research interests or professional goals.
A formal mentor match is good option for identifying someone who is interested in serving in a capacity as a mentor. Typically the mentor program will ask you questions about what you are looking for in a mentor relationship, your research interests, your rank, and your professional interests. The program will try to match you with a mentor based on this information. If you are paired with someone through a program, you can be confident that your mentor wants to help you. These relationships can be very valuable, but, as with all mentor-mentee relationships, it requires initiative on the part of the mentee. It is the mentee’s responsibility to drive the mentor-mentee relationship. Mentees should identify why they want a mentor and reach out to the mentor and ask for help in areas where they can benefit the most. One criticism of formal matching programs is that they may not always result in the best “fit.” Even if you do not think the match is the best fit, there are still a number of benefits you can derive from the relationship. Your research interests do not have to perfectly overlap for you to benefit from the relationship. Indeed, most successful scholars have a wealth of information, advice, and perspective to offer junior colleagues. It is up to the mentee to identify areas where your needs or interests intersect with the mentor’s strengths, experiences, and interests — and to capitalize on these opportunities.
A second option is to develop a more informal mentor relationship. To do this, mentees should identify someone in the field who has similar research interests or professional goals. Mentees should identify different opportunities to get to know scholars with similar interests and try to develop these relationships from there. For example, you may have the opportunity to establish relationships with scholars when you present research on the same panel, when someone shows interest in your work by offering comments or questions about your research (or vice versa), or even when you have the opportunity to bring a guest speaker to your university. By following up with people after the initial meeting and/or taking them up on their offer to read and comment on your research, you can begin to establish relationships with them. These relationships may take time to develop and they may be difficult establish if you are new to the profession or do not know many scholars in your field. Finally, when attempting to establish more informal mentor relationships, it is important to be self-aware. Some people will show interest in you and be eager to get to know and help you, others will not, and no one is obligated to do so. Respect people’s rights to not be interested in you and try not to take it personal.
Brett Ashley Leeds: My view is that it is less important to find one person that can be identified as “a mentor” and instead to focus on finding mentoring, even if it comes from a variety of people. I encourage scholars to identify people who have skills, abilities, and/or information that they think would be useful to them– basically people they would like to emulate in particular areas of their work. Approach these folks politely in person or by email (for instance, asking to have coffee at a conference) and ask questions. Some will not be responsive, but many will be responsive and helpful. Follow up with those who are helpful. In some cases a relationship will develop.
R. Michael Alvarez: What are the most important “dos” and “don’ts” for a scholar who is in a mentoring relationship?
Brett Ashley Leeds: Since below I cover some tips for mentors, here are some tips for mentees: (1) Figure out what it is you want to know/learn. Think of both specific and general questions so you are prepared to ask when the opportunity arises. (2) Recognize the time and costs of what you ask and make things as easy as possible for your mentor by reminding him/her of past interactions and explaining the specific feedback you are looking for. (3) Understand that ultimately you are responsible for your own decisions. Ask your mentor to explain why he/she believes a particular action/approach is best, and for major decisions, seek advice from multiple people. (4) Let your mentors know about the outcomes. For example, if a mentor helps you with a paper, send a note when the paper is accepted for publication.
Leslie A. Schwindt-Bayer: For mentees, be assertive and discuss with your mentor when your relationship begins just what you each want from the relationship and are willing to commit to it. If you need something from your mentor, don’t wait for him/her to reach out to you. Email, call, or arrange to meet with your mentor at a conference. Since the mentee is the one who needs the mentoring relationship the most, the mentee needs to take the initiative to ask for help or guidance from the mentor.
Tiffany D. Barnes: Establish clear expectations and boundaries. Tell your mentor what you are hoping to get out of a mentoring relationship, and don’t be afraid to ask your mentor for help in areas where you could benefit the most. That said, it is important to acknowledge that your mentor may not always be willing or able to help you in the ways you want. Respect these boundaries and do not take them personal.
When establishing boundaries, it is important to respect your mentor’s time and to be cognizant and courteous with the time you ask of your mentor. For example, if your mentor agrees to meet with you for half an hour, pay attention to the time and wrap up your meeting in a timely manner. Your mentor will likely appreciate not having to cut you short, and, if they know you respect their time, it may make them more likely to make time for you in the future.
Don’t expect any single mentor to fulfill all of your mentoring needs. Different people, depending on their experience and expertise, have different things to offer. Try to identify the areas where your mentor is most likely to be of help to you and build on these strengths. Along these same lines, although your mentor likely gives great advice, you cannot expect them to have the answer to all of your questions. It is important to weight their point of view carefully and to seek out a number of different perspectives.
Seek to develop a number of mentoring relationships. It can be useful to have mentors within your own department, in your university (but outside your department), and in the discipline more broadly. Moreover, it is often just as useful to develop relationships with senior mentors, as it is to develop relationships with peer mentors.
R. Michael Alvarez: What are the responsibilities of a mentor?
Brett Ashley Leeds (1) Create an environment in which you can provide effective constructive criticism. This tends to require first establishing an environment of mutual respect. (2) Know what you know and what you don’t, and know that your experience is not universal. (3) Always explain why you are giving the advice you are giving and be willing to consider alternatives. (4) Recognize that in the end, your mentee should make his/her own decisions and may not always take all of your advice. (5) Recognize how important your opinion may be to your mentee; wield this power responsibly.
Tiffany D. Barnes: A mentor should establish clear boundaries with their mentee. Be honest and upfront the role you are and are not willing to play as a mentor. Be clear about your time constraints and the amount of time you are willing to commit to your mentee.
Leslie A. Schwindt-Bayer: If it is a formal mentoring relationship, make sure you and your mentee establish ground rules at the beginning about what each of you wants from the relationship and are willing to give to it. Don’t commit to something you aren’t willing to follow through with and be sure to follow through with whatever you commit to do for your mentee. If you can only commit to an hour of time twice a semester, that is fine, but make sure your mentee knows that and agrees that it is sufficient for him/her. If you are willing to provide general guidance but don’t want to read/comment on your mentee’s work, that is fine. But, again, make sure your mentee knows that from the beginning. Keep in mind that your mentee may place very high value on your advice and guidance so give it carefully.
R. Michael Alvarez: What are the personal and professional benefits of being a mentor?
Leslie A. Schwindt-Bayer: Too numerous to list in a short response!
Brett Ashley Leeds: It has often been said that one only really knows something when she can teach it to others. Mentoring gives me an opportunity to clarify and articulate my views on professional issues and research in a way that I otherwise might not. I frequently learn in the act of mentoring. The main benefits, however, are personal, and come from the satisfaction of helping others to achieve their goals and the feeling of paying forward what has been done in the past for me.
R. Michael Alvarez: How can professional organizations (like the Society for Political Methodology) facilitate professional mentoring?
Brett Ashley Leeds: The most important thing that professional organizations can do is provide opportunities that encourage interaction among scholars who don’t already know one another, and particularly between junior and senior scholars. Small conferences, dinners, and receptions help a lot with this. Poster sessions in which junior scholars are matched with senior discussants also help.
Tiffany D. Barnes: In my experience professional organizations play both, an important formal and informal role in facilitating professional mentoring.
Professional organization can formally facilitate mentoring relationship by matching mentors with mentees. I have two different successful mentoring relationships that were products of mentoring matches. This is a great way to help young scholars identify someone in the profession who is willing to serve as a mentor.
Professional organizations can also facilitate mentoring by simply providing both professional and social opportunities for junior scholars to meet likeminded senior (and junior!) colleagues. By becoming involved in professional organizations that align with your professional interests you will establish relationships with colleagues in your field. Most of these relationships will emerge naturally and develop slowly over time. Although you may not formally call the individuals you meet here “mentors,” they will become an important part of your mentoring community.
Leslie A. Schwindt-Bayer: One of many ways is a formal mentoring program. The Visions in Methodology mentoring program is a fantastic example, but it is only for women. This is a very positive feature of the program because women in a field with a small representation of women face different and sometimes more challenging sets of obstacles than men. However, plenty of men in the field would also benefit immensely from mentoring and so offering a similar program for men or a program that is open to both women and men, if it does not already exist, would help to facilitate formal professional mentoring in the methods subfield.
Improving the transparency of the research published in Political Analysis has been an important priority for Jonathan Katz and I as co-editors of the journal. We spent a great deal of time over the past two years developing and implementing policies and procedures to insure that all studies published in Political Analysis have replication data available through the journal’s Dataverse. At this point in time, we have over 220 studies available in the journal’s Dataverse archive, and those studies have had more than 14,400 downloads. We see this as a major accomplishment for Political Analysis.
We are also optimistic that soon many political science journals will join us in implementing similar replication standards. An increasing number of journals developing and implementing replication standards will improve the quality of research in political science, aid in the distribution of materials that can be used in our classrooms, and make the publication process more straightforward for authors.
In late September, Jonathan and I had the opportunity to participate in a two-day “Workshop on Data Access and Research Transparency” at the University of Michigan. The workshop is part of an initiative sponsored by the American Political Science Association (APSA) to develop a discipline-wide discussion of how to improve research transparency in political science. The primary goal was to bring the editors of the primary journals in political science into this conversation. While there is no doubt that there was widespread agreement among the journal editors present that making research more transparent and making data more accessible are important goals, there are still open questions about how such goals can be implemented.
One of the major products of this workshop was a statement of principles for political science journals. While the statement has not yet been released, it contains a short set of principles, the most important of which are that the signing journals will require authors to make replication materials accessible, and that the signing journals will take steps to make the research published in their journal more transparent. Political Analysis is one of the signatories of this statement: we will continue to work to improve the accessibility of data and other research materials for the papers we publish in Political Analysis, as well as assist other journals as they work to develop their own replication and research transparency standards.
Updated and clarified standards for how authors should present empirical results in their submissions, in particular tables and figures.
More detailed instructions on our replication requirement.
Encouragement and guidance for authors who wish to pre-register their research studies.
We hope that other journals will follow our lead, and that they will quickly develop strong standards for replication and research transparency. The APSA initiative is laudable, and it is helping to position political science as a leader in these areas, certainly in the social sciences but also throughout the sciences and humanities. We welcome the APSA DART initiative, and will continue to work to position Political Analysis as a leader in developing and implementing data access and research transparency standards.
There’s a lot of interesting social science research these days. Conference programs are packed, journals are flooded with submissions, and authors are looking for innovative new ways to publish their work.
This is why we have started up a new type of research publication at Political Analysis, Letters.
Research journals have a limited number of pages, and many authors struggle to fit their research into the “usual formula” for a social science submission — 25 to 30 double-spaced pages, a small handful of tables and figures, and a page or two of references. Many, and some say most, papers published in social science could be much shorter than that “usual formula.”
We have begun to accept Letters submissions, and we anticipate publishing our first Letters in Volume 24 of Political Analysis. We will continue to accept submissions for research articles, though in some cases the editors will suggest that an author edit their manuscript and resubmit it as a Letter. Soon we will have detailed instructions on how to submit a Letter, the expectations for Letters, and other information, on the journal’s website.
We have named Justin Grimmer and Jens Hainmueller, both at Stanford University, to serve as Associate Editors of Political Analysis — with their primary responsibility being Letters. Justin and Jens are accomplished political scientists and methodologists, and we are quite happy that they have agreed to join the Political Analysis team. Justin and Jens have already put in a great deal of work helping us develop the concept, and working out the logistics for how we integrate the Letters submissions into the existing workflow of the journal.
I recently asked Justin and Jens a few quick questions about Letters, to give them an opportunity to get the word out about this new and innovative way of publishing research in Political Analysis.
Political Analysis is now accepting the submission of Letters as well as Research Articles. What are the general requirements for a Letter?
Letters are short reports of original research that move the field forward. This includes, but is not limited to, new empirical findings, methodological advances, theoretical arguments, as well as comments on or extensions of previous work. Letters are peer reviewed and subjected to the same standards as Political Analysis research articles. Accepted Letters are published in the electronic and print versions of Political Analysis and are searchable and citable just like other articles in the journal. Letters should focus on a single idea and are brief—only 2-4 pages and no longer than 1500-3000 words.
Why is Political Analysis taking this new direction, looking for shorter submissions?
Political Analysis is taking this new direction to publish important results that do not traditionally fit in the longer format of journal articles that are currently the standard in the social sciences, but fit well with the shorter format that is often used in the sciences to convey important new findings. In this regard the role model for the Political Analysis Letters are the similar formats used in top general interest science journals like Science, Nature, or PNAS where significant findings are often reported in short reports and articles. Our hope is that these shorter papers also facilitate an ongoing and faster paced dialogue about research findings in the social sciences.
What is the main difference between a Letter and a Research Paper?
The most obvious difference is the length and focus. Letters are intended to only be 2-4 pages, while a standard research article might be 30 pages. The difference in length means that Letters are going to be much more focused on one important result. A letter won’t have the long literature review that is standard in political science articles and will have much more brief introduction, conclusion, and motivation. This does not mean that the motivation is unimportant; it just means that the motivation has to briefly and clearly convey the general relevance of the work and how it moves the field forward. A Letter will typically have 1-3 small display items (figures, tables, or equations) that convey the main results and these have to be well crafted to clearly communicate the main takeaways from the research.
If you had to give advice to an author considering whether to submit their work to Political Analysis as a Letter or a Research Article, what would you say?
Our first piece of advice would be to submit your work! We’re open to working with authors to help them craft their existing research into a format appropriate for letters. As scholars are thinking about their work, they should know that Letters have a very high standard. We are looking for important findings that are well substantiated and motivated. We also encourage authors to think hard about how they design their display items to clearly convey the key message of the Letter. Lastly, authors should be aware that a significant fraction of submissions might be desk rejected to minimize the burden on reviewers.
You both are Associate Editors of Political Analysis, and you are editing the Letters. Why did you decide to take on this professional responsibility?
Letters provides us an opportunity to create an outlet for important work in Political Methodology. It also gives us the opportunity to develop a new format that we hope will enhance the quality and speed of the academic debates in the social sciences.
Research transparency is a hot topic these days in academia, especially with respect to the replication or reproduction of published results.
There are many initiatives that have recently sprung into operation to help improve transparency, and in this regard political scientists are taking the lead. Research transparency has long been a focus of effort of The Society for Political Methodology, and of the journal that I co-edit for the Society, Political Analysis. More recently the American Political Science Association (APSA) has launched an important initiative in Data Access and Research Transparency. It’s likely that other social sciences will be following closely what APSA produces in terms of guidelines and standards.
One way to increase transparency is for scholars to “preregister” their research. That is, they can write up their research plan and publish that prior to the actual implementation of their research plan. A number of social scientists have advocated research preregistration, and Political Analysis will soon release new author guidelines that will encourage scholars who are interested in preregistering their research plans to do so.
In order to facilitate further discussion of the pros and cons of research preregistration, I recently asked Jaime Monogan to write a brief essay that outlines the case for preregistration, and I also asked Joshua Tucker to write about some of the concerns that have been raised about how journals may deal with research preregistration.
* * * * *
The pros of preregistration for political science
By Jamie Monogan, Department of Political Science, University of Georgia
Howard Tilton Library Computers, Tulane University by Tulane Public Relations. CC-BY-2.0 via Wikimedia Commons.
Study registration is the idea that a researcher can publicly release a data analysis plan prior to observing a project’s outcome variable. In a Political Analysis symposium on this topic, two articles make the case that this practice can raise research transparency and the overall quality of research in the discipline (“Humphreys, de la Sierra, and van der Windt 2013; Monogan 2013).
Together, these two articles describe seven reasons that study registration benefits our discipline. To start, preregistration can curb four causes of publication bias, or the disproportionate publishing of positive, rather than null, findings:
Preregistration would make evaluating the research design more central to the review process, reducing the importance of significance tests in publication decisions. Whether the decision is made before or after observing results, releasing a design early would highlight study quality for reviewers and editors.
Preregistration would help the problem of null findings that stay in the author’s file drawer because the discipline would at least have a record of the registered study, even if no publication emerged. This will convey where past research was conducted that may not have been fruitful.
Preregistration would reduce the ability to add observations to achieve significance because the registered design would signal in advance the appropriate sample size. It is possible to monitor the analysis until a positive result emerges before stopping data collection, and this would prevent that.
Preregistration can prevent fishing, or manipulating the model to achieve a desired result, because the researcher must describe the model specification ahead of time. By sorting out the best specification of a model using theory and past work ahead of time, a researcher can commit to the results of a well-reasoned model.
Additionally, there are three advantages of study registration beyond the issue of publication bias:
Preregistration prevents inductive studies from being written-up as deductive studies. Inductive research is valuable, but the discipline is being misled if findings that are observed inductively are reported as if they were hypothesis tests of a theory.
Preregistration allows researchers to signal that they did not fish for results, thereby showing that their research design was not driven by an ideological or funding-based desire to produce a result.
Preregistration provides leverage for scholars who face result-oriented pressure from financial benefactors or policy makers. If the scholar has committed to a design beforehand, the lack of flexibility at the final stage can prevent others from influencing the results.
Overall, there is an array of reasons why the added transparency of study registration can serve the discipline, chiefly the opportunity to reduce publication bias. Whatever you think of this case, though, the best way to form an opinion about study registration is to try it by preregistering one of your own studies. Online study registries are available, so you are encouraged to try the process yourself and then weigh in on the preregistration debate with your own firsthand experience.
* * * * *
Experiments, preregistration, and journals
By Joshua Tucker, Professor of Politics (NYU) and Co-Editor, Journal of Experimental Political Science
I want to make one simple point in this blog post: I think it would be a mistake for journals to come up with any set of standards that involves publically recognizing some publications as having “successfully” followed their pre-registration design while identifying others publications as not having done so. This could include a special section for articles that matched their pre-registration design, an A, B, C type rating system for how faithfully articles had stuck with the pre-registration design, or even an asterisk for articles that passed a pre-registration faithfulness bar.
Let me be equally clear that I have no problem with the use of registries for recording experimental designs before those experiments are implemented. Nor do I believe that these registries should not be referenced in published works featuring the results of those experiments. On the contrary, I think authors who have pre-registered designs ought to be free to reference what they registered, as well as to discuss in their publications how much the eventual implementation of the experiment might have differed from what was originally proposed in the registry and why.
My concern is much more narrow: I want to prevent some arbitrary third party from being given the authority to “grade” researchers on how well they stuck to their original design and then to be able to report that grade publically, as opposed to simply allowing readers to make up their own mind in this regard. My concerns are three-fold.
First, I have absolutely no idea how such a standard would actually be applied. Would it count as violating a pre-design registry if you changed the number of subjects enrolled in a study? What if the original subject pool was unwilling to participate for the planned monetary incentive, and the incentive had to be increased, or the subject pool had to be changed? What if the pre-registry called for using one statistical model to analyze the data, but the author eventually realized that another model was more appropriate? What if survey questions that was registered on a 1-4 scale was changed to a 1-5 scale? Which, if any of these, would invalidate the faithful application of the registry? Would all of them together? It seems to the only truly objective way to rate compliance is to have an all or nothing approach: either you do exactly what you say you do, or you didn’t follow the registry. Of course, then we are lumping “p-value fishing” in the same category as applying a better a statistical model or changing the wording of a survey question.
This bring me to my second point, which is a concern that giving people a grade for faithfully sticking to a registry could lead to people conducting sub-optimal research — and stifle creativity — out of fear that it will cost them their “A” registry-faithfulness grade. To take but one example, those of us who use survey experiments have long been taught to pre-test questions precisely because sometime some of the ideas we have when sitting at our desks don’t work in practice. So if someone registers a particular technique for inducing an emotional response and then runs a pre-test and figures out their technique is not working, do we really want the researcher to use the sub-optimal design in order to preserve their faithfulness to the registered design? Or consider a student who plans to run a field experiment in a foreign country that is based on the idea that certain last names convey ethnic identity. What happens if the student arrives in the field and learns that this assumption was incorrect? Should the student stick with the bad research design to preserve the ability to publish in the “registry faithful” section of JEPS? Moreover, research sometimes proceeds in fits and spurts. If as a graduate student I am able to secure funds to conduct experiments in country A but later as a faculty member can secure funds to replicate these experiments in countries B and C as well, should I fear including the results from country A in a comparative analysis because my original registry was for a single country study? Overall, I think we have to be careful about assuming that we can have everything about a study figured out at the time we submit a registry design, and that there will be nothing left for us to learn about how to improve the research — or that there won’t be new questions that can be explored with previously collected data — once we start implementing an experiment.
At this point a fair critique to raise is that the points in preceding paragraph could be taken as an indictment of registries generally. Here we venture more into simply a point of view, but I believe that there is a difference between asking people to document what their original plans were and giving them a chance in their own words — if they choose to do so — to explain how their research project evolved as opposed to having to deal with a public “grade” of whatever form that might take. In my mind, the former is part of producing transparent research, while the latter — however well intentioned — could prove paralyzing in terms of making adjustments during the research process or following new lines of interesting research.
This brings me to my final concern, which is that untenured faculty would end up feeling the most pressure in this regard. For tenured faculty, a publication without the requisite asterisks noting registry compliance might not end up being too big a concern — although I’m not even sure of that — but I could easily imagine junior faculty being especially worried that publications without registry asterisks could be held against them during tenure considerations.
The bottom line is that registries bring with them a host of benefits — as Jamie has nicely laid out above — but we should think carefully about how to best maximize those benefits in order to minimize new costs. Even if we could agree on how to rate a proposal in terms of faithfulness to registry design, I would suggest caution in trying to integrate ratings into the publication process.
The views expressed here are mine alone and do not represent either the Journal of Experimental Political Science or the APSA Organized Section on Experimental Research Methods.
Heading image: Interior of Rijksmuseum research library. Rijksdienst voor het Cultureel Erfgoed. CC-BY-SA-3.0-nl via Wikimedia Commons.
Despite what many of my colleagues think, being a journal editor is usually a pretty interesting job. The best part about being a journal editor is working with authors to help frame, shape, and improve their research. We also have many chances to honor specific authors and their work for being of particular importance. One of those honors is the Miller Prize, awarded annually by the Society for Political Methodology for the best paper published in Political Analysis the proceeding year.
“..the article tackles an difficult and pervasive problem—interference among units—in a novel and compelling way. Rather than treating spillover effects as a nuisance to be marginalized over or, worse, ignored, Bowers et al. use them as an opportunity to test substantive questions regarding interference … Their work also brings together causal inference and network analysis in an innovative and compelling way, pointing the way to future convergence between these domains.”
In other words, this is an important contribution to political methodology.
I recently posed a number of question to one of the authors of the Miller Prize paper, Jake Bowers, asking him to talk more about this paper and its origins.
Jake Bowers: Let me provide a little background for readers not already familiar with randomization-based statistical inference.
Randomized designs provide clear answers to two of the most common questions that we ask about empirical research: The Interpretation Question: “What does it mean that people in group A act differently from people in group B?” and The Information Question: “How precise is our summary of A-vs-B?” (Or, more defensively, “Do we really have enough information to distinguish A from B?”).
If we have randomly assigned some A-vs-B intervention, then we can answer the interpretation question very simply: “If group A differs from group B, it is only because of the A-vs-B intervention. Randomization ought to erase any other pre-existing differences between groups A and B.”
In answering the information question, randomization alone also allows us to characterize other ways that the experiment might have turned out: “Here are all of the possible ways that groups A and B could differ if we re-randomized the A-vs-B intervention to the experimental pool while entertaining the idea that A and B do not differ. If few (or none) of these differences is as large as the one we observe, we have a lot of information against the idea that A and B do not differ. If many of these differences are as large as the one we see, we don’t have much information to counter the argument that A and B do not differ.”
Of course, these are not the only questions one should ask about research, and interpretation should not end with knowing that an input created an output. Yet, these concerns about meaning and information are fundamental and the answers allowed by randomization offer a particularly clear starting place for learning from observation. In fact, many randomization-based methods for summarizing answers to the information question tend to have validity guarantees even with small samples. If we really did repeat the experiment all the possible ways that it could have been done, and repeated a common hypothesis test many times, we would reject a true null hypothesis no more than α% of the time even if we had observed only eight people (Rosenbaum 2002, Chap 2).
In fact a project with only eight cities impelled this paper. Costa Panagopoulos had administered a field experiment of newspaper advertising and turnout to eight US cities, and he and I began to discuss how to produce substantively meaningful, easy to interpret, and statistically valid, answers to the question about the effect of advertising on turnout. Could we hypothesize that, for example, the effect was zero for three of the treated cites, and more than zero for one of the treated cites? The answer was yes.
I realized that hypotheses about causal effects do not need to be simple, and, furthermore, they could represent substantive, theoretical models very directly. Soon, Mark Fredrickson and I started thinking about substantive models in which treatment given to one city might have an effect on another city. It seemed straightforward to write down these models. We had read Peter Aronow’s and Paul Rosenbaum’s papers on the sharp null model of no effects and interference, and so we didn’t think we were completely off base to imagine that, if we side-stepped estimation of average treatment effects and focused on testing hypotheses, we could learn something about what we called “models of interference”. But, we had not seen this done before. So, in part because we worried about whether we were right about how simple it was to write down and test hypotheses generated from models of spillover or interference between units, we wrote the “Reasoning about Interference” paper to see if what we were doing with Panagopoulos’ eight cities would scale, and whether it would perform as randomization-based tests should. The paper shows that we were right.
R. Michael Alvarez: In your paper, you focus on the “no interference” assumption that is widely discussed in the contemporary literature on causal models. What is this assumption and why is it important?
Jake Bowers: When we say that some intervention, (Zi), caused some outcome for some person, (i), we often mean that the outcome we would have seen for person (i) when the intervention is not-active, (Zi=0) — written as (y{i,Zi=0}) — would have been different from the outcome we would have seen if the intervention were active for that same person (at that same moment in time), (Zi=1), — written as (y{i,Z_i=1}). Most people would say that the treatment had an effect on person (i) when (i) would have acted differently under the intervention than under the control condition such that y{i,Zi=1} does not equal y{i,Zi=0}. David Cox (1958) noticed that this definition of causal effects involves an assumption that an intervention assigned to one person does not influence the potential outcomes for another person. (Henry Brady’s piece, “Causation and Explanation in Social Science” in the Oxford Handbook of Political Methodology provides an excellent discussion of the no-interference assumption and Don Rubin’s formalization and generalization of Cox’s no-interference assumption.)
As an illustration of the confusion that interference can cause, imagine we had four people in our study such that (i in {1,2,3,4}). When we write that the intervention had an effect for person (i=1),(y{i=1,Z1=1} does not equal y{i=1,Z1=0}), we are saying that person 1 would act the same when (Z{i=1}=1) regardless of how intervention was assigned to any other person such that
If we do not make this assumption then we cannot write down a treatment effect in terms of a simple comparison of two groups. Even if we randomly assigned the intervention to two of the four people in this little study, we would have six potential outcomes per person rather than only two potential outcomes (you can see two of the six potential outcomes for person 1 in above). Randomization does not help us decide what a “treatment effect” means and six counterfactuals per person poses a challenge for the conceptualization of causal effects.
So, interference is a problem with the definition of causal effects. It is also a problem with estimation. Many folks know about what Paul Holland (1986) calls the “Fundamental Problem of Causal Inference” that the potential outcomes heuristic for thinking about causality reveals: we cannot ever know the causal effect for person (i) directly because we can never observe both potential outcomes. I know of three main solutions for this problem, each of which have to deal with problems of interference:
Jerzy Neyman (1923) showed that if we change our substantive focus from individual level to group level comparisons, and to averages in particular, then randomization would allow us to learn about the true, underlying, average treatment effect using the difference of means observed in the actual study (where we only see responses to intervention for some but not all of the experimental subjects).
Don Rubin (1978) showed a Bayesian predictive approach — a probability model of the outcomes of your study and a probability model for the treatment effect itself allows you can predict the unobserved potential outcomes for each person in your study and then take averages of those predictions to produce an estimate of the average treatment effect.
Ronald Fisher (1935) suggested another approach which maintained attention on the individual level potential outcomes, but did not use models to predict them. He showed that randomization alone allows you to test the hypothesis of “no effects” at the individual level. Interference makes it difficult to interpret Neyman’s comparisons of observed averages and Rubin’s comparison of predicted averages as telling us about causal effects because we have too many possible averages.
It turns out that Fisher’s sharp null hypothesis test of no effects is simple to interpret even when we have unknown interference between units. Our paper starts from that idea and shows that, in fact, one can test sharp hypotheses about effects rather than only no effects.
Note that there has been a lot of great recent work trying to define and estimate average treatment effects recently by folks like Cyrus Samii and Peter Aronow, Neelan Sircar and Alex Coppock, Panos Toulis and Edward Kao, Tyler Vanderweele, Eric Tchetgen Tchetgen and Betsy Ogburn, Michael Sobel, and Michael Hudgens, among others. I also think that interference poses a smaller problem for Rubin’s approach in principle — one would add a model of interference to the list of models (of outcomes, of intervention, of effects) used to predict the unobserved outcomes. (This approach has been used without formalization in terms of counterfactuals in both the spatial and networks models worlds.) One might then focus on posterior distributions of quantities other than simple differences of averages or interpret such differences reflecting the kinds of weightings used in the work that I gestured to at the start of this paragraph.
R. Michael Alvarez: How do you relax the “no interference” assumption in your paper?
Jake Bowers: I would say that we did not really relax an assumption, but rather side-stepped the need to think of interference as an assumption. Since we did not use the average causal effect, we were not facing the same problems of requiring that all potential outcomes collapse down to two averages. However, what we had to do instead was use what Paul Rosenbaum might call Fisher’s solution to the fundamental problem of causal inference. Fisher noticed that, even if you couldn’t say that a treatment had an effect on person (i), you could ask whether we had enough information (in our design and data) to shed light on a question about whether or not the treatment had an effect on person (i). In our paper, Fisher’s approach meant that we did not need to define our scientifically interesting quantity in terms of averages. Instead, we had to write down hypotheses about no interference. That is, we did not really relax an assumption, but instead we directly modelled a process.
Rosenbaum (2007) and Aronow (2011), among others, had noticed that the hypothesis that Fisher is most famous for, the sharp null hypothesis of no effects, in fact does not assume no interference, but rather implies no interference (i.e., if the treatment has no effect for any person, then it does not matter how treatment has been assigned). So, in fact, the assumption of no interference is not really a fundamental piece of how we talk about counterfactual causality, but a by-product of a commitment to the use of a particular technology (simple comparisons of averages). We took a next step in our paper and realized that Fisher’s sharp null hypothesis implied a particular, and very simple, model of interference (a model of no interference). We then set out to see if we could write other, more substantively interesting models of interference. So, that is what we show in the paper: one can write down a substantive theoretical model of interference (and of the mechanism for an experimental effect to come to matter for the units in the study) and then this model can be understood as a generator of sharp null hypotheses, each of which could be tested using the same randomization inference tools that we have been studying for their clarity and validity previously.
R. Michael Alvarez: What are the applications for the approach you develop in your paper?
Jake Bowers: We are working on a couple of applications. In general, our approach is useful as a way to learn about substantive models of the mechanisms for the effects of experimental treatments.
For example, Bruce Desmarais, Mark Fredrickson, and I are working with Nahomi Ichino, Wayne Lee, and Simi Wang on how to design randomized experiments to learn about models of the propagation of treatments across a social network. If we think that an experimental intervention on some subset of Facebook users should spread in some certain manner, then we are hoping to have a general way to think about how to design that experiment (using our approach to learn about that propagation model, but also using some of the new developments in network-weighted average treatment effects that I referenced above). Our very early work suggests that, if treatment does propagate across a social network following a common infectious disease model, that you might prefer to assign relatively few units to direct intervention.
In another application, Nahomi Ichino, Mark Fredrickson, and I are using this approach to learn about agent-based models of the interaction of ethnicity and party strategies of voter registration fraud using a field experiment in Ghana. To improve our formal models, another collaborator, Chris Grady, is going to Ghana to do in-depth interviews with local party activists this fall.
R. Michael Alvarez: Political methodologists have made many contributions to the area of causal inference. If you had to recommend to a graduate student two or three things in this area that they might consider working on in the next year, what would they be?
Jake Bowers: About advice for graduate students: Here are some of the questions I would love to learn about.
How should we move from formal, equilibrium-oriented, theories of behavior to models of mechanisms of treatment effects that would allow us to test hypotheses and learn about theory from data?
How can we take advantage of estimation-based procedures or procedures developed without specific focus on counterfactual causal inference if we want to make counterfactual causal inferences about models of interference? How should we reinterpret or use tools from spatial analysis like those developed by Rob Franzese and Jude Hayes or tools from network analysis like those developed by Mark Handcock to answer causal inference questions?
How can we provide general advice about how to choose test-statistics to summarize the observable implications of these theoretical models? We know that the KS-test used in our article is pretty low-powered. And we know from Rosenbaum (Chap 2, 2002) that certain classes of test statistics have excellent properties in one-dimension, but I wonder about general properties of multi-parameter models and test statistics that can be sensitive to multi-way differences in distribution between experimental groups.
How should we apply ideas from randomized studies to the observational world? What does adjustment for confounding/omitted variable bias (by matching or “controlling for” or weighting) mean in the context of social networks or spatial relations? How should we do and judge such adjustment? Would might what Rosenbaum-inspired sensitivity analysis or Manski-inspired bounds analysis might mean when we move away from testing one parameter or estimating one quantity?
R. Michael Alvarez: You do a lot of work with software tool development and statistical computing. What are you working on now that you are most excited about?
Jake Bowers: I am working on two computationally oriented projects that I find very exciting. The first involves using machine learning/statistical learning for optimal covariance adjustment in experiments (with Mark Fredrickson and Ben Hansen). The second involves collecting thousands of hand-drawn maps on Google maps as GIS objects to learn about how people define and understand the places where they live in Canada, the United Kingdom, and the United States (with Cara Wong, Daniel Rubenson, Mark Fredrickson, Ashlea Rundlett, Jane Green, and Edward Fieldhouse).
When an experimental intervention has produced a difference in outcomes, comparisons of treated to control outcomes can sometimes fail to detect this effect, in part, because the outcomes themselves are naturally noisy in comparison to the strength of the treatment effect. We would like to reduce the noise that is unrelated to treatment (say, remove the noise related to background covariates, like education) without ever estimating a treatment effect (or testing a hypothesis about a treatment effect). So far, people shy away from using covariates for precision enhancement of this type because of every model in which they soak up noise with covariates is also a model in which they look at the p-value for their treatment effect. This project learns from the growing literature in machine learning (aka statistical learning) to turn specification of the covariance adjustment part of a statistical model over to an automated system focused on the control group only which thus bypasses concerns about data snooping and multiple p-values.
The second project involves using Google maps embedded in online surveys to capture hand-drawn maps representing how people respond when asked to draw the boundaries of their “local communities.” So far we have over 7000 such maps from a large survey of Canadians, and we plan to have data from this module carried on the British Election Study and the US Cooperative Congressional Election Study within the next year. We are using these maps and associated data to add to the “context/neighborhood effects” literature to learn how psychological understandings of place by individuals relates to Census measurements and also to individual level attitudes about inter-group relations and public goods provision.
Headline image credit: Abstract city and statistics. CC0 via Pixabay.
Introduction from Michael Alvarez, co-editor of Political Analysis:
Questions about data access, research transparency and study replication have recently become heated in the social sciences. Professional societies and research journals have been scrambling to respond; for example, the American Political Science Association established the Data Access and Research Transparency committee to study these issues and to issue guidelines and recommendations for political science. At Political Analysis, the journal that I co-edit with Jonathan N. Katz, we require that all of the papers we publish provide replication data, typically before we send the paper to production. These replication materials get archived at the journal’s Dataverse, which provides permanent and easy access to these materials. Currently we have over 200 sets of replication materials archived there (more arriving weekly), and our Dataverse has seen more than 13,000 downloads of replication materials.
Due to the interest in replication, data access, and research transparency in political science and other social sciences, I’ve asked a number of methodologists who have been front-and-center in political science with respect to these issues to provide their thoughts and comments about what we do in political science, how well it has worked so far, and what the future might hold for replication, data access, and research transparency. I’ll also be writing more about what we have done at Political Analysis.
The first of these discussions are reflections from Nathaniel Beck, Professor of Politics at NYU, who is primarily interested in political methodology as applied to comparative politics and international relations. Neal is a former editor of Political Analysis, chairs our journal’s Advisory Board, and is now heading up the Society for Political Methodology’s own committee on data access and research transparency. Neal’s reflections provide some interesting perspectives on the importance of replication for his research and teaching efforts, and shed some light more generally on what professional societies and journals might consider for their policies on these issues.
Research replication in social science: reflections from Nathaniel Beck
Replication and data access has become a hot topic throughout the sciences. As a former editor of Political Analysis and the chair of the Society for Political Methodology‘s Data Access and Research Transparency (DA-RT) committee, I have been thinking about these issues a lot lately. But here I simply want to share a few recent experiences (two happy, one at this moment less so) which have helped shape my thinking on some of these issues. I note that in none of these cases was I concerned that the authors had done anything wrong, though of course I was concerned about the sensitivity of results to key assumptions.
The first happy experience relates to an interesting paper on the impact of having an Islamic mayor on educational outcomes in Turkey by Meyerson published recently in Econometrica. I first heard about the piece from some students, who wanted my opinion on the methodology. Since I am teaching a new (for me) course on causality, I wanted to dive more deeply into the regression discontinuity design (RDD) as used in this article. Coincidentally, a new method for doing RDD was presented at the recent (2014) meetings of the Society for Political Methodology by Rocio Titiunik. I want to see how her R code worked with interesting comparative data. All recent Econometrica articles are linked to both replication and supplementary materials on the Econometrica web site. It took perhaps 15 minutes to make sure that I could run Stata on my desktop and get the same results as in the article. So thanks to both Meyerson and Econometrica for making things so easy.
I gained from this process, getting a much better feel for real RDD data analysis so I can say more to my students than “the math is correct.” My students gain by seeing a first rate application that interests them (not a toy, and not yet another piece on American elections). And Meyerson gains a few readers who would not normally peruse Econometrica, and perhaps more cites in the ethnicity literature. And thanks to Titiunik for making her R code easily accessible.
The second happy experience was similar to the first, but also opened my eyes to my own inferior practice. At the same Society meetings, I was the discussant on a paper by Grant and Lebo on using fractional integration methods. I had not thought about such methods in a very long time, and believed (based on intuition and no evidence to the contrary) that using fractional integration methods led to no changes in substantive findings. But clearly one should base arguments on evidence and not intuition. I decided to compare the results of a fractional integration study by Box-Steffensmeier and Smith with the results of a simpler analysis. Their piece had a footnote saying the data were available through the ICPSR (excellent by the standards of 1998). Alas, on going to the ICPSR web site I could not find the data (noting that the lots of things have happened since 1998 and who knows if my search was adequate). Fortunately I know Jan so I wrote to her, and she kindly replied that the data were on her Dataverse at Harvard. A minute later I had the data and was ready to try to see if my intuitions might indeed be supported by evidence.
Typing on Keyboard – Male Hand by Dave Dugdale. CC BY-SA 2.0 via Flickr.
This experience made me think: could someone find my replication data sets? For as long as I can remember (at least back to 1995), I always posted my replication data sets somewhere. Articles written until 2003 sent readers my public ftp site at UCSD. But UCSD has changed the name and file structure of that server several times since 2003, and for some reason they did not feel obligated to keep my public ftp site going (and I was not worried enough about replication to think of moving that ftp site to NYU). Fortunately I can usually find the replication files if anyone writes me, and if I cannot, my various more careful co-authors can find the data. But I am sure that I am not the only person to have replication data on obsolete servers. Thankfully Political Analysis has required me to put my data on the Political Analysis Dataverse so I no longer have to remember to be a good citizen. And my resolution is to get as many replication data sets from old pieces on my own Harvard Dataverse. I will feel less hypocritical once that is done. It would be very nice if other authors emulated Jan!
The possibly less happy outcome relates to the recent article in PNAS on a Facebook experiment on social contagion. The authors, in a footnote, said that replication data was available by writing to the authors. I wrote twice, giving them a full month, but heard nothing. I then wrote to the editor of PNAS who informed me that the lead author had both been on vacation and was overwhelmed with responses to the article. I am promised that the check is in the mail.
What editor wants to be bothered by fielding inquiries about replication data sets? What author wants to worry about going on vacation (and forgetting to set a vacation message)? How much simpler the world would have been for the authors, editor, and me, if PNAS simply followed the good practice of Political Analysis, the American Journal of Political Science, the Quarterly Journal of Political Science, Econometrica, and (if rumors are correct) soon the American Political Science Review of demanding that authors post, either on the journal web site or the journal Dataverse, all replication materials before an article is actually published? Why does not every journal do this?
A distant second best is to require authors to post their replication on their personal website. As we have seen from my experience, this often leads to lost or non-working URLs. While the simple solution here is the Dataverse, surely at a minimum authors should provide a standard Document Object Identifier (DOI) which should persist even as machine names change. But the Dataverse solution does this, and so much more, that it seems odd in this day and age for all journals not to use this solution. And we can all be good citizens and put our own pre-replication standard datasets on our own Dataverses. All of this is as easy (and maybe) easier than maintaining private data web pages, and one can rest easy that one’s data will be available until either Harvard goes out of business or the sun burns out.
Featured image: BalticServers data center by Fleshas CC-BY-SA-3.0 via Wikimedia Commons.
One of the most common questions that scholars confront is trying to find the right journal for their research papers. When I go to conferences, often I am asked: “How do I know if Political Analysis is the right journal for my work?”
This is an important question, in particular for junior scholars who don’t have a lot of publishing experience — and for scholars who are nearing important milestones (like contract renewal, tenure, and promotion). In a publishing world where it may take months for an author to receive an initial decision from a journal, and then many additional months if they need to revise and resubmit their work to one or more subsequent journals, selecting the most appropriate journal can be critical for professional advancement.
So how can a scholar try to determine which journal is right for their work?
The first question an author needs to ask is how suitable their paper is for a particular journal. When I meet with my graduate students, and we talk about potential publication outlets for their work, my first piece of advice is that they should take a close look at the last three or four issues of the journals they are considering. I’ll recommend that they look at the subjects that each journal is focusing on, including both substantive topics and methodological approaches. I also tell them to look closely at how the papers appearing in those journals are structured and how they are written (for example, how long the papers typically are, and how many tables and figures they have). The goal is to find a journal that is currently publishing papers that are most closely related to the paper that the student is seeking to publish, as assessed by the substantive questions typically published, the methodological approaches generally used, paper framing, and manuscript structure.
Potential audience is the second consideration. Different journals have different readers — meaning that authors can have some control over who might be exposed to their paper when they decide which journals to target for their work. This is particularly true for authors who are working on highly interdisciplinary projects, where they might be able to frame their paper for publication in related but different academic fields. In my own work on voting technology, for example, some of my recent papers have appeared in journals that have their primary audience in computer science, while others have appeared in more typical political science journals. So authors need to decide in many cases which audience they want to appeal two, and make sure that when they submit their work to a journal that appeals to that audience that the paper is written in an appropriate manner for that journal.
Peer reviewer for Scientific Review by Center for Scientific Review. Public domain via Wikimedia Commons.
However, most authors will want to concentrate on journals in a single field. For those papers, a third question arises: whether to target a general interest journal or a more specialized field journal. This is often a very subjective question, as it is quite hard to know prior to submission whether a particular paper will be interesting to the editors and reviewers of a general interest journal. As general interest journals often have higher impact factors (I’ll say more about impact factors next), many authors will be drawn to submit their papers to general interest journals even if that is not the best strategy for their work. Many authors will “start high”, that is begin with general interest journals, and then once the rejection letters pile up, they will move to the more specialized field journals. While this strategy is understandable (especially for authors who are nearing promotion or tenure deadlines), it may also be counterproductive — the author will likely face a long and frustrating process getting their work published, if they submit first to general interest journals, get the inevitable rejections, and then move to specialized field journals. Thus, my advice (and my own practice with my work) is to avoid that approach, and to be realistic about the appeal of the particular research paper. That is, if your paper is going to appeal only to readers in a narrow segment of your discipline, then send it to the appropriate specialized field journal.
A fourth consideration is the journal’s impact factor. Impact factors are playing an increasingly important role in many professional decisions, and they may be a consideration for many authors. Clearly, an author should generally seek to publish their work in journals that have higher impact than those that are lower impact. But again, authors should try to be realistic about their work, and make sure that regardless of the journal’s impact factor that their submission is appropriate for the journal they are considering.
Finally, authors should always seek the input of their faculty colleagues and mentors if they have questions about selecting the right journal. And in many fields, journal editors, associate editors, and members of the journal’s editorial board will often be willing to give an author some quick and honest advice about whether a particular paper is right for their journal. While many editors shy away from giving prospective authors advice about a potential submission, giving authors some brief and honest advice can actually save the editor and the journal a great deal of time. It may be better to save the author (and the journal) the time and effort that might get sunk into a paper that has little chance at success in the journal, and help guide the author to a more appropriate journal.
Selecting the right journal for your work is never an easy process. All scholars would like to see their work published in the most widely read and highest impact factor journals in their field. But very few papers end up in those journals, and authors can get their work into print more quickly and with less frustration if they first make sure their paper is appropriate for a particular journal.
Heading image: OSU William Oxley Thompson Memorial Library Stacks by Ibagli. Public Domain via Wikimedia Commons.
I recently had the opportunity to talk with Lonna Atkeson, Professor of Political Science and Regents’ Lecturer at the University of New Mexico. We discussed her opinions about improving survey methodology and her thoughts about how surveys are being used to study important applied questions. Lonna has written extensively about survey methodology, and has developed innovative ways to use surveys to improve election administration (her 2012 study of election administration is a wonderful example).
In the current issue of Political Analysis is the Symposium on Advances in Survey Methodology, which Lonna and I co-edited; in addition to the five research articles in the Symposium, we wrote an introduction that puts each of the research articles in context and talks about the current state of research in survey methodology. Also, Lonna and I are co-editing the Oxford Handbook on Polling and Polling Methods, which is in initial stages of development.
It’s well-known that response rates for traditional telephone surveying have declined dramatically. What’s the solution? ow can survey researchers produce quality data given low response rates with traditional telephone survey approaches?
What we’ve learned about response rates is they are not the be all or end all as an evaluative tool for the quality of the survey, which is a good thing because response rates are ubiquitously low! There is mounting evidence that response rates per se are not necessarily reflective of problems in nonresponse. Nonresponse error appears to be more related to the response rate interacting with the characteristic of the nonrespondent. Thus, if survey topic salience leads to response bias then nonresponse error becomes a problem, but in and of itself response rate is only indirect evidence of a potential problem. One potential solution to falling response rates is to use mixed mode surveys and find the best contact and response option for the respondent. As polling becomes more and more sophisticated, we need to consider best contact and response methods for different types of sample members. Survey researchers need to be able to predict the most likely response option for the individual and pursue that strategy.
Mixed mode surveys use multiple methods to contact or receive information from respondents. Thus, mixed mode surveys involve both mixtures of data collection and communications with the respondent. For example, a mixed mode survey might contact sample members by phone or mail and then have them respond to a questionnaire over the Internet. Alternatively a mixed mode survey might allow for multiple forms of response. For example, sample frame members may be able to complete the interview over the phone, by mail, or on the web. Thus a respondent who does not respond over the Internet may in subsequent contact receive a phone call or a FTF visit or may be offered a choice of response mode on the initial contact.
When you see a poll or survey reported online or in the news media, how do you determine if the poll was conducted in a way that has produced reliable data? What indicates a high-quality poll?
This is a difficult question because all polls are not created equally and many reported polls might have problems with sampling, nonresponse bias, question wording, etc. The point being that there are many places where error creeps into your survey not just one and to evaluate a poll researchers like to think in terms of total survey error, but the tools for that evaluation are still in the development stage and is an area of opportunity for survey researchers and political methodologists. We also need to consider a total survey error approach in how survey context, which now varies tremendously, influences respondents and what that means for our models and inferences. This is an area for continued research. Nevertheless, the first criteria for examining a poll ought to be its transparency. Polling data should include information on who funded the poll, a copy of the instrument, a description of the sampling frame, and sampling design (e.g. probability, non-probability, the study size, estimates of sampling error for probability designs, information on any weighting of the data, and how and when the data were collected). These are basic criteria that are necessary to evaluate the quality of the poll.
Survey research is a rapidly changing environment with new methods for respondent contacting and responding. Perhaps the biggest change in the most recent decade is the move away from predominantly interviewer driven data collection methods (e.g. phone, FTF) to respondent driven data collection methods (e.g. mail, Internet, CASI), the greater use of mixed mode surveys, and the introduction of professional respondents who participate over long periods of time in discontinuous panels. We are just beginning to figure out how all these pieces fit together and we need to come up with better tools to assess the quality of data we are obtaining. The future of polling and its importance in the discipline, in marketing, and in campaigns will continue, and as academics we need to be at the forefront of evaluating these changes and their impact on our data. We tend to brush over the quality of data in favor of massaging the data statistically or ignoring issues of quality and measurement altogether. I’m hoping the changing survey environment will bring more political scientists into an important interdisciplinary debate about public opinion as a methodology as opposed to the study of the frequencies of opinions. To this end, I have a new Oxford Handbook, along with my co-editor Mike Alvarez, on polling and polling methods that will take a closer look at many of these issues and be a helpful guide for current and future projects.
In your recent research on election administration, you use polling techniques as tools to evaluate elections. What have you learned from these studies, and based on your research what do you see are issues that we might want to pay close attention to in this fall’s midterm elections in the United States?
We’ve learned so much from our election administration work about designing polling places, training poll workers, mixed mode surveys and more generally evaluating the election process. In New Mexico, for example, we have been interviewing both poll workers and voters since 2006, giving us five election cycles, including 2014, that provide an overall picture of the current state of election administration and how it’s doing relative to past election cycles. Our multi-method approach provides continuous evaluation, review, and improvement to New Mexico elections. This fall I think there are many interesting questions. We are interested in some election reform questions about purging voter registration files, open primaries, the straight party ballot options and felon re-enfranchisement. We are also especially interested in how voters decide whether to vote early or on Election Day and on Election Day where they decide to vote if they are using voting convenience centers instead of precincts. This is an important policy question, but where we place vote centers might impact turnout or voter satisfaction or confidence. We are also very interested in election lines and their impact on voters. In 2012 we found that voters on average can fairly easily tolerate lines of about ½ an hour, but feel there are administrative problems when lines grow longer. We want to continue to drill down on this question and examine when lines deter voters or create poor experiences that reduce the quality of their vote experience.
Lonna Rae Atkeson is Professor of Political Science and Regents’ Lecturer at the University of New Mexico. She is a nationally recognized expert in the area of campaigns, elections, election administration, survey methodology, public opinion and political behavior and has written numerous articles, book chapters, monographs and technical reports on these topics. Her work has been supported by the National Science Foundation, the Pew Charitable Trusts, the JEHT Foundation, the Galisano Foundation, the Bernalillo County Clerk, and the New Mexico Secretary of State. She holds a BA in political science from the University of California, Riverside and a Ph.D. in political science from the University of Colorado, Boulder.
R. Michael Alvarez is a professor of Political Science at Caltech. His research and teaching focuses on elections, voting behavior, and election technologies. He is editor-in-chief of Political Analysis with Jonathan N. Katz.
Political Analysis chronicles the exciting developments in the field of political methodology, with contributions to empirical and methodological scholarship outside the diffuse borders of political science. It is published on behalf of The Society for Political Methodology and the Political Methodology Section of the American Political Science Association. Political Analysis is ranked #5 out of 157 journals in Political Science by 5-year impact factor, according to the 2012 ISI Journal Citation Reports. Like Political Analysis on Facebook and follow @PolAnalysis on Twitter.
Subscribe to the OUPblog via email or RSS.
Subscribe to only politics and political science articles on the OUPblog via email or RSS.
Empirical work in political science must be based on strong scientifically-accurate measurements. However, the problem of measurement error hasn’t been sufficiently addressed. Recently, Willem Saris and Daniel Oberski’s Survey Quality Prediction software was developed to better predict reliability and method variance, and is receiving the 2014 Warren J. Mitofsky Innovators Award from the American Association for Public Opinion Research. I sat down with Political Analysis-contributor Daniel Oberski, a postdoctoral researcher in latent variable modeling and survey methodology at Tilburg University’s department of methodology and statistics, to discuss the software, surveys, latent variables, and interdisciplinary research.
Survey questions are important measurement instruments, whose design and use we need to approach scientifically. After all, even though nowadays we have “big data” and neuroimaging, one of the most effective ways of finding things out about people remains to just ask them (see also this keynote lecture by Mick Couper). But survey questions are not perfect: we must recognize and account for measurement error. That is the main motivation for SQP.
Willem started working on estimating measurement error together with the University of Michigan’s Frank Andrews in the late 1980s and has made it his life’s work to gather as much information as possible about the quality of different types of survey questions. In physics, it is quite customary to dedicate one’s career to measurement; my father, for example, spent the better part of his measuring how background radiation might interfere with CERN experiments – just so this interference could later be corrected for. In the social sciences, this kind of project is rare. Thanks to Willem’s decades of effort, however, we were able to perform a meta-analysis over the results of his experiments, linking questions’ characteristics to their reliability so that this could be accounted for.
We then created a web application that allows the user to predict reliability and method variance from a question’s characteristics, based on a meta-analysis of over 3000 questions. The goal is to allow researchers to recognize measurement error, choose the best measurement instruments for their purpose, and account for the effects of errors in their analyses of interest.
How can survey researchers use the SQP software package, and why is it an important tool for survey researchers?
People who use surveys are often (usually?) interested in relationships between different variables, and measurement error can wreak havoc on such estimates. There are two possible reactions to this problem.
The first is hope: maybe measurement error is not that bad, or perhaps bias will be in some known direction, for example towards zero. Unfortunately, we see in our experiments that this hope is, on average, unfounded.
The second possibility is to estimate measurement error so that it can be accounted for. Accounting for measurement error can be done using any of a number of well-known and easily available methods, such as structural equation modeling or Bayesian priors. The tricky part lies in estimating the amount of measurement error. Not every researcher will have the resources and opportunity to conduct a multitrait-multimethod experiment, for example. Another issue is how to properly account for the additional uncertainties involved with the measurement error correction itself.
This is where SQP comes in.
Anybody can code the survey question they wish to analyze on a range of characteristics and obtain an estimate of the reliability and amount of common method bias in that question. An estimate of the prediction uncertainty is also given. This information can then be used to correct estimates of relationships between variables for measurement error.
The SQP software package seems to be part of a general trend, where survey researchers are developing tools and technologies to automate and improve aspects of survey design and implementation. What other aspects of survey research do you see being improved by other tools in the near future?
SQP deals with measurement error, but there is also nonresponse error, coverage error, editing/processing error, etc. Official statistics agencies as well as commercial survey companies have been developing automated tools for survey implementation since the advent of computer-assisted data collection. More recent is the incorporation of survey experiments to aid decisions on survey design. This is at least partly driven by more easily accessible technology, by the growth of survey methodology as it is being discovered by other fields, and by some US institutions’ insistence on survey experiments. All of this means that we are gathering evidence at a fast rate on a wide range of quality issues in surveys, even if that evidence is not always as accessible as we would like it to be.
I can very well imagine that meta-analyses over this kind of information are eventually encoded into expert systems. These would be like SQP, but for nonresponse, noncoverage, and so on. This could allow for the kind of quality evaluations and predictions on all aspects of survey research that is necessary for social science. It would be a large project, but it is doable.
Political Analysis has published two of your papers that focus on “latent variables.” What is the connection between your survey methodology research and your work on latent variables?
I have heard it claimed that all variables of interest are observed and latent variables are useful as an intermediary tool at best. I think the opposite is true. Social science is so difficult because the variables we observe are almost never the variables of actual interest: there is always measurement error. That is why we need latent variable models to model relationships between the variables of actual interest.
On the one hand, latent variable models have been mostly developed within fields that traditionally were not overly concerned with representativeness, although that could be changing now. On the other hand, after some developments in the early 1960s at the US Census Bureau, the survey methodology field has some catching up to do on advances in measurement error modeling since those times. Part of what I do involves “introducing” different fields’ methods to each other so both are improved. Another part, which concerns the papers in PA, is about solving some of the unique, new problems that come up when you do that.
For example, when correcting measurement error (misclassification) by fixing an estimate of the error rates in maximum likelihood analysis, how does one properly account for the fact that this error rate is itself only an estimate, as it would be when obtained from SQP? We dealt with this for linear models in a paper with Albert Satorra in the journal Structural Equation Modeling, while a PA paper, on which Jeroen Vermunt and I are co-authors with our PhD student Zsuzsa Bakk, deals with categorical latent and observed variables. Another problem in survey methodology is how to decide that groups are “comparable enough” for the purposes at hand (a.k.a. “comparability”, “equivalence”, or “invariance”). I introduced a tool for looking at this problem using latent variable models in this PA paper.
Your research is quite interdisciplinary. What advice would you give to graduate students who are interested in work on interdisciplinary topics like you have? Any tips for how they might seek to publish their work?
I never really set out to be interdisciplinary on purpose. To some extent it is because I am interested in everything, and to another it is all part of being a statistician. Tukey supposedly said, “you get to play in everyone’s backyard,” and whether or not he really said it, that is also what I love about it.
I am only in the beginning of my career (I hope!) and not sure I am in a position to hand out any sage advice. But one tip might be: don’t assume something is not related to what you know about just because it sounds different. Ask yourself, “how would what this person is saying translate into my terms?” It helps to master, as much as possible, some general tool for thinking about these things. Mine is structural equation models, latent variable models. But any framework should work just as well: hierarchical Bayesian inference, counterfactuals, missing data, experimental design, randomization inference, graphical models, etc. Each of these frameworks, when understood thoroughly, can serve as a general language for understanding what is being said, and once you get something down in your own language, you usually have some immediate insights about the problem or at least the tools to get them.
As for publishing, I imagine my experience is rather limited relative to a typical graduate student’s advisors’. From what I can tell so far it is mostly about putting yourself in the place of your audience and understanding what makes your work interesting or useful for them specifically. An “interdisciplinary” researcher is by definition a bit of an outsider. That makes it all the more important to familiarize yourself intimately with the journal and with the wider literature in that field on the topic and closely related topics, and show how your work connects with that literature. This way you do not just “barge in” but can make a real contribution to the discussion being held in that field. At Political Analysis I received some help from the reviewers and editor in doing that and I am grateful for it; this ability to welcome work that borrows from “outside” and see its potential for the field is a real strength of the journal.
Daniel Oberski is a postdoctoral researcher at the
R. Michael Alvarez is a professor of Political Science at Caltech. His research and teaching focuses on elections, voting behavior, and election technologies. He is editor-in-chief of Political Analysis with Jonathan N. Katz.
Political Analysis chronicles the exciting developments in the field of political methodology, with contributions to empirical and methodological scholarship outside the diffuse borders of political science. It is published on behalf of The Society for Political Methodology and the Political Methodology Section of the American Political Science Association. Political Analysis is ranked #5 out of 157 journals in Political Science by 5-year impact factor, according to the 2012 ISI Journal Citation Reports. Like Political Analysis on Facebook and follow @PolAnalysis on Twitter.
Subscribe to the OUPblog via email or RSS.
Subscribe to only politics and political science articles on the OUPblog via email or RSS.
By R. Michael Alvarez
After my co-editor, Jonathan N. Katz, and I took over editorship of Political Analysis in January 2010, one of our primary goals was to extend the readership and intellectual reach of our journal. We wished to grow our readership internationally, and to also deepen our reach outside of political science, into other social sciences.
0 Comments on Political Analysis and social media: A case study for journals as of 1/1/1900
Academic publishing is not as simple as it may appear. University presses such as Oxford and Fordham range from large to small; for-profit publishers such as Wiley and Elsevier must appeal to both academics and shareholders; start-ups such as Academia.edu and WriteLatex are fulfilling smaller services; and niche publishers, such as Hurst, offer tremendous depth and breadth of specific subject areas.
The post Change in publishing: A Q&A with Michael Dwyer appeared first on OUPblog.