Methods War: How Ideas Matter within Political Science

I am not an axe-murderer.  I am just an historical institutionalist who happens to work within comparative politics.

I work on the judiciary.  On social movements.  On micro-level politics.  I am interested in practices on the ground; in what happens at the informal level; in processes; in ideas.  It is unlikely that you will find me citing Schumpeter, although it is possible on an outside day. You are more likely to find me citing James Scott, Pierre Bourdieu, Emile Durkehim or Max Weber.  Call me a classicist.

I do in-depth interviews centering on people’s life stories around a political question at hand.  And I do political ethnography.  I collect government statistics when I am overseas, the types that appear in foreign languages and are not available on the internet.  I have constructed and conducted one national-level survey.  It is hard to write and conduct one’s own survey, particularly in a foreign language taking into consideration local customs and sensitivities.  The work that we do in qualitative, field work based political science is significant, rigorous, and thoughtful.

And, yet, as a qualitative researcher, my work is largely suspect within my sub-field.  I often wonder why.

I was thoroughly trained in qualitative field methods.  I was trained in the methodological issues, ethical problems, and practical application of qualitative field methods in practice on-research-site.  I was trained in these methods at several universities and through three Social Science Research Council international dissertation workshops.

Today, some U.S. political science journals are requiring that qualitative researchers attach an appendix of their field notes and/or other supporting data to articles that they submit for consideration for publication.  Some justify this movement as an attempt to make qualitative work both transparent and reproducible – as if transparency and reproducibility in the scientific method is something we qualitative researchers simply never considered before their brilliant idea; as if we were not trained and raised on Thomas Kuhn.

Good qualitative work is transparent.  It is transparent in the sense that the methods are discussed and the links between argument and evidence are elaborated along specific methodological criteria.  It is reproducible.  Any scholar can spend ten to twenty years of his or her life, learn qualitative methods, learn the necessary languages, live in the field, gather archival data (which is illegal to share under most circumstances), conduct interviews, and engage in years-long rigorous observations of a political context on-site.  That is what qualitative field research is.  If the work is laid out correctly, an outside qualitative scholar can quite easily evaluate it and argue yay or nay as to its relative accuracy, validity, and other issues relating to its relative methodological soundness.  If it is not laid out correctly, that is also easy for a qualitatively-trained scholar to observe.

This movement in U.S. political science journals is anti-intellectual in that it does not show respect for the various forms of deep and rigorous methodological training in which political scientists at different programs engage.  So, for example, anyone who does qualitative methods knows well that field notes are almost always confidential under human subjects provisions within one’s university and are thus – literally – illegal to share.

Faculty can be fired for sharing such materials.  So, our top-ranking U.S. political science journals, in their great wisdom and understanding of the broad range of their own discipline, are giving qualitative faculty the choice between publish-or-perish, or be fired for breaching the law by sharing materials that are confidential.  That kind of equation is called a hostile takeover.  This on top of the fact that these materials are proprietary work product that a scholar spends years collecting. Qualitative scholars cannot just turn to a pre-collected data set to do our work year after year, as valuable as that work is.  Our methods require that we (continually) collect the material on our own in the field, often in foreign languages, and under difficult living conditions.

Asking a quantitative scholar – usually – for his or her data set means asking him or her to provide a pre-established collection of material that he/she did not collect personally.

Asking a qualitative researcher to provide his or her (usually confidential) qualitatively collected data is asking him or her to hand over sometimes ten years’ worth of research in the field.  Sometimes twenty.  Sometimes thirty. For what?  So that someone else can play with it and try to prove him or her wrong on the fly without any country knowledge or time in the field; or, worse yet, with country knowledge, but no understanding of qualitative methods?

There is a cannibalistic quality to this move in political science.  It is ugly.  It is exceedingly unethical.  It has everything to do with individual egos, methodological control of the institutions of our discipline, and lack of power-sharing across methods.  In a number of cases, it – horrifyingly, for this day and age – also has to do with posturing amongst specific university political science programs.  Their posturing and competition does a disservice to the discipline.  It harms all of us.  It harms the integrity of the intellectual process.  None of it has anything to do with theory building.

A more traditional method of arriving at the question of rigor in a given qualitative article would be the, apparently passé, professional peer review process, in which qualitatively trained researchers review the work of other qualitatively trained researchers.

By qualitatively trained researchers, I do not mean those who claim, today, to do ‘mixed-methods’. ‘Mixed-methods,’ as far as I have been able to observe – call me an ethnographic observer within the discipline – means, roughly, ‘I do both math and statistics.’  Large-N is also not qualitative.  ‘Process-tracing’ is not qualitative unless it is qualitative.

As much as you may love both country and western, it is not the same as rock n’ roll.  You may love it all.  But few are equally moved by both ends of the spectrum to be able to teach them equally well.  If you are not one of the handful of scholars who has truly spent as much time on the ground in the field as in data sets, do not presume to claim to be me.  I do not claim to be you.

If you work on a concept derived from qualitative work, say, informal processes, cite the concept as it comes from the qualitative literature.  This is also required by law.  Then, do your quantitative work with it.  But do not claim that you are now a qualitative scholar, and that you can therefore legitimately represent (read that, displace) qualitative scholars in disciplinary institutions and journals.

Further Reading on E-International Relations

Tags: , , , , , , ,
  • Shannon

    How would you like to make your data open to assessment by others? Clearly, there are ethical issues with revealing information (such as interview subject names) that might violate the rights of your research subjects. But, science requires skeptical review of claims made by researchers. How would you suggest that field work and field notes be subject to outside review?

  • Patricia J. Woods

    Peer review is the most classic method of review of research across the social sciences and sciences. It is not true across the sciences that raw data has to be shared for skeptics to poke holes in. It is sometimes considered proprietary for many years. This move in Political Science is not for scientific merit or for theory building. It is for skeptics of qualitative research, writ large. That is more cannibalism and narrowness among those ruling the journals rather than about theory building. I would suggest a return to tradition: peer review by those trained in qualitative work. The point of peer review is not supposed to be to take down whole research methods, but to evaluate the merit of a work based upon the methodological criteria of its own methods. Only qualitative scholars are trained in this way. The reverse is also true. Presumably, quantitative scholars would not want qualitative scholars reviewing the merits of their work.

  • Patricia J. Woods

    Qualitative data cannot be made available in the same way that quantitative data can be. The rules for the data are different. These differences are simply facts of human subjects and ethics associated with these methods. Quantitative scholars in Political Science have to accept these differences. They cannot keep barring great qualitative research from the top journals of the discipline. It is fundamentally un-representative and anti-scientific. What we do is science, too, just different rules and different methods. They cannot legitimately keep shutting us out of the top journals of our discipline.

  • Shannon

    I did not say that the rules had to be the same. But you do have to subject your claims to review by others. How would you set up rules that allow for review by others? It is unrealistic to think everyone will just trust the claims you make just because you make them. So, I ask again, how would you make your data open to assessment by others?

  • Roger

    Please answer two questions.

    1. Can you provide a link to a political science journal that requires qualitative scholars to attach field notes as a condition for submitting an article for review?

    2. Can you provide a link to a political science journal that requires qualitative scholars to attach field notes as a condition for publication?

    I am not looking for articles by others that repeat the claim that journals do these things, I am looking for an actual valid link to an actual political science journal that has either of these requirements.

    Thank you in advance for responding.

  • williamjkelleherphd

    I hardily agree with everything Patricia says in defense of qualitative research, including the too obvious power play of the Physics Envy bunch to dominate the political science profession. But Shannon raises an important issue.

    The problem here is, when research notes must be kept private to protect the persons involved, how can one scholar scrutinize the claims made by another scholar based on the proprietary data?

    The extreme scenario question is, how can the political science profession guard against fraudulent clams being accredited as valid?

    In law, sometimes evidence is scrutinized “in camera.” That is, the lawyers present their sensitive evidence to the judge in chambers. The same rules of evidence are applied in this proceeding as would be applied in a public trial. In this way, bad or fraudulent evidence can be detected and exposed. Claims based on bad evidence can be dismissed, and the ruling can be made public without revealing the nature of the bad evidence.

    Before an in camera proceeding is conducted, the lawyers have to make their case as to why it is necessary. The political science profession could devise a set of rules for ordering in camera proceedings where a sound case has been made for its necessity. Experienced retired judges can be hired to conduct the proceedings, and apply the well-established rules of scholarship, such as for data collection, and the common law rules of evidence. Of course, variations on this model are possible. But the point is, the profession can guard against fraudulent research claims made in its name without dismissing in total qualitative research.

    William J. Kelleher, Ph.D.

  • Roger

    The silence here is deafening. Professor Woods, Professor Kelleher, or others who have claimed that political science journals require submission of field notes, please provide an active link to a single journal that does this.

Please Consider Donating

Before you download your free e-book, please consider donating to support open access publishing.

E-IR is an independent non-profit publisher run by an all volunteer team. Your donations allow us to invest in new open access titles and pay our bandwidth bills to ensure we keep our existing titles free to view. Any amount, in any currency, is appreciated. Many thanks!

Donations are voluntary and not required to download the e-book - your link to download is below.