Evaluating the Research Methods of Three Modern Classics of Comparative Politics

The main aim of this essay will be to explore and theoretically evaluate the research designs of three classics of comparative politics: Putnam’s case study method in Making Democracy Work, Linz’s small-N research design in “The Perils of Presidentialism” and Amorim Neto & Cox’s large-N statistical analysis in “Electoral Institutions, Cleavage Structures, and the Number of Parties”. To do this, I will split my essay into two separate sections. In the first section of the essay I intend to illustrate the strengths and weaknesses of the different comparative research designs, focusing exclusively on the types found in the three main texts under evaluation. After a short introduction, some important focuses, amongst others, will be on the objectives of case studies, small-N studies and large-N studies, their evolution throughout the 20^th century, when they are best used, what their purposes are and what they intend to achieve.

The second section of the essay will be a critical evaluation of how successful one particular text – Linz’s “The Perils of Presidentialism” – has been in the execution of the authors’ objectives, looking at other work in the field to assess if its aims have been fully achieved. In short, I will consider how convincing the text is as a comparative study. I will then conclude the essay by considering how alternative research designs may have improved or worsened the selected study, again drawing on important academic works to support my theories and assertions.

Section 1

Robert Putnam’s Making Democracy Work undoubtedly brought the concept of social capital to the forefront of the socio-political sphere. His celebrated work is a key example of a case study, “an intensive analysis of an individual unit (as a person, event or community) stressing developmental factors in relation to environment or context.”[1] Putnam’s research design has a broad analysis that compares 20 regions across Italy – a MSSD. He exploits a natural experiment to assess the difference institutional reform makes to institutional performance over his 20 year period of study. Upon recognising disparities in regional findings, he then assesses the reasons for cross-sectional and cross-temporal variation in institutional performance, moving to a more in-depth focus on 6 regions.

Although the case study discipline is beginning to decline in the modern academic sphere, the strengths of this kind of research design are still evident. Prominent examples from the field are Tocqueville’s Democracy in America (1888) and Lijphart’s The Politics of Accommodation (1968). Landman highlights some of the main strengths of case studies such as these, writing:

“Single-country studies provide contextual description, develop new classifications, generate hypotheses, confirm and inform theories, and explain the presence of deviant countries identified through cross-national comparison.”[2]

In short, the case study method allows an intensive depth study of a unit with limited resources. As Landman statement suggests, they are extremely flexible and can serve a multitude of purposes. Making Democracy Work is also a prime example of another key strength of case studies – utilising process-tracing to uncover evidence of causal mechanisms or to explain outcomes.[3] For example, Putnam is able to trace the roots of modern civic community and institutional performance back to Italy’s “Golden Age” in the 14^th century through his historical analysis. Other comparative methods, such as large-N, are far less conducive to this type of process tracing and unearthing of causal mechanisms. They are often able to say what happened, but not necessarily how it happened. The detail in which these mechanisms are explained is also far less extensive.

Despite the strengths highlighted above, there are also some considerable limitations to the case study method that have been recorded extensively in the literature of the social sciences. A key limitation, again highlighted by Landman, is that “inferences made from single-country studies are necessarily less secure than those made from the comparison of several or many countries.”^[4] Generalizations about other units cannot be made from a case study, unlike with large-N statistical analysis. Furthermore, case studies are usually better at description than establishing causation. The case study method is also considered intensive rather than extensive, as it tends to offer a great deal of in-depth analysis of a single unit at the expense of breadth analysis. With Putnam, for example, the study has a sole focus on Italy and Italian institutions and social capital over 600 years. George & Bennett draw upon the importance of this point, stating that another potential pitfall of the case study method is the “selection bias” that occurs when a unit is chosen on its “intrinsic historical importance”, or “on the accessibility of evidence”[5] This form of “selection bias” could certainly be evident in Putnam’s research design. Finally, where larger-N studies often confirmatory in their nature, seeking to either confirm or reject hypotheses, case studies tend to be more exploratory, attempting to gain new insights into a topic or unit from which new hypotheses might be later developed.

Juan Linz’s “Perils of Presidentialism” differs to Putnam’s work in that it is a small-N research design that compares the effect presidential and parliamentary regime types on democratic stability. Small-N research designs such as Linz’s usually consist of 2-12 intentionally selected cases and are therefore too limited to conduct statistical analysis as with large-N designs.[6] Linz’s research design is a qualitative comparison of a small number of cases selected from Latin America and Western Europe, with a particular focus on the USA also. Many have argued that his case selection is based upon MSSD. The main aim of “Perils of Presidentialism” was to explore political processes over time and within cases as a means of showing that “the superior historical performance of parliamentary democracies is no accident.”[7]

As with the case study method, there are numerous advantages to using small-N research designs – often called the comparative method – in comparative politics. One of the most significant strengths of using the comparative method comes from the intentional selection of cases as previously mentioned. Not only can it be a substitute for the experimental control evident in large-N analysis, the intentional selection of cases that share similar characteristics means that hypothesis testing is made easier.[8] A similarly significant advantage of using this method in comparison to the large-N statistical method is that concepts and ideas used in small-N studies can be operationalized at a lower level of abstraction, meaning that concepts are at a lower risk of being stretched. The result of this is greater confidence that chosen concepts are being accurately measured. Collier’s The Comparative Method also highlights the fact that small-N designs such as Linz’s allow for an intensive analysis of a few cases with limited energy expenditure, financial resources and time. These intensive analyses can be more fruitful than superficial statistical analysis of many cases that can be extremely time-consuming and difficult to execute successfully. The collection of large data sets has also proved extremely difficult.[9]

It could be argued that the weaknesses of the comparative method, or small-N research designs, out-weigh its strengths. Where the selection of cases can work as an experimental control, Landman, amongst others, has highlighted case selection as one of the major pitfalls of small-N designs, stating that “the selection of cases in the absence of any rules of inquiry can lead to insecure inferences, limited findings, and, in some cases, simply incorrect conclusions about a particular topic.”[10] Similarly, the issue of “many variables, small number of cases” is raised extensively in the literature (e.g. Lijphart, 1971; Goggin, 1986). Problems often arise when comparing few countries when there are more factors identified explaining the observed outcome than there are countries observed. Because small-N analysis involves hand-picked specific cases, there are often many variables linking the cases that are not central to the study, hence “too many variables, not enough cases.”[11] A way of potentially solving this problem, as highlighted by Lijphart, is adding more cases to the equation. This eventually becomes problematic when too many cases are added and the research passes from the small-N comparative method in to the large-N statistical analysis method. Using the small-N design that Linz has can therefore be extremely problematic.

Octavio Amorim Neto & Gary Cox’s “Electoral Institutions, Cleavage Structures, and the Number of Parties” is an example of the third type of research design – a quantitative large-N statistical analysis. Amorim Neto & Cox utilise a large-N cross-case statistical analysis which analyses the degree of correlation between the number of effective parties, various measures of electoral system permissiveness and ethnic fragmentation in electoral systems around the world. The design is also cross-sectional, as data is collected from 54 elections around the world (c.1985) and observes both parliamentary and presidential elections. Multivariate regression is used to assess the different impact each of the independent variables has on the dependant variable. Their conclusion is that “the effective number of parties appears to depend on the product of social heterogeneity and electoral permissiveness.”[12]

A particularly key advantage of using the large-N statistical method is the fact that statistical controls can be used to rule out rival explanations for an observed outcome and to control for potentially confounding factors.[13] In small-N analysis, it is clear that a lack of statistical control can lead to the production of incorrect of flawed results. Large-N analyses also allow for an extensive coverage of countries over both space and time.[14] In Democracy and Development (2000), for example, Prezworski et al. use 150 countries over a 40 year period in their analysis. Unlike in case studies and small-N designs, large-N designs are better able to avoid selection bias because units are usually randomly selected. Furthermore, large-N designs are better able to highlight ‘deviant’ or ‘outlier’ countries whose outcomes are not as expected from the study. The nature of such designs also means that generalizations can be made about the wider population from their findings because theories are tested with a far greater number of cases that are more representative of the wider population, whether it be individuals, groups or countries. As Coppedge states, “the principal advantage of the kind of theory that emerges from large-sample work is that it is relatively general, both in its aspirations and in its empirical grounding.”[15]

As made apparent by Collier, one of the main disadvantages of using the statistical method in a study is the difficulty in “collecting adequate information in a sufficient amount of time” when resources and time are limited.[16] Large-N analysis are, without doubt, extremely time consuming and often expensive; they also assume a certain amount of mathematical or computing knowledge in their execution and interpretation. Moreover, only certain types of data can be used using this method of analysis and the reliability of data extracted from developing countries, for example, is often questionable. This means that relevant data can sometimes be omitted or incorrect, meaning inferences can also be misleading. Even when data is available, its higher level of abstraction can sometimes lead to concept stretching, where we cannot be sure if the researcher is still measuring what he/she originally intended to. Another critique of the method is that large-N studies often rely upon assumptions that may not hold true, e.g. there are no omitted variables, cases are fully independent, or unit homogeneity. Finally, many who interpret large-N statistical analysis often mistake correlation for causation. Correlation simply highlights the direction and strength of a relationship, not causation.

Section 2

Regardless of its wider political impact, Juan Linz’s “The Perils of Presidentialism” has come under a vast amount of criticism in the social sciences for being too implicit and underdeveloped. Horowitz, for example, describes Linz’s claims that parliamentary governments is better able to stabilise democracy than presidential government as being “based on a regionally skewed and highly selective sample of comparative experience, principally from Latin America,” that “rest on a mechanistic, even caricatured, view of presidency.”[17] Criticising Linz choice of Latin America as antagonistic examples of unstable presidential systems, Horowitz highlights the inherited post-colonial Westminster parliamentary systems in Africa and Asia as equally destabilising and not at all conducive to democracy. This is because their winner-takes-all features allow any party with a majority to seize power.[18] Linz makes only a passing comment on these important examples that oppose his central causal arguments. Horowitz’s argument here could also be that a way to avoid the “regionally skewed” design that Linz’s work represents would be to adopt a MDSD – comparing various parliamentary and presidential systems from across the world as cases – instead of his MSSD design. This could also avoid the ‘selection bias’ that is often present through hand-picking cases in a small-N research design.

Similarly, both Horowitz and Shugart & Cary highlight Linz’s omission of the variation in electoral systems, party systems and presidential powers as a reason for “The Perils of Presidentialism” being a somewhat unconvincing comparative study. Shugart & Cary’s Presidents and Assemblies emphasises the variation in presidential systems, introducing semi-presidentialism, premier-presidential and president-parliamentary systems that are only briefly touched upon by Linz in his paper. France and Germany, for example, are mentioned but not elaborated on. When Shugart & Cary include these regime types in their analysis, they find that only twelve ‘full’ presidential systems had broken down in the twentieth century in comparison to twenty one parliamentary systems,[19] contradicting Linz’s argument that parliamentary systems are more conducive to stable democracy. In continuation with this, Horowitz draws on three examples from the USA, Nigeria and Sri Lanka to highlight the variations in electoral systems, processes and party systems that produce presidential executives.[20] Once again these variations are only touched upon in passing by Linz, whose analysis is underdeveloped and openly omitting important cases that contradict what Horowitz called his “caricatured view on presidency.” This caricatured view is also highlighted by Mainwaring & Shugart, who argue that presidentialism is predicated upon a system of checks and balances which usually inhibit winner-takes-all tendencies that Linz describes as a central feature of presidential systems.[21]

Further criticisms of Linz’s design come from Cheibub’s Presidentialism, Parliamentarism, and Democracy. Cheibub opposes Linz’s argument that the historical evidence of democratic breakdown shows presidential regimes to be less democratically stable, instead stating that presidential democracies fail because they arise in countries with a higher probability of democratic breakdown, regardless of regime type.[22] Cheibub highlights the importance of economic development in the survival of democratic regime types. Countries with parliamentary systems happen to be more wealthy, and therefore more likely to survive.[23] Economic development is not the only variable omission present in Linz’s work either, as Cheibub highlights the importance of geographical location and size of the country as well as economic development as important to democratic longevity. Cheibub also questions Linz’s decision to omit coalitions as a variable, an electoral outcome Linz holds in high regard in terms of democratic stability and performance. Again, Linz does not draw on any of these points in his qualitative analysis. A final point highlighted by Cheibub is that impeachment, something Linz asserts is extremely difficult in presidential systems, happened six separate times in 1990s Latin America alone, four of which were passed. This again shows some of Linz’s assertions to be incorrect or at least without empirical support.

It could be argued that Prezworski et al’s 1996 study What Makes Democracy Work? represents a superior research design to Linz’s. This is because Prezworski’s large-N statistical analysis of 135 countries since 1950 controls for affluence, economic performance, international climate and political learning/experience with democracy.[24] Linz’s small-N research design – a design that is never properly justified – has no explicit control variables, even though both studies have a similar aim of determining what/why democratic regimes prevail. The exploratory nature of Linz’s study, who openly admits is only to “recover a debate on the role of alternative democratic institutions in building state democracies,”[25] means that there is no explicit hypothesis testing per se. Prezworski, on the other hand, extensively tests his hypothesis on a large-N and can therefore make generalizations from his inferences. It could be argued that these factors make it much more useful to the social scientist studying the reasons for the longevity of democratic regimes. This is not to say, however, that Prezworski’s findings do not support Linz’s findings. Prezworski does find that “Linz is right about the durability of alternative institutional arrangements,”[26] although factors such as economic development are more important in determining the permanence of specific regime types. This shows that the statistical method has some features that could have improved Linz’s study.

Conclusions

Section 1 of this essay has shown the various strengths and weaknesses of each comparative research design. Case studies allow for an extremely in-depth analysis of a single unit with limited resources. As Landman shows, they can also have a multitude of purposes e.g. developing new classifications, generating hypotheses, confirming and informing theories etc. They are, however, limited. Inferences made from case studies are less secure and generalizations cannot be made from their findings. They tend to be extremely descriptive, being considered intensive rather than extensive because breadth analysis gives way to in-depth analysis. George & Bennett’s theory of potential “selection bias” can also be problematic.

Similarly, small-N research designs have a multitude of strengths and weaknesses. The intentional selection of cases can work as a substitute for experimental control found in large-N statistical analysis. Operationalizing concepts at a lower level of abstraction means that concept stretching is far less likely than in large-N research designs. The comparative method also allows for an intense analysis of a few countries when resources and time are low. On the other hand, though, small-N research designs often suffer from the “many variables, small number of cases,” where there are more observed explanatory factors than there are cases. This is just one of the problems with hand-picking cases, another being the potential of insecure inferences and limited findings.

Finally, the evaluation of large-N research designs shows it to have many strengths and weaknesses also. The fact they allow for statistical controls is one of their greatest strengths. They also allow for an extensive coverage of many cases over both space and time, as with Amorim Neto & Cox. The random selection of cases removes the issue of selection bias in the traditional sense. The ability to highlight ‘deviant’ or ‘outlier’ countries is also a key strength of the large-N design. They are, however, time-consuming, involve a lot of resources and assume a level of mathematical and computing knowledge from the outset. The collection of relevant and correct data, especially from developing countries can also be problematic.

Section 2 of the essay has shown that, upon evaluation, Juan Linz’s “The Perils of Presidentialism” has many flaws in both its research design and its execution. Although Prezworski et al. have shown Linz’s findings to be correct, Horowitz demonstrates that Linz’s case choices are not only unjustified, they are “regionally-skewed.” Evidence from post-colonial Africa and Asia juxtapose Linz’s theory, so a switch from MSSD to MDSD may have therefore improved the design. Moreover, Cheibub’s work shows a blatant omission of important variables in Linz’s research design, as well as an ignorance of the variation in presidential regimes. This, combined with Shugart & Cary’s work that shows a lack of acknowledgement for variation in electoral systems and the party system, demonstrates an undeveloped design with unconvincing causal inferences. Cheibub also counters Linz’s central causal argument by suggesting that regime type is not necessarily the driving force behind democratic longevity, but the individual situations in which the system is born. The importance geographical size, location and economic development as key factors in the survival of a regime type are not highlighted. This once again shows important omissions in the execution of his work. The large-N statistical method has features that could improve Linz’s design, as it allows for what has been described by Kerlinger as “the three criteria of the ideal research design: (1) that the design answer the research question; (2) that it introduce the element of control for extraneous independent variables; and (3) that it permit the investigator to generalize from his or her findings.”^[27] An intrinsic weaknesses of case studies and small-N designs, including Linz’s, is that the second and third of these points are often unobtainable.

Bibliography

Amorim Neto, Octavio & Gary Cox, “Electoral Institutions, Cleavage Structures, and the Number of Parties”, American Journal of Political Science 41(1), Indiana: Midwest Political Science Association, Jan., 1997, pp. 149-174.

Boix, Carles & Daniel N. Posner, “Making Social Capital Work: A Review of Robert Putnam’s Making Democracy Work: Civic Traditions in Modern Italy”, The Weatherhead Centre For International Affairs 96(4), Harvard University, June, 1996, pp. 1-22.
Accessed on: < http://www.wcfia.harvard.edu/sites/default/files/96-04.pdf > (06/12/2012)

Cheibub, José Antonio, Presidentialism, Parliamentarism, and Democracy, New York: Cambridge University Press, 2007.

Clark, William Roberts, Matt Golder & Sona Nadenichek Golder, Principles of Comparative Politics (2^nd ed.), London: Sage, 2013.

Collier, David, “The Comparative Method”, in Ada W. Finifter (ed.), Political Science: The State of the Discipline II, Washington, D.C.: The American Political Science Association, 1993, pp. 105-119.

Coppedge, Michael, “Theory Building and Hypothesis Testing: Large- vs. Small-N Research on Democratization”, Paper prepared for presentation at the Annual Meeting of the Midwest Political Science Association, Chicago, Illinois, April 25-27, 2002.

Flyvbjerg, Bent, “Case Study”, in Norman K. Denzin & Yvonna S. Linclon (eds.), The Sage Handbook of Qualitative Research (4^th ed.) California: Sage, 2011, pp. 301-316.

George, Alexander L. & Andrew Bennett, Case Studies and Theory Development in the Social Sciences, Massachusetts: MIT Press, 2005.

Gerring, John, “What Is a Case Study and What Is It Good For?”, American Political Science Review 98(2), Cambridge: University of Cambridge Press, May, 2004, pp. 341-354.

Goggin, Malcolm L., “The “Too Few Cases/Too Many Variables” Problem in Implementation Research”, The Western Political Quarterly 39(2), Utah: University of Utah, Jun., 1986, pp. 328-347.

Horowitz, Donald L., “Comparing Democratic Systems”, Journal of Democracy 1(4), Maryland: The Johns Hopkins University Press, Fall 1990, pp. 73-79.

Kerlinger, Fred N., Foundations of Behavioral Research (2^nd ed.), New York: Holt, Rinehart and Winston, 1973.

Landman, Todd, Issues and Methods in Comparative Politics: An Introduction (3^rd ed.), London: Routledge, 2008.

Lijphart, Arend, “Comparative Politics and the Comparative Method,” American Political Science Review 65(3), Cambridge: University of Cambridge Press, Sep., 1971, pp. 682-693.

Linz, Juan J., “The Perils of Presidentialism,” Journal of Democracy 1(1), Maryland: The Johns Hopkins University Press, Win., 1990, pp. 51-56.

Mainwaring, Scott & Matthew S. Shugart, “Juan Linz, Presidentialism, and Democracy: A Critical Appraisal”, Comparative Politics 29(4) Ph.D. Program in Political Science of the City University of New York, Jul., 1997, pp. 449-471.

Morgan-Jones, Edward, “Modern Classics of Comparative Politics,” Lectures at the University of Kent, Canterbury, 27^th Sep.-14^th. Dec, 2012.

Prezworski, A. et al., Democracy and Development: Political Institutions and Well-Being in the World, 1950-1990, Cambridge: Cambridge University Press, 2000.

Prezworski, A. et al., “What Makes Democracy Endure?”, Journal of Democracy 7(1), Maryland: The Johns Hopkins University Press, Win., 1996, pp. 39-55.

Putnam, Robert D., Making Democracy Work: Civic Traditions in Modern Italy, NJ: Princeton University Press, 1993.

Shugart, Matthew Soberg & John M. Carey, Presidents and Assemblies: Constitutional Design and Electoral Dynamics, Cambridge: Cambridge University Press, 1992.

Further Reading on E-International Relations

• Emancipation and Epistemological Hierarchy: Why Research Methods Are Always Political
• Offensively Realist? Evaluating Trump’s Economic Policy Towards China
• Mass Atrocities and Western Imperialism: Evaluating “Responsibility to Protect”
• Between Pepe and Beyoncé: The Role of Popular Culture in Political Research
• Ya Basta! A Case for Social Movements in Critical Norm Research
• Gramscian Notions: Helpful for Research into Digital and Tech Corporations?