Does it pay to be rigorous? The link between validity, reliability and impact in management case-study research

In management research, article impact is becoming an increasingly popular metric for assessing a scholar’s influence, and articles based on case studies are among the most impactful scholarly contributions in strategy research and general management. At the same time, the methodological rigor of case study research as a tool for generating or testing management theory remains an ongoing, unresolved issue (e.g. Campbell, 1975; Miles, 1979; March et al., 1991; Yin, 1981; Weick, 2007; Siggelkow, 2007; Piekkari, Welch, and Paavilainen, 2009; Gibbert, Ruigrok, and Wicki, 2008; Gibbert and Ruigrok, forthcoming). The research proposed here focuses on the case study method in management research and tests whether (lack of) methodological rigor in terms of validity and reliability influences the impact of case study articles, defined as summed counts of references in the scientific literature to a ‘cited’ research article. The usage of the case study as a research tool in business management is characterized by a particular disciplinary problem. On the one hand, the methodological rigor of case studies as a tool for generating and testing theory in management has been criticized like no other method, particularly in terms of validity and reliability. To date, these preoccupations remain unresolved, the main reason being that procedures for assessing the rigor of qualitative research such as the case study are much less standardized than those of other, quantitative, methods which in turn makes it very hard indeed to get case-study work published in top-tier outlets. On the other hand, once their rigor has been ascertained by the relevant gatekeepers (that is, once they are published), case studies as a tool for generating and testing theory have provided the general management field with its most impactful insights (e.g., Penrose, 1960; Chandler, 1962; Pettigrew, 1973). Influential scholars have recently argued that papers building theory from cases are not only among the “most interesting” (Eisenhardt and Graebner, 2007; Barunek, Rynes, and Ireland, 2006), but are also among the most impactful papers in the academic community (in terms of citation counts, e.g. Eisenhardt, 1989b; Gersick, 1988). Their impact may be due to the fact that case studies are considered most appropriate as tools in the critical, early phases of a new management theory, when key variables and their relationships are being explored (Yin, 1994; Eisenhardt, 1989). A rigor problem in the early stages of theory development would therefore have ripple-effects throughout later stages when relationships between variables are elaborated and tested (Eisenhardt and Graebner, 2007). Furthermore, case studies represent a methodology that is ideally suited to creating managerially-relevant knowledge, as they are typically carried out in close interaction with practitioners, and deal with real management situations (Amabile et al, 2001; Leonard-Barton, 1990). However, as Scandura and Williams remind us, “without rigor, relevance in management research cannot be claimed” (Scandura and Williams, 2000: 1263). Thus, examining the link between case-study rigor and impact is not only timely, but also fundamental for ascertaining the very foundations on which management as an academic discipline and professional vocation is built. The present study is the first to examine this important link.