Examining the generalizability of research findings from archival data

Open

Andrew Delios, Elena Giulia Clemente, Tao Wu, Hongbin Tan, Yong Wang, Michael Gordon, Domenico Viganola, Zhaowei Chen, Anna Dreber, Magnus Johannesson, Thomas Pfeiffer, Eric Luis Uhlmann, Ahmad M. Abd Al-Aziz, Ajay T. Abraham, Jais Trojan, Matus Adamkovic, Elena Agadullina, Jungsoo Ahn, Cinla Akinci, Handan Akkas, David Albrecht, Shilaan Alzahawi, Marcio Amaral-Baptista, Rahul Anand, Kevin Francis U. Ang, Frederik Anseel, John Jamir Benzon R. Aruta, Mujeeba Ashraf, Bradley J. Baker, Xueqi Bao, Ernest Baskin, Hanoku Bathula, Christopher W. Bauman, Jozef Bavolar, Secil Bayraktar, Stephanie E. Beckman, Aaron S. Benjamin, Stephanie E. V. Brown, Jeffrey Buckley, Ricardo E. Buitrago, Jefferson L. Bution, Nick Byrd, Clara Carrera, Eugene M. Caruso, Minxia Chen, Lin Chen, Eyyub Ensari Cicerali, Eric D. Cohen, Marcus Crede, Jamie Cummins, Linus Dahlander, David P. Daniels, Lea Liat Daskalo, Ian G. J. Dawson, Martin V. Day, Erik Dietl, Artur Domurat, Jacinta Dsilva, Christilene Du Plessis, Dmitrii I. Dubrov, Sarah Edris, Christian T. Elbaek, Mahmoud M. Elsherif, Thomas R. Evans, Martin R. Fellenz, Susann Fiedler, Mustafa Firat, Raquel Freitag, Rémy A. Furrer, Richa Gautam, Dhruba Kumar Gautam, Brian Gearin, Stephan Gerschewski, Omid Ghasemi, Zohreh Ghasemi, Anindya Ghosh, Cinzia Giani, Matthew H. Goldberg, Manisha Goswami, Lorenz Graf-Vlachy, H. Rajeshwari, Jennifer A. Griffith, Dmitry Grigoryev, Jingyang Gu, Allegre L. Hadida, Andrew C. Hafenbrack, Sebastian Hafenbrädl, Jonathan J. Hammersley, Hyemin Han, Jason L. Harman, Andree Hartanto, Alexander P. Henkel, Yen-Chen Ho, Benjamin C. Holding, Felix Holzmeister, Alexandra Horobet, Tina S.-T. Huang, Yiming Huang, Jeffrey R. Huntsinger, Katarzyna Idzikowska, Hirotaka Imada, Rabia Imran, Michael J. Ingels, Bastian Jaeger, Steve M. J. Janssen, Fanli Jia, Alfredo Jiménez, Jason Lu Jin, Niklas Johannes, Daniel Jolles, Bibiana Jozefiakova, Pavol Kačmár, Tamara Kalandadze, Kyriaki Kalimeri, Polly Kang, Jaroslaw Kantorowicz, Didar Karada, Hamid Karimi-Rouzbahani, Daisy Mui Hung Kee, Lucas Keller, Haider A. Khan, Mikael Knutsson, Olga Kombeiz, Aleksey Korniychuk, Marta Kowal, Johannes Leder, Liang Wenhao Liang, Taegyeong Liew, Fangwen Lin, Chengwei Liu, Bin Liu, Maria Cristina Longo, Andrey Lovakov, Mei Peng Low, Gerardus J. M. Lucas, Oliver Lukason, Albert L. Ly, Zhuoran Ma, Alexander Mafael, Soheil Mahmoudkalayeh, David Manheim, Alfred Marcus, Melvin S. Marsh, Jolie M. Martin, Luis E. Martinez, Mario Martinoli, Marcel Martončik, Theodore C. Masters-Waage, Rui Mata, Hamid Mazloomi, Randy J. Mccarthy, Philip Millroth, Mahima Mishra, Supriti Mishra, Alexander Mohr, David Moreau, Annalisa Myer, Amos Nadler, Sudhir Nair, Gustav Nilsonne, Pawe Niszczota, Aoife O'Mahony, Marc Oberhauser, Tomasz Obloj, Mehmet A. Orhan, Flora Oswald, Tobias Otterbring, Philipp E. Otto, Ivar Padrón-Hernández, Alan J. Pan, Mariola Paruzel-Czachura, Gerit Pfuhl, Angelo Pirrone, Simon Porcher, John Protzko, Shelly Qi, Rima-Maria Rahal, Md. Shahinoor Rahman, Michelle L. Reina, Satyanarayana Rentala, Zahid Riaz, Ivan Ropovik, Lukas Röseler, Robert M. Ross, Amanda Rotella, Leopold H. O. Roth, Thomas J. Roulet, Matthew M. Rubin, Andre Sammartino, Johann Sanchez, Adrian D. Saville, Michael Schaerer, Joyce Elena Schleu, Leo Schmallenbach, Landon Schnabel, Frederik Schulze Spüntrup, Birga M. Schumpe, Tony Senanayake, Raffaello Seri, Feng Sheng, Roary E. Snider, Di Song, Victoria Song, Sylwia E. Starnawska, Kai A. Stern, Samantha M. Stevens, Eirik Strømland, Wunhong Su, Hao Sun, Kevin P. Sweeney, Reina Takamatsu, Maria Terskova, Kian Siong Tey, Warren Tierney, Mariya M. Todorova, Daniel Tolstoy, Lasse Torkkeli, Joshua M. Tybur, Francisco J. Valderrey, Ana Maria Vallina-Hernandez, Ranjith P. Vasudevan, Gudivada Venkat Rao, Antoine Vernet, Tiia Vissak, Hinrich Voss, Thorsten Wahle, Jonathan Wai, Lauren E.T. Wakabayashi, Junnan Wang, Peng Wang, Yating Wang, Robert W. Warmenhoven, Karl Wennberg, Georg Wernicke, Jan K. Woike, Conny E. Wollbrant, Greg Woodin, Joshua D. Wright, Qiong Xia, Zhenzhen Xie, Sangsuk Yoon, Wenlong Yuan, Lin Yuan, Meltem Yucel, Zhao Zheng, Haibo Zhou, Cristina Zogmaister, Ro'i Zultan

2022 Proceedings of the National Academy of Sciences of the United States of America Vol. 119 Issue 30 Article Cited by 46 Quartile

Abstract

This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability-for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples. © 2022 National Academy of Sciences. All rights reserved.

Affiliations

Department of Strategy and Policy, National University of Singapore, 15 Kent Ridge Drive, Singapore, 119245, Singapore; Stockholm School of Economics, Sveavägen 65, Stockholm, 13 83, Sweden; Advanced Institute of Business, Tongji University, Tongji Building A, 1500 Siping Rd., Shanghai, 200092, China; North Shore Mail Centre, New Zealand Institute for Advanced Study, Massey University, Private Bag 102904, Auckland, 0745, New Zealand; Global Indicators Department DEC, World Bank Group, 1818 H Street NW, Washington, 20433, DC, United States; Department of Economics, University of Innsbruck, Universitaetsstrasse 15, Innsbruck, 6020, Austria; Department of Organizational Behaviour, INSEAD Singapore, INSEAD Asia Campus, 1 Ayer Rajah Avenue, Singapore, 138676, Singapore; School of Management, Xi'an Jiaotong University, 28 West Xianning Road, Shaanxi, Xi'an, 710049, China; School of Management and Economics and Shenzhen Finance Institute, Chinese University of HongKong, Shenzhen (CUHK-Shenzhen), Shenzhen, 518000, China; Swedish House of Finance, Drottninggatan 98, Stockholm, 111 60, Sweden; Faculty of Arts and Humanities, British University in Egypt (BUE), Egypt; Seattle University, United States; UK (School of Pysch), Keele University, Newcastle, United Kingdom; Faculty of Arts, Institute of Social Sciences CSPS, University of Presov, Slovakia; National Research University Higher School of Economics; Western University, Ivey business school, Canada; University of St Andrews, United Kingdom; Ankara Science University, Turkey; Maastricht University, Netherlands; Graduate School of Business, Stanford University, United States; Center for International Studies - ISCTE - University Institute of Lisbon, Portugal; Aarhus BSS, Denmark; Value Care Health Systems; UNSW Sydney Business School, Australia; De la Salle University, Manila, Philippines; University of Punjab, Lahore, Pakistan; Temple University, United States; INSEAD, France; Saint Joseph's University, United States; University of Auckland, New Zealand; University of California, Irvine, United States; University of Manitoba, Canada; TBS Business School; Madison College, United States; University of Illinois at Urbana-Champaign, United States; Texas a and M University, United States; Athlone Institute of Technology and Department of Learning, Faculty of Engineering and Informatics, KTH Royal Institute of Technology, Sweden; Universidad del Rosario, Colombia; School of Economics, Business and Accountancy at University of Sao Paulo, Brazil; Stevens Institute of Technology, United States; UCLA Anderson School of Management, United States; Nisantasi University, Istanbul, Turkey; State University of Campinas, Brazil; Iowa State University, United States; Ghent University, Belgium; ESMT, Berlin, Germany; NUS Business School, National University of Singapore, Singapore; Ben-Gurion University of Negev, Israel; University of Southampton, United Kingdom; Memorial University of Newfoundland, Canada; Loughborough University, United Kingdom; Kozminski University, Poland; University of Balamand Dubai, United Arab Emirates; Singapore Management University, Singapore; HSE University Basic Research Program, Moscow, Russian Federation; School of Business and Economics, Maastricht University, Netherlands; Department of Management, Aarhus University, Denmark; Leicester University and University of Birmingham, United Kingdom; School of Human Sciences, University of Greenwich, United Kingdom; Trinity Business School, Trinity College Dublin, Ireland; Vienna University of Economics and Business, Austria; University of Alberta, Canada; Federal University of Sergipe, Brazil; University of Virginia, United States; University of Delaware, United States; Faculty of Management, Tribhuvan University, Kathmandu, Nepal; University of Oregon, United States; University of Kent, United Kingdom; School of Psychological Sciences, Macquarie University, Australia; Tilburg University, Netherlands; DiECO, Università degli Studi dell'Insubri, Italy; Yale University, United States; Institute of Business Management, GLA University, Mathura, India; TU Dortmund University, Germany; Peter T. Paul College of Business and Economics, University of New Hampshire, United States; HEC, Montreal, Canada; University of Hong Kong, Hong Kong; Karnataka State Open University, India; University of Cambridge, United Kingdom; Foster School of Business, University of Washington, United States; IESE Business School; Dept. of Psychology, Western Illinois University, United States; University of Alabama, United States; Louisiana State University, United States; Tsinghua University, China; Open University of Netherlands, Netherlands; National Chung Hsing University, China; Department of Sociology, Karolinska Institutet, Sweden; Department of Economics, University of Innsbruck, Austria; Bucharest University of Economics Studies, Romania; University College London, United Kingdom; Nanjing University, China; Loyola University Chicago, United States; Kozminski University; University of Kent, United Kingdom; Dhofar University, Oman; Vrije Universiteit Amsterdam, Netherlands; University of Nottingham Malaysia, Malaysia; Seton Hall University, United States; Department of Management, KEDGE Business School, France; Advanced Institute of Business, Tongji University, China; Oxford Internet Institute, University of Oxford, United Kingdom; University of Essex, United Kingdom; Olomouc University Social Health Institute, Palacky University Olomouc, Czech Republic; Department of Psychology, Faculty of Arts, University in Košice, Pavol Jozef ŠAfárik, Slovakia; Ostfold University College, Norway; ISI Foundation; National University of Singapore, Singapore; Leiden University, Netherlands; Lancaster University, United Kingdom; St. Joseph's College, NY, United States; Universiti Sains Malaysia, Malaysia; Department of Psychology, University of Konstanz, Germany; University of Denver, United States; Linköping University, Sweden; BNU-HKBU United International College; Copenhagen Business School, Denmark; University of Wroclaw, Poland; University of Bamberg, Germany; Xiamen University, China; Duke University, United States; Università di Milano-Bicocca, Italy; University of Nottingham Ningbo China, China; Department of Economics and Business, University of Catania, Italy; University of Macau, Macao; Universiti Tunku Abdul Rahman, Malaysia; Nottingham University Business School, University of Nottingham, United Kingdom; Loma Linda University, United States; Center for Retailing, Stockholm School of Economics Elizabeth A. Mahar, Sweden; University of Haifa, Israel; University of Minnesota Carlson School, United States; Georgia Southern University, United States; Alpha Edison, United States; Trinity University, United States; Sant'Anna School of Advanced Studies, Italy; Faculty of Arts, Institute of Psychology, University of Prešov, Prešov, Slovakia; University of Dayton, United States; University of Basel, Switzerland; Rennen School of Business, France; Northern Illinois University, United States; Department of Psychology, Uppsala University, Sweden; Symbiosis Institute of Business Management, Symbiosis International University, Pune, India; International Management Institute Bhubaneswar, WU Vienna, Austria; School of Psychology, University of Auckland, New Zealand; Department of Psychology, Graduate Center, City University of New York (CUNY), NY, United States; Fabriik; Peter B. Gustavson School Business, University of Victoria, Canada; Department of Clinical Neuroscience, Karolinska Institutet, Sweden; Pozna University of Economics and Business, Poland; School of Psychology, Cardiff University, United Kingdom; Friedrich-Alexander University Erlangen-Nürnberg, Germany; Metis Lab, EM Normandie Business School, France; Pennsylvania State University, United States; University of Agder, Norway; European University Viadrina, Germany; Hitotsubashi University, Japan; Beijing Normal University, China; University of Silesia in Katowice, Poland; UiT Arctic University of Norway, Norway; Centre for Philosophy of Natural and Social Science, London School of Economics and Political Science, United Kingdom; IAE Paris Université Paris I Panthéon-Sorbonne, France; University of California, Santa Barbara, United States; Max Planck Institute for Research on Collective Goods, Germany; Department of Psychology, University of Chittagong, Bangladesh; University of Mary Hardin-Baylor, United States; Bharathidasan Institute of Management, India; Lahore School of Economics, Pakistan; Institute for Research and Development of Education and University of Presov, Faculty of Education, Charles University, Czech Republic; HEC, Paris, France; Macquarie University, Australia; Department of Psychology, Kingston University London, United Kingdom; University of Vienna, Austria; English Language and Linguistics, University of Birmingham, United Kingdom; University of Melbourne, Australia; Gordon Institute of Business Science, University of Pretoria, South Africa; Lee Kong Chian School of Business, Singapore Management University, Singapore; Radboud University, Netherlands; University of Mannheim, Germany; Cornell University, United States; Institute for Globally Distributed Open Research and Education; University of Amsterdam, Netherlands; University of Southern Denmark and DiECO, COMAC, Denmark; School of Management, Zhejiang University, China; University of Arkansas, United States; University of Stirling, United Kingdom; Fordham University, United States; SUNY Empire State College, United States; University of North Carolina, Chapel Hill, United States; University of Plymouth, United Kingdom; Western Norway University of Applied Sciences, Norway; Hangzhou Dianzi University, China; School of Management, Xiamen University, China; Western Kentucky University, United States; Graduate School of Education, Kyoto University, Japan; WU Vienna, Austria; Stockholm School of Economics, Sweden; LAB University of Applied Sciences, Finland; HAN University, Netherlands; Tecnologico de Monterrey, Mexico; Pontificia Universidad Catolica de Valparaiso, Chile; Cms business school, JAIN (deemed to be university), India; Department of HRM, Andhra University, SRF-ICSSR, India; University of Tartu, Estonia; University of Florida, United States; Università degli Studi dell'Insubria, Italy