Martin Schweinsberg, Michael Feldman, Nicola Staub, Olmo R. van den Akker, Robbie C.M. van Aert, Marcel A.L.M. van Assen, Yang Liu, Tim Althoff, Jeffrey Heer, Alex Kale, Zainab Mohamed, Hashem Amireh, Vaishali Venkatesh Prasad, Abraham Bernstein, Emily Robinson, Kaisa Snellman, S. Amy Sommer, Sarah M.G. Otner, David Robinson, Nikhil Madan, Raphael Silberzahn, Pavel Goldstein, Warren Tierney, Toshio Murase, Benjamin Mandl, Domenico Viganola, Carolin Strobl, Catherine B.C. Schaumans, Stijn Kelchtermans, Chan Naseeb, S. Mason Garrison, Tal Yarkoni, C.S. Richard Chan, Prestone Adie, Paulius Alaburda, Casper Albers, Sara Alspaugh, Jeff Alstott, Andrew A. Nelson, Eduardo Ariño de la Rubia, Adbi Arzi, Štěpán Bahník, Jason Baik, Laura Winther Balling, Sachin Banker, David AA Baranger, Dale J. Barr, Brenda Barros-Rivera, Matt Bauer, Enuh Blaise, Lisa Boelen, Katerina Bohle Carbonell, Robert A. Briers, Oliver Burkhard, Miguel-Angel Canela, Laura Castrillo, Timothy Catlett, Olivia Chen, Michael Clark, Brent Cohn, Alex Coppock, Natàlia Cugueró-Escofet, Paul G. Curran, Wilson Cyrus-Lai, David Dai, Giulio Valentino Dalla Riva, Henrik Danielsson, Rosaria de F.S.M. Russo, Niko de Silva, Curdin Derungs, Frank Dondelinger, Carolina Duarte de Souza, B. Tyson Dube, Marina Dubova, Ben Mark Dunn, Peter Adriaan Edelsbrunner, Sara Finley, Nick Fox, Timo Gnambs, Yuanyuan Gong, Erin Grand, Brandon Greenawalt, Dan Han, Paul H.P. Hanel, Antony B. Hong, David Hood, Justin Hsueh, Lilian Huang, Kent N. Hui, Keith A. Hultman, Azka Javaid, Lily Ji Jiang, Jonathan Jong, Jash Kamdar, David Kane, Gregor Kappler, Erikson Kaszubowski, Christopher M. Kavanagh, Madian Khabsa, Bennett Kleinberg, Jens Kouros, Heather Krause, Angelos-Miltiadis Krypotos, Dejan Lavbič, Rui Ling Lee, Timothy Leffel, Wei Yang Lim, Silvia Liverani, Bianca Loh, Dorte Lønsmann, Jia Wei Low, Alton Lu, Kyle MacDonald, Christopher R. Madan, Lasse Hjorth Madsen, Christina Maimone, Alexandra Mangold, Adrienne Marshall, Helena Ester Matskewich, Kimia Mavon, Katherine L. McLain, Amelia A. McNamara, Mhairi McNeill, Ulf Mertens, David Miller, Ben Moore, Andrew Moore, Eric Nantz, Ziauddin Nasrullah, Valentina Nejkovic, Colleen S Nell, Andrew Arthur Nelson, Gustav Nilsonne, Rory Nolan, Christopher E. O'Brien, Patrick O'Neill, Kieran O'Shea, Toto Olita, Jahna Otterbacher, Diana Palsetia, Bianca Pereira, Ivan Pozdniakov, John Protzko, Jean-Nicolas Reyt, Travis Riddle, Amal (Akmal) Ridhwan Omar Ali, Ivan Ropovik, Joshua M. Rosenberg, Stephane Rothen, Michael Schulte-Mecklenbeck, Nirek Sharma, Gordon Shotwell, Martin Skarzynski, William Stedden, Victoria Stodden, Martin A. Stoffel, Scott Stoltzman, Subashini Subbaiah, Rachael Tatman, Paul H. Thibodeau, Sabina Tomkins, Ana Valdivia, Gerrieke B. Druijff-van de Woestijne, Laura Viana, Florence Villesèche, W. Duncan Wadsworth, Florian Wanders, Krista Watts, Jason D Wells, Christopher E. Whelpley, Andy Won, Lawrence Wu, Arthur Yip, Casey Youngflesh, Ju-Chi Yu, Arash Zandian, Leilei Zhang, Chava Zibman, Eric Luis Uhlmann
In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists’ gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic choices are made, the analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded, resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions play a critical role in driving the reported empirical results, underscoring the need for open data, systematic robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for organizations and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal analyses by data scientists, are discussed. © 2021 The Author(s)
ESMT, Berlin, Germany; University of Zurich, Switzerland; Tilburg University, Netherlands; Tilburg University and Utrecht University, Netherlands; University of Washington, United States; ESMT Berlin and Indiana University, Germany; ESMT Berlin and Humboldt University Berlin, Germany; INSEAD, France; Marshall School of Business, University of Southern California, United States; Imperial College Business School, United Kingdom; Indian School of Business, India; University of Sussex Business School, United Kingdom; School of Public Health, University of Haifa, Israel; INSEAD, Singapore; Waseda University, Japan; Stockholm School of Economics, Sweden; Independent researcher; KU Leuven, Belgium; IBM, Germany; Wake Forest University, United States; University of Texas at Austin, United States; Stony Brook University, United States; University of Nairobi, Kenya; University of Groningen, Netherlands; University of California, Berkeley, United States; Massachusetts Institute of Technology, United States; University of Kentucky, United States; California State University-Dominguez Hills, United States; The Prague College of Psychosocial Studies, Czech Republic; Copenhagen Business School, Denmark; University of Utah, United States; University of Pittsburgh, United States; University of Glasgow, United Kingdom; Texas A&M University, United States; Illinois Institute of Technology, United States; Eskisehir Osmangazi University, Turkey; Imperial College London, United Kingdom; Northwestern University, United States; Edinburgh Napier University, United Kingdom; University of Navarra, Spain; University of Michigan, United States; Yale University, United States; Universitat Oberta de Catalunya, Spain; Michigan State University, United States; St. Michael's Hospital, University of Toronto, Canada; Department of Mathematics and Statistics, University of Canterbury, New Zealand; Linköping University, Sweden; Universidade Nove de Julho, Brazil; Lucerne University of Applied, Sciences and Arts, Switzerland; Lancaster University, United Kingdom; Universidade Federal de Santa Catarina, Brazil; Indiana University, United States; ETH Zurich, Switzerland; Pacific Lutheran University, United States; Rutgers University, United States; Leibniz Institute for Educational Trajectories, Germany, & Johannes Kepler University Linz, Austria; Okayama University, Japan; University of Notre Dame, United States; University of Bath, University of Essex, United Kingdom; University of Chicago, United States; School of Management, Xiamen University, China; Elmhurst College, United States; Columbia University Medical Center, United States; University of Washington, & Indiana University, United States; University of Oxford, & Coventry University, United Kingdom; Harvard University, United States; University of Vienna, Austria; University College London, United Kingdom; York University, United Kingdom; Department of Clinical Psychology, Utrecht University, the Netherlands, & Group of Health Psychology, KU Leuven, Belgium; Nanyang Technological University, Singapore; University of Colorado, Colorado Springs, United States; Queen Mary University of London, United Kingdom; University of Copenhagen, Denmark; Singapore Management University, Singapore; McD Tech Labs, United States; School of Psychology, University of Nottingham, United Kingdom; Novo Nordisk, Denmark; University of Idaho, United States; University of St Thomas, United States; Heidelberg University, Germany; University of Edinburgh, United Kingdom; Eli Lilly, United States; University of Nis, Faculty of Electronic Engineering, Serbia; George Washington University, United States; Karolinska Institutet and Stockholm University, Sweden; University of Oxford, United Kingdom; University of Maryland, Baltimore County, United States; The University of Western, Australia; Open University of Cyprus, Cyprus; National Research University, Higher School of Economics, Russian Federation; University of California, Santa Barbara, United States; McGill University, Canada; National Institutes of Health/National Institute of Mental Health, United States; The University of Sheffield, United Kingdom; Charles University, Faculty of Education, Institute for Research and Development of Education, Czech Republic & University of Presov, Faculty of Education, Slovakia; University of Tennessee, Knoxville, United States; University of Bern, Switzerland & Max Planck Institute for Human Development, Germany; Washington University in St. Louis, United States; Dalhousie University, Canada; University of Illinois at Urbana-Champaign, United States; Institute of Evolutionary Biology, University of Edinburgh, United Kingdom; Colorado State University, United States; CSU, United States; Rasa Technologies, United States; Oberlin College, United States; Stanford University, United States; University of Granada, Spain; Radboud University Nijmegen, Netherlands; University of Hawaii, United States; Microsoft & Rice University, United States; University of Amsterdam, Netherlands; Dartmouth College, United States; College of Charleston, United States; Department of Ecology and Evolutionary Biology, University of California, Los Angeles, United States; The University of Texas at Dallas, School of Behavioral and Brain Sciences, United States; Division of Affinity Proteomics, Department of Protein Science, KTH Royal Institute of Technology & SciLifeLab, Sweden; University of Ljubljana, Slovenia
