R. Silberzahn, E.L. Uhlmann, D.P. Martin, P. Anselmi, F. Aust, E. Awtrey, Š. Bahník, F. Bai, C. Bannard, E. Bonnier, R. Carlsson, F. Cheung, G. Christensen, R. Clay, M.A. Craig, A. Dalla Rosa, L. Dam, M.H. Evans, I. Flores Cervantes, N. Fong, M. Gamez-Djokic, A. Glenz, S. Gordon-Mckeon, T.J. Heaton, K. Hederos, M. Heene, A.J. Hofelich Mohr, F. Högden, K. Hui, M. Johannesson, J. Kalodimos, E. Kaszubowski, D.M. Kennedy, R. Lei, T.A. Lindsay, S. Liverani, C.R. Madan, D. Molden, E. Molleman, R.D. Morey, L.B. Mulder, B.R. Nijstad, N.G. Pope, B. Pope, J.M. Prenoveau, F. Rink, E. Robusto, H. Roderique, A. Sandberg, E. Schlüter, F.D. Schönbrodt, M.F. Sherman, S.A. Sommer, K. Sotak, S. Spain, C. Spörlein, T. Stafford, L. Stefanutti, S. Tauber, J. Ullrich, M. Vianello, E.-J. Wagenmakers, M. Witkowiak, S. Yoon, B.A. Nosek
Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 (Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results. © The Author(s) 2018.
Organisational Behaviour, University of Sussex Business School, United Kingdom; Organisational Behaviour Area, INSEAD Asia Campus, Singapore; Department of Psychology, University of Virginia, United States; Department of Philosophy, Sociology, Education and Applied Psychology, University of Padua, Italy; Department of Psychology, University of Cologne, Germany; Department of Management, University of Cincinnati, United States; Department of Management, Faculty of Business Administration, University of Economics, Prague, Czech Republic; Department of Management and Marketing, Hong Kong Polytechnic University, Hong Kong; Department of Psychology, University of Liverpool, United Kingdom; Department of Economics, Stockholm School of Economics, Sweden; Department of Psychology, Linnaeus University, Sweden; School of Public Health, University of Hong Kong, Hong Kong; Berkeley Institute for Data Science, University of California, Berkeley, United States; Department of Psychology, College of Staten Island, City University of New York, United States; Department of Psychology, New York University, United States; Faculty of Economics and Business, University of Groningen, Netherlands; Division of Neuroscience and Experimental Psychology, University of Manchester, United Kingdom; Westat, Rockville, MD, United States; Department of Marketing and Supply Chain Management, Temple University, United States; Department of Management and Organizations, Kellogg School of Management, Northwestern University, United States; Department of Psychology, University of Zurich, Switzerland; Washington, D.C, United States; School of Mathematics and Statistics, University of Sheffield, United Kingdom; Swedish Institute for Social Research (SOFI), Stockholm University, Sweden; Department of Psychology, Ludwig-Maximilians-Universität München, Germany; College of Liberal Arts, University of Minnesota, United States; School of Management, Xiamen University, Malaysia; College of Business, Oregon State University, United States; Department of Psychology, Federal University of Santa Catarina, Brazil; School of Business, University of Washington Bothell, United States; School of Mathematical Sciences, Queen Mary University of London, United Kingdom; School of Psychology, University of Nottingham, United Kingdom; Department of Psychology, Northwestern University, United States; School of Psychology, Cardiff University, United Kingdom; Department of Economics, University of Maryland, United States; Department of Economics, Brigham Young University, United States; Department of Psychology, Loyola University Maryland, United States; Rotman School of Management, University of Toronto, Canada; Department of Social Sciences and Cultural Studies, Institute of Sociology, Justus Liebig University, Giessen, Germany; United States Military Academy at West Point, United States; Department of Marketing and Management, SUNY Oswego, United States; John Molson School of Business, Concordia University, Canada; Lehrstuhl für Soziologie, insb. Sozialstrukturanalyse, Otto-Friedrich-Universität Bamberg, Germany; Department of Psychology, University of Sheffield, United Kingdom; Department of Psychological Methods, University of Amsterdam, Netherlands; Poznań, Poland; Center for Open Science, Charlottesville, VA, United States
