Week 9
Researchers are incentivized to be first, not right
The Challenge
Historically, academic institutions have mostly rewarded single actors who are the first to publish important or impactful results. Universities hired and promoted scholars based mostly on their publication records. Agencies, such as the NIH and NSF, evaluated scholars on their publication records. This system incentivized researchers to exaggerate their individual contributions, and potentially to hoard data and hide mistakes, in the interests of competition. It might also have incentivized individual labs and researchers to work alone, even when collaborations could be better for scientific progress. How can we fix the incentives?
Yarkoni, T (2018) It's not the incentives it's you
A 5min long discussion about incentives during promotion and hiringfrom the launch of CORES, centre for reproducible research at Stanford; minutes 41-46.
How to change incentives: The Wellcome Foundation’s “Guidance” for research organisations on how to implement responsible and fair approaches for research assessment.
Should advertising of ‘open’ behaviors be opt-in or by external audit? Episode of the Everything Hertz podcast discussing controversy about transparency audits (start at 3:30)
Tiokhin, L., Yan, M., & Morgan, T. J. (2021). Competition for priority harms the reliability of science, but reforms can help. Nature human behaviour, 1-11.
The Tool
Practical skills activity
1. Here is the job description for the open tenure-track position in BCS and Picower. In response to input from last year’s class, the language has been updated. What do you think of the current nod to open science? What changes would you recommend?
2. Look at the BCS and School of Science websites. How could these organizations express commitments to open science that would fulfill e.g. the Wellcome Trust “guidance”? Suggest language, and where it would go on the website.
3. Find a journal that uses Open Science badges, and find a paper that has badges. Do the authors include the corresponding badge on their CV or website?
4. Find an author in a discipline close to yours who advertises their open science practices on their CV or personal website.
Useful links and resources
Open science badges And a discussion on twitter about whether badges are a good idea or a bad one.
Open Research Funder’s group:Incentivizing the sharing of research outputs through research assessment: a funder implementation blueprintDownload Incentivizing the sharing of research outputs through research assessment: a funder implementation blueprint
Moher, D., Naudet, F., Cristea, I. A., Miedema, F., Ioannidis, J. P., & Goodman, S. N. (2018). Assessing scientists for hiring, promotion, and tenure. PLoS biology, 16(3), e2004089.
Open science at universities, including example language for job applications and policies
The National Academies Roundtable on Aligning Incentives for Open Science; the full reportDownload full report from their most recent workshop.
OurResearch.org has a whole bunch of interesting projects to increase openness, including some specifically about getting credit for scientific softwareand other kinds of impact
Transparency and Openness Promotion (TOP)guidelines: resources
An empirical project for assessing the credibility of research claims, SCORE.
Final report of MIT’sAd Hoc Task Force on Open Access to MIT's Research and the announcement of the Inaugural MIT Prize for Open data.
Marder, E. (2017). Scientific Publishing: Beyond scoops to best practices.Elife, 6, e30076.
Alperin, J. P., Nieves, C. M., Schimanski, L. A., Fischman, G. E., Niles, M. T., & McKiernan, E. C. (2019).Meta-Research: How significant are the public dimensions of faculty work in review, promotion and tenure documents?. ELife, 8, e42254
Singh Chawla, D. (2016). The unsung heroes of scientific software.Nature News, 529(7584), 115.
MIT has a longstanding relationship with an external company called Academic Analytics. But the use of this company’s data in academic decision making at other institutions is controversial. Here’s the faculty report about the controversy at Rutgers: Report and Resolution on Use of Academic Analytics