Week 5

Mistakes go uncaught and uncorrected.

The Challenge

The scientific process is supposed to accumulate better information, and identify and discard mistakes and false beliefs. Incorrect information or interpretations can be introduced into the literature and persist for a while; but ideally, these errors will be found and weeded out. This week, we will talk about how mistaken or false information enters the scientific literature, how it can be prevented, how it can be found, and what happens when it is. In a later week, we will also talk about the professional incentives that promote or prevent people from correcting mistakes. 

Talk by Simine Vazire at Open Science 2019.

Bishop, D. V. (2018). Fallibility in science: Responding to errors in the work of oneself and others. Advances in Methods and Practices in Psychological Science, 1(3), 432-438. 

This twitter thread a grad student wrote about finding a bug in their code, and reckoning with the consequences. Sep 6 2021. 

At least the abstract of: Hosseini, M., Hilhorst, M., de Beaufort, I., & Fanelli, D. (2018). Doing the right thing: A qualitative investigation of retractions due to unintentional error. Science and engineering ethics, 24(1), 189-206

In part 1 of your response paper, describe something you learned from these resources you didn’t already know, and/or a personal experience you’ve had that really brought home the challenge of catching and correcting mistakes and errors, in your own science or in your lab or in your field. 

The Tool

Practical skills assignment

1. Use a tool for making dynamic executable document (R Markdown; Jupiter Notebook; Matlab Notebook) to write this week’s assignment. In your executable document, use your own data (or data from another paper) to make at least one reproducible figure. 

2. Zotero has partnered with Retraction Watch to automatically flag retracted papers in a user’s repository. Download Zotero if you do not already have it, and either import a repository from your current citation manager, or manually import a few papers of your choice if you do not use a citation manager. Look through one of your repositories, and see if any papers are flagged as retracted. If not, add a retracted paper (you can find many on Retraction Watch’s database) to the repository. Is it correctly flagged as retracted? Do you think this flag is noticeable enough to be a useful tool in preventing use of retracted papers?

 

In part 2 of your response paper, describe what you did in fulfilling this activity. What snags did you hit? What made this process easier or more difficult? Did you find any errors?

The Critical Evaluation

The Challenge: Incorrect information or interpretations can be introduced into the literature and persist for a while; but ideally, these errors will be found and weeded out. This paragraph could describe something specific you learned from the readings you didn’t already know, and/or a personal experience you’ve had that really brought home the challenge of science as self-correcting.

The tool: Describe what you did in fulfilling this activity. What snags did you hit? What made this process easier or more difficult? Did you find any errors?

  • Download RStudio and complete the RMarkDown lessons. Try producing a figure from data you’ve collected or data from your lab. If you don’t have access to such data, find an openly shared RMarkDown document and reproduce a figure from that paper.

  • Download Zotero if you don’t already have it, and check for retracted papers. If none are flagged, add a retracted paper from Retraction Watch and see if Zotero correctly flags it as retracted. Is this flag noticeable enough to be a useful tool in preventing use of retracted papers?

Critical evaluation of the tool. What is the promise of this tool in addressing this challenge? What are the biggest obstacles? 

This response paper should be about a page long, single-spaced.