CODECHECK

Declarations

Affiliate editor of bioRxiv; editorial board of Gigabyte.

Acknowledgements

Mozilla mini science grant, UK Software Sustainability Institute, NWO. Editors @ Gigascience, eLife, Scientific Data.

British Neuroscience Award Team Credibility Prize (2024).

Slides

HTML slides (CC BY 4.0) are available at https://tinyurl.com/cdchk2606 (Grant McDermott).

CODECHECK in one slide

1. We take your paper, code and datasets.

2. We run your code on your data.

3. If our results match your results, go to step 5.

4. Else we talk to you to find out where code broke. If you fix your code or data, we return to step 2 and try again.

5. We write a report summarising that we could reproduce your finding.

6. We work with you to freely share your paper, code, data and our reproduction.

Premise

We should be sharing material on the left, not the right.

“Paper as advert for Scholarship” (Buckheit & Donoho, 1995)

The CODECHECK philosophy

• Systems like Code Ocean set the bar high by “making code reproducible forever for everyone”.

• CODECHECK simply asks “was the code reproducible once for someone else?”

• We check the code generates expected number of output files.

• The contents of those output files are not checked, but are available for others to see.

• The validity of the code is not checked.

• What does it mean for two results to be “the same”?

Case study: regulation of pupil size in natural vision across the human lifespan (Lazar et al 2024)