CODECHECK

Declarations

Affiliate editor of bioRxiv; editorial board of Gigabyte.

Acknowledgements

Mozilla mini science grant, UK Software Sustainability Institute, NWO. Editors @ Gigascience, eLife, Scientific Data.

British Neuroscience Award Team Credibility Prize (2024).

Slides

HTML slides (CC BY 4.0) are available at https://tinyurl.com/cdchk2604 (Grant McDermott).

What started this?

CODECHECK in one slide

1. We take your paper, code and datasets.

2. We run your code on your data.

3. If our results match your results, go to step 5.

4. Else we talk to you to find out where code broke. If you fix your code or data, we return to step 2 and try again.

5. We write a report summarising that we could reproduce your finding.

6. We work with you to freely share your paper, code, data and our reproduction.

Premise

We should be sharing material on the left, not the right.

“Paper as advert for Scholarship” (Buckheit & Donoho, 1995)

CODEOCEAN

Live demo

https://codeocean.com/explore?query=Nature%20Neuroscience&page=1&filter=all&refine=journal

e.g. let’s see: Neural basis of concurrent deliberation toward a choice and confidence judgment - public (Christopher R Fetsch & Miguel Vivar-Lazo)

direct link

Concerns

• As author: can take a long time to establish reproducible capsule.

• As reader: Everything runs in cloud, rather than your computer. Free cloud compute is likely to be very limited.

• Some jobs may take weeks to run…

The CODECHECK philosophy

• Systems like Code Ocean set the bar high by “making code reproducible forever for everyone”.

• CODECHECK simply asks “was the code reproducible once for someone else?”

• We check the code generates expected number of output files.

• The contents of those output files are not checked, but are available for others to see.

• The validity of the code is not checked.

• What does it mean for two results to be “the same”?

Dimensions for codechecks

Where is the neuroscience?

• This is not neuro-specific; we work across disciplines.

• Depending on your project, there may be data to analyse, or simulations to run.

• We did several reproductions of Covid papers, including the Imperial “Report 9” model.

Case study: regulation of pupil size in natural vision across the human lifespan (Lazar et al 2024)