CODECHECK

Independent execution of computations underlying research articles

Stephen J Eglen
sje30@cam.ac.uk

University of Cambridge

Daniel Nüst

TU Dresden

April 24, 2024

Declarations

Affiliate editor of bioRxiv; editorial board of Gigabyte.

Acknowledgements

Mozilla mini science grant, UK Software Sustainability Institute, NWO. Editors @ Gigascience, eLife, Scientific Data.

Thank you BNA for awarding us the Team Credibility Prize 2024.

Slides

Slides (CC BY 4.0) are available in HTML format from https://tinyurl.com/codecheck2024-02. (Grant McDermott).

CODECHECK in one slide

  1. We take your paper, code and datasets.

  2. We run your code on your data.

  3. If our results match your results, go to step 5.

  4. Else we talk to you to find out where code broke. If you fix your code or data, we return to step 2 and try again.

  5. We write a report summarising that we could reproduce your finding.

  6. We work with you to freely share your paper, code, data and our reproduction.

Premise


image/svg+xml Data Code Models Results Stats Paper .PDF In the lab Outside Notebooks Sharing ReuseTransparencyCollaboration Images: Noto Emoji (Apache 2.0)

We should be sharing material on the left, not the right.

“Paper as advert for Scholarship” (Buckheit & Donoho, 1995)

The CODECHECK philosophy / 1

for who? how long? CODECHECK everyone CODE OCEAN "for ever"

for who? how long? CODECHECK everyone CODE OCEAN "for ever" one person CODECHECK one pointin time

The CODECHECK philosophy / 2

  • We check the code generates expected number of output files.

  • The contents of those output files are not checked, but are available for others to see.

  • The validity of the code is not checked.

Where is the neuroscience?

  • This is not neuro-specific; we work across disciplines.

  • Depending on your project, there may be data to analyse, or simulations to run.

  • We did several reproductions of Covid papers, including the Imperial “Report 9” model.

Case study: regulation of pupil size in natural vision across the human lifespan (Lazar et al 2024)

Who does the work?

  1. AUTHOR provides code/data and instructions on how to run.

  2. CODECHECKER runs code and writes certificate.

  3. PUBLISHER oversees process, helps depositing artifacts, and persistently publishes certificate.

Who benefits?

  1. AUTHOR gets early check that “code works”; gets snapshot of code archived and increased trust in stability of results.

  2. CODECHECKER gets insight in latest research and methods, credit from community, and citable object.

  3. PUBLISHER Gets citable certificate with code/data bundle to share and increases reputation of published articles.

  4. PEER REVIEWERS can see certificate rather than check code themselves.

  5. READER Can check certificate and build upon work immediately.

Our register of certificates

https://codecheck.org.uk/register/

See for example certificate 2020-010 (Imperial’s “Report 9”).

Figure 1

Limitations

  1. CODECHECKER time is valuable, so needs credit.

  2. Very easy to cheat the system, but who cares?

  3. Author’s code/data must be freely available.

  4. Deliberately low threshold for gaining a certificate.

  5. High-performance compute is a resource drain.

  6. Cannot (yet) support all thinkable/existing workflows and languages.

Next steps

  1. New project https://codecheck.org.uk/nl aiming to produce 50 codechecks for papers from Netherlands.

  2. Embedding into journal workflows.

  3. Training a community of codecheckers.

  4. Funding for a codecheck editor.

  5. Come and get involved

Further information: http://codecheck.org.uk and our research article.