Code release integrity: if you don't have a public code release, don't say that you do.
30 Dec 2021While reading arXiv papers, I frequently encounter the following:
our code and dataset is at this Github link …
I go to the Github link, and the repository is … empty, or an advertisement for the paper. There is no code!
My advice: don’t write that the code is available when it is not.
I will go through a few scenarios and hopefully give folks an idea of what other actions they can take besides false advertising with respect to the code. First, let’s take a look at what got us here: the Open Science movement and shaming people for not including a code release with a paper. I’ll note that code releases are not required at the venues where I submit papers.
Open science: it is complicated.
Occasionally, researchers are chided by those in the community for not releasing their code. An unfortunate side effect of the open science movement – which promotes code sharing – is that some believe they know what you do or do not have the time, funding, and resources to do. Real people and organizations have real constraints and different ways of assessing performance.
I have written about open science and expectations before. I love it when code is available. However, unless I am going to facilitate the release in some way by offering time or resources and get involved with a collaboration, I think it unreasonable to set any kind of timeline on how someone else is going to spend their efforts in a very different organization.
Author’s viewpoint: I plan to release the code eventually!
In the situation where your paper is ready, but the code cannot be released publicly yet, a statement similar to the below would work:
We plan to release our code at this link …
People interested in our code are encouraged to contact author X at [email].
At your mostly empty repository, you could provide instructions for those who are interested in updates to watch the repository. Alternately, the README could list a contact email and you can send out updates via email.
Then, when you release the code, you can update your arXiv paper concerning the code’s availability. I also add the DOI to papers when they are published, on the front page of the arXiv version. These are standard practices when posting papers on arXiv.
Reader’s viewpoint: I would love to use the code, but it is missing!
In this case, communicate with authors to express that you would like to try their code and why. It may be that you are sent an executable. I have sent non-public code to others upon request.
Even if you communicate, you may get no answer. I have also had this happen.