Benefits to authors of non-anonymous preprints.
Written by Dmytro Mishkin and Amy Tabb, August 21, 2020.
One of the most popular reactions to our previous post was: “we generally agree, so let’s allow preprints before acceptance, but in anonymous form only.” The current post addresses this idea, and to do it properly, we need to go back to the essentials and try to answer these questions: “What is a scientific publication for?,” “What are the functions of peer review?” and “How is it related to science?” Prepare yourself for a long read.
The first point is that “science” as a process of mining and updating knowledge by using the scientific method does not equal modern “science” as a set of management and organizational routines (grants, conferences, publications, etc.). This latter form of science, which we designate science-as-implemented, is just a current form of how science is performed in modern society. Having that in mind, let’s revisit the purpose of scientific publications and scientific conferences in computer science and engineering in particular.
For the author, conferences provide:
Out the six functions, only 2.5 (dissemination, feedback, and part of certification) are related to the science as a knowledge mining process, or the first definition of science. We will get back to it shortly, now functions for the audience:
For the audience:
Partly “certification” serves science as a knowledge mining process, reducing a barrier to build on top of others’ work. The rest of the functions serve the science-as-implemented current model of professional scientific work and help the community to cope with resources (time, money, attention) scarcity.
So far, so good. All of these functions are built on top of double-blind peer review as the certification engine for most cases. However, there are two problems with these conference functions currently: peer review quality and inherent subjectivity, caused by the prefiltering function. The first problem is, at least in an ideal world, solvable, while the other is not – by design.
Peer review quality is neither 100% precision (all accepted papers are good), nor 100% recall (all good submissions are accepted). Let’s forget about recall for now and focus on a precision for a moment. There is quite a bit of evidence, that papers, passed through peer review could contain critical flaws, methodological and factual errors. For example,
Let’s recall now the Prefiltering and Formal goodies functions of the conference. We argue that the main point of the current review process is not answering the question: “Are the results in the paper correct?,” instead: “Is the paper worthy to be published at our precious conference?” And while the first question is related to the paper’s technical aspects and is mostly objective, the current one is subjective by definition. For example:
What’s the problem with this rule? It implies that bad reviews are bad solely because of factual errors. It implies that writing a technical paper is primarily about providing objective or factual information, and that the more subjective aspects of paper-writing aren’t important. The first implication is obviously false, and the second one is more subtly wrong.[…] In a sense, most computer science papers have two components. The first are technical components: theory, algorithms, and experimental results. The second, however, is an argument about why the technical components are significant.” (emphasis ours)
What is the place of double-blind peer review here? It (let’s assume it is true) increases the fairness of the answer on the “Is the paper worthy to be published at our precious conference?” question by removing bias towards big names, because otherwise a reviewer could think “If X has done this, then it would be worthy”. Thus, double-blind review should assure also fairer credit assignment, and all the rest of the attributes and benefits of science-as-implemented.
Note, the point above has nothing to do with science as knowledge mining. The found fact about the universe is fact regardless of who found it and who would be credited for it. The benefits of double-blind review lie not in the science area, but in the science-as-implemented area.
This doesn’t undermine the importance of the fairness, credit assignment or benefits of double blind reviews. Rather the opposite – we are trying to draw the reader`s attention to the bigger picture before jumping to conclusions.
Having established that, let’s overview, what are the problematic aspects of anonymous preprints for those who are supposed to benefit from double-blind peer review, i.e. early career researchers from unknown labs.
In our previous post, we did not handle anonymous preprints deeply. They are not currently implemented in venues where we submit papers. But we do have thoughts about them.
We consider anonymous preprints to be a special case of delaying release of a paper until acceptance. Requiring anonymity has harms that impact the research community, but particularly early career researchers (ECRs) as explained in “Hands off arXiv!”. However, we will explain some of these issues in more detail.
After our previous post, besides comments about anonymous preprints, there were other suggestions about how authors could “just wait n months”, “just submit application packets this other way,” “just allow the system to correct for x, y, z”, “just…” do more work or wait to be noticed. We see these suggestions as contortions to get around barriers imposed by one’s own research community. Furthermore, we see these contortions as unnecessary, because of the greater cost to those without stability or security. Not everyone in our research community can assume security in their employment, visa status, finances, health, etc. It is our hope here to outline more clearly what some of these costs are to ECRs with respect to the delay of preprints and the special case of anonymous preprints.
We will discuss costs to ECRs from the opposite direction – by discussing all of the opportunities that come from non-anonymous preprints. Without non-anonymous preprints permitted, the costs then are missed opportunity costs. Keep in mind, that ECRs from small labs typically have fewer opportunities than ECRs from large labs and universities very well known in the field.
The most obvious benefit of non-anonymous preprints is the straightforward distribution of authors’ research to the community. This research and ideas can then be clarified and explained through interaction or blog posts out in the open. Additionally, public code and data releases could be done without the administrative overhead of keeping track of anonymity at all stages. Code contributions could then also be accepted. While the research could also be distributed in the anonymous preprint case, the need to maintain anonymity restricts some forms of feedback from the authors.
The benefits to authors of non-anonymous preprints, particularly ECRs, is that they can build professional identities around the work instead of waiting until acceptance. Activities include collaborations, building informal networks of support outside of their home institution, and organizing events. All scholarly work can be listed on their C.V. – including preprints –- to show research output.
Increased visibility through preprints allows others to become aware of these individuals. Examples include recruiters, who in our experience, contact candidates versus candidates submitting application packets. Moreover, other examples include fellow researchers who are looking for speakers at local meetups, non-academic conferences, workshops, departmental seminars, and so on.
On the topic of the “wait 3-n” months argument, there has recently been some discussion about conference acceptances as a requirement for acceptance to Ph.D. programs and some jobs (see Andreas Madsen for one account). We see this as a troubling trend. Under the prior requirements that one must travel for a paper publication, we want to explore what a top conference paper acceptance signals on a candidate’s C.V. One, it signals good quality work (in the technical sense) whose arguments for value are accepted by the community. But what else does it signal?
(this is a non-exhaustive list.)
Once you start to interrogate these signals, I hope you get the idea that requiring top conference acceptances would favor those with the luck to fulfill all of these conditions. If you want to change the demographics of your departments and companies and you use top conference acceptance as a criterion, you should have a deep think about what that really means.
While virtual conferences have alleviated some of these issues, they do not alleviate these issues from [ – 2019], and there is no guarantee that virtual presentations will be allowed if / when covid-19 ends in the future. Moreover, some conference attendance and printing fees remain expensive, e.g. £450 for virtual ECCV 2020 (point 1).
Now as one can see, double-blind review combined with banning named preprints does not come for free. While removing one bias towards big names, it introduces other biases and limitations, which can be more harmful than the originals.
Is it worthwhile to preprint? It depends on a specific situation, that is why we are arguing for allowing researchers to decide how to share their work.
Stay safe and sane.
Sincerely yours, Dmytro Mishkin and Amy Tabb.
© Amy Tabb 2018-2021. All rights reserved. The contents of this site reflect my personal perspectives and not those of any other entity.