Constructive review examples.
18 Aug 2020Everyone hates reviewer 2, but how do you avoid being reviewer 2? Here are some examples of constructive feedback. Details matter!
I am working on this over time – catch up with me on Twitter @amy_tabb.
Back to Tips and Tricks Table of Contents.
Back to the academic papers section.
Quick Links.
Example 1. The scenario: a paper that does not identify what the contribution is, or identify how the work relates to other methods.
Reviewer 2 might write: “You did nothing.”
A constructive review:
I was unable to determine what the contributions of the paper were, or how this method relates to existing methods. The introduction mentions methods X and Y for this task Z, but how exactly does this method solve problems that X and Y do not? I did look for this information, through text or an enumerated list, and was unable to find it.”
References: Basic submission checklist # 9.
Example 2. The scenario: the paper is complete, but the reviewer is excited about some potential experiments and new papers out.
Reviewer 2 might write: “This paper needs evaluations on benchmarks X, Y, Z and also needs to cite these recent papers.”
A constructive review:
Suggestions.
These suggestions are for the authors’ information only, and it is not necessary for the authors to respond to any of these points. I was interested to see the experiments with dataset A, and was curious to see how well this method would do on benchmark X. Additionally, while reading this paper I thought benchmarks Y and Z, which were not originally constructed for the task in this paper, are related and I think that this method could be applied to those tasks with some alteration.There are some very new papers out that may be relevant to this work, [ citation ], [ citation ], [ citation ].
References: Reviewing Tutorial, Suggestions; arXiv paper explainer, # 4.
Example 3. The scenario: Great Britain versus USA, just with spellings.
Reviewer 2 might write: “There are too many typos. Minimize->minimise, among->amoung, color->colour, optimize->optimise, etc..”
A constructive review:
THIS SPACE INTENTIONALLY LEFT BLANK
References: Reviewing Tutorial, Typographical errors and writing style (second paragraph).
Example 4. The reviewer is an expert and can only see the failure cases.
Reviewer 2 might writes 2-3 pages about the general field, and goes into great detail about the failure cases.
A constructive review:
Strengths : [summarize the things that the paper does well.]
Weaknesses : [ if the paper did not list any failure cases, mention that there are frequently failure cases with class of methods X, and the paper would be strengthened by an identification and discussion of these failure cases. Examples of failure cases are A, B, and C.]
Suggestions. These suggestions are for the authors’ information only, and it is not necessary for the authors to respond to any of these points. [ If the paper did not discuss failure cases, the reviewer can put some of their commentary and thoughts about extraneous issues here. ]
References: Basic submission checklist # 13. Remember, research is not a zero-sum game.
Example 5. The authors are new to the field. Their method is generally sound, but it is hard to understand given the non-standard vocabulary.
Reviewer 2 might write: “This paper doesn’t make any sense. They aren’t using the established terms of our community.”
A constructive review:
Strengths : [summarize that the things that the paper does well, with the caveats that some sections were difficult to assess.]
Weaknesses : I had a hard time understanding the work in this papers, as the paper uses its own vocabulary to name a common problem in the computer vision community, where it is called \(A\). In the paper, this problem is called: \(B\), \(C\), and \(D\). I was unsure if in the description of the method, the paper was referring to the same problem, or different aspects of the problem.
The naming inconsistency makes it hard to both understand and situate the work with respect to the original field as well as within the paper. References to papers on the topic of \(A\) are [ citation ], [ citation ], [ citation ].
References: Hands off arXiv!, ‘focus’ section.
Example 6. The paper is written by non-native English speakers. There are sections where it is hard to understand the science in the paper because of the writing.
Reviewer 2 might write: “Get a native speaker to edit this paper. I can’t understand it.”
A constructive review:
Strengths : [ … ]
Weaknesses : There were instances where I could not understand the science in the paper because of the writing. One example is: [ copy-paste the text ]. Here, I cannot tell if \(A\) is applied to object \(B\) or to object \(C\). A similar confusion occurs in
[ copy-paste the text ] and
[ copy-paste the text ]
where I cannot tell exactly how the method works by reading the text. An algorithm box to support the text and/or revision may be helpful. I read these sections multiple times each, and at the conclusion, I still was unsure of what was meant by this text.
Suggestions. …
Typos:
- most occurrences of ‘triangle’ -> ‘triangular’.
- etc. other typos if the reviewer has the energy.
References:
- Science Magazine: “Science’s English dominance hinders diversity—but the community can work toward change,” by Rodrigo Pérez Ortega;
- Reviewing Tutorial, Typographical errors and writing style;
- “What can we do to improve peer review in NLP,” Sec. 3, ‘Writing Style’ subsection, by Anna Rogers and Isabelle Augenstein.
If you criticize the writing, criticize it from the view of how it is hard to understand the science in the paper. Of course, we would all love papers that read beautifully. But reviewers are not editors in publishing houses; I believe it is not our role to tell others how to fix their writing, only to explain how we cannot understand their science, if in fact we cannot understand it from the paper.
Example 7. The paper is missing references.
Reviewer 2 might write: “The paper does not cite previous similar work.”
A constructive review:
Strengths : [ … ]
Weaknesses : The paper does not cite previous similar work. The paper’s claims of novelty (claims 2, 3, and 5 on lines 300-340) are reduced when compared to the previous similar work. For instance, in [ citation ] the same problem as in this paper is treated in ways \(A\), but their method is different in ways \(B\) and \(C\). [ citation ] also considers the same problem and domain as this paper and has a comparable method in manner \(D\); and the computational complexity of their algorithm is less than the method is this paper (\(\mathcal{O}(n\log(n))\) versus \(\mathcal{O}(n^2)\)).
Suggestions. These suggestions are for the authors’ information only, and it is not necessary for the authors to respond to any of these points. Here are some recent papers on this problem that may be relevant to your future work. They were released close to your submission date: [ citation ], [ citation ], [ citation ], [ citation ].
References:
This example inspired by a Twitter poll from Dmytro Mishkin and reply from David Picard.
- Reviewing Tutorial, specificity.
- arXiv paper explainer, timing on when to list a recommended paper in a review.