My reviewing style, or how to review technical papers when you've not been taught how.
09 Jun 2019This post concerns the basics of reviewing an engineering or computer science paper, with an emphasis on how best to conserve your, and the authors’ time. Other resources has some other guides I found when I was writing this post.
June 9, 2019, updated resources July 31, 2019
Back to Tips and Tricks Table of Contents
Back to the academic papers section
I did not have any instruction in reviewing – though evidently this exists at some universities. I adopted my current approach by:
- reading reviews of my own papers.
- when I review, look at what the other reviewer did, if this is available.
- observing author responses to my and other reviewers’ reviews (for journal papers).
- observing complaints about reviewers on Twitter: Reviewer 2/@GrumpyReviewer2, Sh*tMyReviewersSay/@YourPaperSucks.
My philosophy on paper assessment.
The golden rule of reviewing is to review others’ papers as you would have them review your papers. If the paper moves the field forward (a few different interpretations here), is based on sound scientific principles, and I can understand it by reading it twice (bonus points for once!), then I am likely to be positive about the paper in my assessment. It may be that I am not a personal fan of the technique, have found that it does not work in my domain, etc., but I try to read the paper as objectively as possible and assess it according to the criteria above. There is a lot of commentary about the sheer number of papers coming out – and yes, as a reader, there is a lot to filter through these days – but the reward structures of many institutions are not likely to change soon regarding publications.
Here’s the basic structure of my reviews. Note that I review conference and journal papers differently, because the structure of submission for those papers is different (page limits, revisions, etc.).
Summary.
In this section, you summarize what the paper is about, its contributions, and impact. The purpose of this paragraph is mainly for the editor to get a sense that you have adequately understood the paper. The editor may use some of this information to generate reports. Your insight is needed! This summary does not need to be long – 3-4 sentences is fine.
Positives and Negatives of the paper.
Assessment:In this section, I’ll write a short paragraph that would support whatever rating I would give the paper (accept, borderline, reject, etc.). Note, that reviewers will not ultimately be in control of accepting or rejecting the paper, so do not list the rating in your review. The editorial team is in charge of those decisions. This paragraph highlights either the biggest strengths, or biggest weaknesses (such as #3), of the paper. Even if the paper is not in good shape, I try to say one positive thing about the work. Most papers I review have graduate student authors, and reviews can be crushing.
In the case that you feel very positively about the paper, you really need to argue for it with the same kind of specificity as you would when the paper has problems. The editor will look at all the reviews and come up with a composite score. If they sees scores like: “blah, neutral, negative”, and yours was the “blah” but should have been “this is great! (with evidence)”, then your lack of work on behalf of the paper will likely affect its outcome. Alternatively, an entire review that reads like: “This paper is awesome/terrible, Q.E.D.” gives little evidence to help the editor make a decision about the paper.
- Enumerated list. Then, if there are criticisms, I create a numbered list of issues. I am very specific. No, “I don’t like this,” or, “it isn’t novel enough.” You have one chance to communicate to the authors – this is a one-way communication! – so it is important to be very clear.
-
By specific, I list line numbers or sections, quote sections of the text so the authors can search for the string, and then explain the criticism. I may use citations to support my criticism. The criticism should be about the work in the paper, not about the authors, their institution, or anything else.
-
My criticisms these days tend to focus on mathematical correctness, missing experimental details, and overstated claims. Many of the items are from my basic submission checklist, and depending on the severity, it may be difficult to understand the paper. Some of these items are easy to fix for the authors, and some are not. In my short paragraph heading this section that gave my assessment of the paper, I reference each problem by its number. For papers that are in very good shape, I may rate each item in terms of how much of a problem I see it in the paper (major or minor), though typically I stick the minor problems in the suggestions section. When the paper is not in good shape, I do not go into detail on minor problems. Conserve your time and stick to major problems in that case.
-
If you find yourself thinking that a particular technique would work really well for the project contained in the paper, even though the paper currently is complete, do not require new experiments. There’s another section for that - Suggestions!
Suggestions
I use this section for comments to the authors. At the heading to this section, I specifically mention that the authors do not have to respond to these suggestions – these comments are for the authors’ information only. The editors, in my experience, will send email to the authors that says they have to respond to every point that the reviewers make. Some authors, then, will sometimes do much more work than the reviewers intended.
Here, I put:
- Literature that I think is related, but is not critical to include. This is great for really recent papers, or potentially tangential topics. Examples are arXiv or new conference papers. Sometimes, authors are not aware of key papers in the field, or miss out of entire portions of the field in their literature review; in that case, it is appropriate to suggest these papers in the Positives and Negatives of the paper section. On the other hand, it is not reasonable to expect authors to know about papers that came out 4-6 months before you received the paper. Usually one’s literature review is completed early on in the writing process, and there is some delay between submission and the paper arriving in your hands. I have a flowchart in another post about how to make these decisions, here.
- “Have you tried . . . “ experimental ideas for the authors. I tend to suggest robustly evaluating parameter choices and assumptions within the current experiments in the paper. With the conference paper page limits, sometimes there is not space for these experiments and discussions so I list them here. For the robotics conferences, there’s no rebuttals or opportunity for revision, so putting these details within the suggestions section signals to the editor that you do not consider these items critical for acceptance. This discussion brings up that the way you review should adapt to the structure of the conference or journal and their guidelines.
- List public datasets that may be relevant to the work, particularly if they are not well-known. In agricultural robotics, we don’t have a great system for publicizing datasets, so this comes up occasionally.
Typographical errors and writing style.
I list these in a separate section. If I find editing errors (such as a few subject/verb agreement mismatches or garbled sentences) or typos, I will list them so the authors can correct them on their next draft. If the paper has one or two writing quirks – a persistent spelling error or grammar problem, for instance – I will mention that. Otherwise, if the paper has a lot of writing problems I will mention in the Positives and Negatives of the paper section that the paper needs an edit for English grammar and style, with a specific explanation of the problem and why that made it hard to understand the work of the paper. I got a reviewer’s comment concerning the writing (“revise the paper for English grammar and style”), on a paper with more authors than usual for me, but supervised by a frequent co-author who is an excellent writer. I re-read the paper and could not find any problems. So specificity helps, if you make these kinds of comments.
Correcting others’ writing style – when the writing is essentially correct – is pedantic and a waste of everyone’s time. Don’t do this. Also, don’t go in for the British versus American (or other styles of) English divide. Let that war die (I studied for a year of undergrad in the U.K.; I’ve seen that play out in certain ways). Sometimes on a second reading, I understand another’s style much better; it is just a matter of getting used to it. I usually end up reading papers I review twice, with some sections requiring re-reads.
Timing, and getting it done.
Personally, my strategy is first, to turn down any review with a less than three week review time, unless it is a second review for a journal paper I have already reviewed or an emergency review for a conference. Then, I print out the paper, and put it in my bag. I also put a note on my calendar for when it is due. I usually have 1-2 papers in my bag, and while I wait in appointments, or am on a plane, I’ll review and make margin notes. Some papers are so poorly written, I need multiple sittings to get through it. I will then write up my notes in a text file. This is convenient for journal papers, because sometimes the authors ignore what you wrote the first time about something major, and then you can just copy and paste it back. Or you might get sent the paper again, from a different journal, unchanged. Also, I don’t need an Internet connection to edit the text file, so I can do this in a variety of places without the editorial manager timing out and losing my review.
Then I put the review on hold for 24 hours. Sometimes I am so frustrated by poor writing that this cool-off period is needed. Then I revise the review for clarity, putting myself in the mind of the graduate student who will probably read it. I fight with the editorial manager tool from the journal, in some cases also answer 20 questions (I just write: “see point #x, y, z” from Positives and Negatives of the paper section), and submit.
The timeline when reviewing conference papers is a lot different than when you are reviewing journal papers. If you agree to review a conference paper, it is important to review the paper close to the time agreed to. At the very least, you need to communicate with the associate editor about your timeline so they can make a plan. I was an associate editor for the first time for ICRA 2019, and the review deadline was close to the CVPR submission deadline; this caused a domino effect of delays with the reviews I was handling. As long as I have some return contact with the reviewer, I’m fine, but when people ghost, that makes it really hard. Are they ill, traveling, completing it but not done yet, and do I need to get another reviewer?
Special cases.
These are some special cases, and in these cases, you’ll need to both conserve your time (the first three) or handle the ethics of reviewing appropriately.
The incomplete conference paper.
Conferences are great. Almost every paper will be reviewed, except for papers that violate explicit rules or are over the page limits. Sometimes authors will send in incomplete papers, though. A variation on this, is that a complete paper is turned in, but the writing is a rough translation of another language/Google translate and it is obvious the advisor did not help edit it.
I get that conference deadlines are six months apart, and sometimes it is good to take advantage of them. So I do review these papers, but I gauge how much time I spend on how complete the paper is. If the page limit is 6 + n, and the paper is only 5 pages and is missing key sections, this paper is not likely to get accepted. For the case that the paper is obviously a graduate student effort with little advisor involvement, I spend more time, because for whatever reason the advisor is not involved. I am sympathetic to graduate students.
The hot mess journal paper.
Sometimes, there are so many problems that if you explained all of them, it would take a day or more. Do not take days out of your own program for someone else’s paper. Especially if these problems are of the mathematical variety, once you find one or two big problems, it is up to the authors to fix the paper and resubmit. It may be that one problem will change the rest of the paper as well, triggering a chain reaction.
You are a specialist reviewer.
In this case, the paper has a broad scope and you have been brought in to review 25% of it because you are a specialist on that topic. In my case, this sometimes happens with Plant Biology papers – I am no biologist, so I do not attempt to review those sections. Instead, I usually have been asked to review because of my expertise in plant phenotyping, so I state that up front in the review, that I am an expert in X, and so I have reviewed the paper concerning topic X. Then, the editorial team and author know why one reviewer is focusing on one set of issues, while the other reviewers are not.
Plagiarism, Conflict of interest, or other not-fun stuff.
If you suspect that there is a case of plagiarism, or after getting the paper you have a conflict of interest, communicate with the editor and explain the situation. They have a lot of experience with these things, and can decide appropriately. In addition, if you get the paper and see that you do not have the expertise to review, you also need to communicate with the editor as soon as possible. It is better to speak early, so that others can make a plan, than to fake your way through.
Conclusion.
This is my personal take on reviewing. It still takes me a longer time to review than I prefer, but I feel that I am getting more efficient. If you have tips, let me know on Twitter @amy_tabb.
Other resources since I’ve written this post.
- A. Tabb: Constructive review examples (work in progress).
- Anna Rogers and Isabelle Augenstein: “What can we do to improve peer review in NLP?”.
Other resources.
I had thought that for all the complaining on Twitter about reviewers and reviewing there would be a lot of other guides. There aren’t (or maybe my combination of search terms is too common?), but here are some good ones I found in the course of writing this post:
- Approximately Correct: Is this a paper review? (humor)
- Alan Meier, from 1992 (paper forms, and mail), but good and relevant.
- PLOS One.
- CVPR 2019 reviewing guidelines (this is a really good guide!).
- Niklas Elmqvist, mistakes new reviewers make,general reviewing in HCI.
- Katsushi Ikeuchi: Rights and Obligations. ( some high-level thoughts about reviewing)
Changelog
- July 31, 2019: added Katsushi Ikeuchi’s presentation to Other Resources.
- November 2020: Added the “other resources since I’ve written this post section.”