I just completed a job as an Associate Chair for ACM SIG CHI Work-in-Progress 2015.
It was my first time writing a meta-review, and I (like everybody else) hunted online for a guide to writing meta-reviews. There was none. So I decided to write one, based on my learning points as I did this meta-review.
EDIT: I have found this Mobile HCI guide to reviewing that is relevant to this discussion.
I completed 7 meta-reviews, which means reading 3-4 reviews per paper, as well as the paper itself. It’s not a lot of experience, so I think there will be amendments to this list of tips in the future. Suggestions are also welcome.
- Get the conditions for acceptance from the relevant conference/journal. For ACM SIG CHI, this document is my main guideline. In addition, for Works-in-Progress at CHI, there is this additional document.
- If you don’t get a set of guidelines like this, your conditions for recommending the acceptance/rejection of a paper will not match the other meta-reviewers’. Although the Paper Chairs will moderate by looking at the justifications for acceptance/rejection, it will make their job easier if your rules match everybody else’s.
- Make sure you understand the conditions for acceptance, and direct questions to the Paper Chairs or to more experienced meta-reviewers if you know any.
- If you have the opportunity, select papers you are interested in. I found it easier to find reviewers because I chose papers that are all within one field of expertise, mine. So I know the field and I know the key authors.
- Download and print all the papers. Read them. Annotate them. Although you are not reviewing the papers, you will have to judge the reviewers’ reviews for fairness and justification of their response, so it’s only fair if you read the papers through, at least once.
- After Step 2 has been completed, start recruiting volunteer reviewers. Generally you are looking for two things – a reasonable amount of expertise in the content of the paper, and an indication that this person would be interested to review this paper. Usually, that means they are conducting similar research.
- Some tips for getting appropriate reviewers include:
- Look at the reference list for the paper. After ensuring there is no conflict of interest, you can try approaching people who have written those papers, to review the current paper.
- Search online for similar keywords in Google Scholar/your domain-specific database, then look at the authors of those
- Call on your network of fellow academics. Knowing people always helps.
- Beg your friends (just kidding, though some would say that works too)
- In your recruitment email, explain why you think this particular person is needed for this paper. It helps them to know they are needed, and clearly shows that you have done your homework recruiting them. It is not just a random recruitment.
Sit back and wait for reviewers to accept and complete the review. At this point in time, keeping an eye on the list of reviewers who have NOT yet responded to your invitation is a good idea. Depending on your timeline, you may want to check in with them after a few days. That way you can quickly take them off the list and put in new reviewers. The closer you are to the deadline, the less likely a reviewer will accept doing the review. So be a desk jockey, and manage the reviewer recruitment process. I checked once a day, and after three days, I politely prompted the reviewer to accept, while offering a deadline for that acceptance. If there was no response within 24 hours, I swapped out the reviewer for a new one. Rinse and repeat.
When the reviews are complete for a paper, do the meta-review.
- Read all the reviews.
- Keep your own annotations (Step 2) in front of you.
- In a separate document, copy and paste each point that each reviewer made for or against the paper, making sure to note which reviewer it came from. Eg. R2: “this paper needs more data”
- When you are done with that, take an overall view of the comments, and start to group them into themes. Name the themes, eg. “Lack of convincing data” or “Key strengths”
- Now start writing the meta-review. The way I see it, a meta-reviewer’s job is to summarize the reviewer’s comments into one main document, balancing the differing opinions and adding your own where necessary. When opinions conflict, look at the reviewer’s expertise in the area to disambiguate, or look at the paper to verify which position seems more justifiable. Your particular conference may have a suggested format. Mine says “Strengths”, “For Improvement” and “Recommendation”.
- Make a recommendation on whether to accept or reject the paper. Justify your recommendations based on the conditions you elicited in Step 1. In other words, look at the conditions for acceptance and see if it matches the points you see in front of you.
- Edit your meta-review. The authors will read it, so it is good to take a kind, but objective tone.
- SUBMIT and congratulate yourself for a job well done. 🙂