Nuts and Bolts of Peer Review II

In the previous posting, I talked about the lead-up to the peer review process. Here, I'll discuss what I look for when I'm reviewing a paper.

The Text
This is the part of the review that usually takes the most time. I read through the entire paper from start to finish, usually in about two sittings. I find that if I try to plow through the whole thing at once, I'll get a little lazy towards the end (especially if it's a really long paper). As I read, I look carefully at every aspect of the text. Is each section logically and carefully written? Are all of the necessary references cited? Could the authors cite a few more papers? Are any portions of the paper redundant or superfluous? Is more detail needed in some sections?

In the introduction, I look to make sure that the authors really introduce their topic. I want to see a good case made for the necessity of the research, with a clear summary of previous work. It doesn't look good if the authors ignore previous publications on the topic--it gives the perception that they didn't do the sufficient background research or that they're downplaying others' contributions and overplaying their own.

In the methods section, I want to see a full explanation of what the authors did. If there's a novel method, it sure as heck had better be written up in detail. In cases of well-established methods, it's usually ok to just refer to a previous paper.

In the results section, I read through and make sure that the results make sense, and that the various graphs and tables match up with the data. For that matter, are there graphs and tables? If there's a description of specimens, I want to see good, clear text that fully describes the specimens under investigation. Figures, photos, and diagrams are important in this regard. Also, it's great to see lots and lots of measurements. If a structure is "relatively large," how large is it, and in relation to what?

The discussion and conclusions are usually where I pay very close attention. This is the "meat" of the paper, and the part that will often have the most impact on future researchers. Do the interpretations flow from the results? Are there alternative interpretations? Do the authors lay out any further questions that arise from the data? How do the interpretations of this paper fit into the broader literature? Once again, I want to see why this paper is important (without overselling the research). Many times, the discussion and conclusions "make or break" the paper for me as a reviewer.

Illustrations
Do the illustrations show what the authors want them to show? Are there enough illustrations? Too many? Is color necessary or helpful? When I'm reviewing illustrations, I try to look at the problem from two different angles. As a paleontologist, I will probably rely on the figures as a reference in the future. So, I want to make sure that the figures are as useful as possible! For photographs, is the resolution appropriate? Is there a scale bar? Are contrast and lighting such that the relevant features are clearly visible? Are the views of the specimens sufficient, or are there other views that might be useful additions to the paper?

The second angle is more of an aesthetic one, although this frequently ties into the previous angle. Are the figures attractive? Is there too much white space, or not enough between panels of a multi-part figure? For graphs, are the symbols legible? If color is used in a chart, is the color necessary or could it be turned into a grayscale without loss of information or clarity (this could save a lot of money for the authors and/or the publisher, because color pages ain't cheap!)? If color is used, are red and green featured in a way that will make life difficult for the color-impaired (there are quite a few of color-blind folks out there)?

Size also matters. Is the full-page illustration suggested by the author really necessary, or could it be a single column figure without loss of information? Is that single-column figure just too small to show this critical feature?

Finally, I'll read the caption with the figure. Does the caption make sense? Is everything labeled correctly?

By and large, paleontologists do a good job with figures. We're usually pretty visual folks, and there are some great graphical eyes out there. So, this is often one of the "fun" parts of the review.

Don't Forget the Basics
There are always some basic tasks that I try to undertake in reviews.

If there is a phylogenetic analysis, I'll run the dataset myself and see if I can duplicate the author's (authors') results. Sometimes an old version of the tree will accidentally "piggyback" into the final version of the manuscript or figures, and it's important to be able to catch this. I'll also "spot check" any matrices to make sure that things are coded correctly (within reason, of course).

For statistics, I want to make sure that A) the methods are appropriate to the question; B) the assumptions of the statistical methods are met; and C) the discussion of the results logically follows from the results themselves. As an author, I try (sometimes less successfully than more successfully) to follow these three criteria when designing and writing up my research.

Finally, I'll check the reference list against the text citations. It's amazing how frequently citations slip between the cracks (even in this age of Zotero and Endnote), even in my own writing. This is always the last thing I do with a paper before writing up my review--I read through the manuscript from start to finish, and mark each citation in the bibliography. If anything's missing (or extra), this is noted for the authors. Missing or extra citations are never a big deal in terms of manuscript acceptibility, but catching them is important for ensuring the overall quality of presentation.


The Final Write-Up
The last step for a reviewer is to write up the review in an intelligible manner, both for the editors and the authors. Of course, they'll all be getting the marked-up copy of the manuscript. But, I also think it's important to have a more narrative summary of the review. This is usually a page or so in length, and I'll cover a handful of topics.

First, I'll write a little summary paragraph--stating authors, title, and the main point of the article. I'll try to summarize my opinion on the clarity of the writing and figures, as well as the general quality of the science and novelty of the research. I also like to provide a brief statement on the probably interested audience for the paper--specialists on a particular taxon, workers in a particular subfield, a general audience, etc. Finally, I provide my opinion on the overall publishability of the paper--major or minor revision, or (very, very rarely) unacceptable in current format. These are just opinions, of course--the editors always have the final say.

Next, there's a section on "general comments." If the authors did something really great, I'll put that here. It's depressing to get a review back where the reviewers don't acknowledge the good points of the paper, and as an author I've always appreciated advice on what I'm doing well. The most important information, though, is the broad comments on the paper. Is the overall analysis well-conceived? Etc.

Finally, I write up a section of "specific comments." These are organized by page, paragraph, line number, and address individual sections of the text. Is there a misprint here? Should they have cited someone else there? Is this or that sentence not completely clear?

Anonymity or No?
In many scientific fields and for some journals, anonymity of reviewers is a given. Ostensibly, this is to protect the reviewers from potential future retribution as well as to ensure that previous personality clashes between the reviewers and the authors don't lead the authors to unreasonably reject comments. Some journals, however, allow one to "sign" a review. When given this option, I'll take it. For one, I try to avoid writing anything that I wouldn't be willing to say to the authors in person (although perhaps exceptional circumstances might someday lead me to reconsider). This doesn't mean I won't dispute crummy research--just that I won't hide behind anonymity in order to pursue a personal agenda. Secondly, I think it's useful to the authors to be able to contact the reviewers if there are any specific questions. This doesn't happen often, but I have occasionally had to pursue this route as an author myself. Finally, paleo is a small field, and how easy is it to remain truly anonymous? There might only be three experts on taxon X, and two of them are authors on the paper. It doesn't take a tenured professor to figure out who at least one of the anonymous reviewers might be (especially when the anonymous reviewer recommends the citation of 12 different papers, all by the same author).

So, that's how I usually go about reviewing papers. Everyone has a slightly different method and emphasis, but this seems to work for me. Any thoughts? Comments? For you authors, what is most useful for you when you receive reviews?

In the final post of this series, responding to reviews. . .