Peer review evaluation by colleagues with similar knowledge and experience is an essential component of research and the self-regulation of professions. The average person does not have the knowledge and experience needed to assess the quality and importance of research. Peers do. Therefore many important decisions about research depend on advice from peers, including:

• which projects to fund (grant reviews),
• which research findings to publish (manuscript reviews),
• which scholars to hire and promote (personnel reviews), and
• which research is reliable (literature reviews and expert testimony).

The quality of the decisions made in each case depends heavily on the quality of peer review.

Peer review can make or break professional careers and directly influence public policy. The fate of entire research programs, health initiatives, or environmental and safety regulations can rest on peer assessment of proposed or completed research projects. For peer review to work, it must be:

• timely,
• thorough,
• constructive,
• free from personal bias, and
• respectful of the need for confidentiality.

Researchers who serve as peer reviewers should be mindful of the public as well as the professional consequences of their evaluations and exercise special care when making these evaluations.

Assessing Quality

The effort researchers put into peer review is for the most part not compensated. Researchers may receive reimbursement for travel and per diem when they attend special grant-review sessions and occasionally are paid a basic daily stipend, but this seldom covers the true cost of reviewing a manuscript or a stack of grant applications. As uncompensated effort, the time researchers devote to peer review can easily take second place to other obligations. Running a crucial experiment or submitting a grant application on time understandably is more important than reviewing someone elses work.

However pressed you are for time, if you agree to do a review, you should find the time to meet your obligation in a timely manner. Research is competitive. Researchers are rewarded for discoveries. They should not lose their priority for a discovery due to the tardiness of a reviewer sending comments on a manuscript. Research is also useful. The announcement of discoveries that can benefit the public should not be delayed because someone who agreed to review a manuscript does not have the time to do the review.

Peers who are asked to make judgments about the quality of a proposed or completed project must do their best to determine whether the work they have been asked to review is internally consistent and conforms to the practices of their field of research. This certainly includes:

• assessing whether the research methods are appropriate;
• checking calculations and/or confirming the logic of important arguments;
• making sure the conclusions are supported by the evidence presented; and
• confirming that the relevant literature has been consulted and cited.

At the very least, peer reviewers should be expected to assess whether the manuscript or proposal under review makes sense and conforms to accepted practices, based on the information presented.

Research that conforms to accepted practices can still have problems. Research quality can be compromised by:

• careless mistakes made in reporting data and/or listing citations;
• the deliberate fabrication and falsification of data;
• improper use of material by others (plagiarism);
• inaccurate reporting of conflicts of interest, contributors/authors; and
• the failure to mention important prior work, either by others or by the researcher submitting a paper for publication.

However, how much peer reviewers can or should do to detect these and other deceptive or sloppy practices remains subject to debate.

Judging Importance

In addition to quality, peer reviewers are also asked to make judgments about the importance of proposed or published research. They are asked to answer questions such as:

• Assuming a researcher could carry out a proposed research project, is it important to do so?
• Are these research results important enough to publish?
• Has a researcher made important contributions to a field of study?
• Is this evidence important enough to be used in setting policy?

Along with quality, judgments about importance essentially determine which research is funded or published and which researchers are hired and relied upon for advice.

Peer reviewers do not always make judgments about importance with an open mind. Studies have shown that they can be swayed by:

• the stature of the researcher who conducted the research or the institution at which the research was conducted;
• country of origin;
• a preference for one research method over another, e.g., a clinical versus a laboratory approach; and
• the outcome of the studies under review.

One way to lessen the impact of bias is to write transparent reviews. By transparent is meant laying out clearly for anyone reading the review how it was prepared, the literature that was used, and the reviewers own possible biases. If reviewers fully and carefully explain how their judgments about importance were made, others can assess whether they want to accept those judgments.

Preserving Confidentiality

Peer reviewers have an obligation to preserve confidentiality during the review process if they have been asked to do so. While this obligation might seem obvious, it can be compromised in some seemingly harmless and other more harmful ways. For example, although researchers sometimes do, it is not acceptable to do any of the following without getting permission:

• ask students or anyone else to conduct a review you were asked to do;
• use an idea or information contained in a grant proposal or unpublished manuscript before it becomes publicly available;
• discuss grant proposals or manuscripts you are reviewing with colleagues in your department or at a professional meeting;
• retain a copy of the reviewed material (generally manuscripts and grant proposals should be shredded or returned after the review is complete); and
• discuss personnel and hiring decisions with colleagues who are not part of the review process.

There may be times when some added advice during a review may be helpful, but reviewers should not seek this advice without getting permission. It may also be tempting to use information in a grant application or manuscript to speed up your own research, but until it has been made public, confidential information is not available for use, even to reviewers. If you are not comfortable protecting confidential information, then do not agree to be a peer reviewer.

Resources

The Graduate School wishes to thank the Office of Research Integrity for the above materials found in the textbook, ORI Introduction to the Responsible Conduct of Research, Nicholas H. Steneck, PhD; Illustrations by David Zinn