Trainees in Peer Review: Our Experience

The American Journal of Neuroradiology (AJNR) receives more than 1400 article submissions every year. When more than 2 years ago, we eliminated case reports, which accounted for between 25% and 35% of all submitted articles, we did not anticipate that this difference would be made up, almost instantly, by an increasing number of original, full-length investigations (we are very happy this happened). Today, nearly 1250 of the total submissions (89%) are full-length, original articles, and the rest are reviews, editorials, letters to the editor, and other types of articles. These have obviously resulted in increasing work for our Senior Editors and reviewers. Our reviewer data base contains the names of more than 2400 clinicians and scientists from all over the world, but in reality, only a relatively small number of them do most of our reviews. Why? Because they never say no, they are excellent reviewers, and neuroradiology has become so subspecialized that it is difficult to find reviewers for a growing number of sophisticated topics such as computer modeling of intra-aneurysm fluid-flow dynamics, kurtosis, and so forth. In this “Perspectives,” my aim is to inform our readers about how we are trying to improve and expand our reviewer data base, especially as it pertains to using neuroradiology trainees in the peer-review process. This information was partially presented at the Radiology Editors Forum in 2012 (this group of editors from imaging-related journals meets once per year).

One can easily improve and increase the number of reviewers by recruiting fellows and residents, using the names of individuals found in the references of submitted articles, accepting personal references from other reviewers, asking the authors of previously submitted articles to contribute, and accepting those individuals who offer unsolicited help (AJNR uses all of these strategies). I have tried to recruit my own fellows who have left for private practice after their training but have found this disappointing because they soon become very busy and reviewing articles turns into a low priority. Once every 2 years, we look at Editorial Board performance in the reviewer data base and purge those individuals who have very low scores and often decline to review articles (you can ask for your scores, and we will be glad to send them). Using only our best reviewers is often problematic because it leads to fast burnout rates. Remember that each review generally takes anywhere from 3 to 8 hours to complete. Because our reviewers do not work exclusively for us, demanding more work from them only leads to their declining requests from other journals that may also need their expertise (there are currently 116 imaging-related journals).

The ever-increasing demands placed on these exceptional individuals requires constant positive feedback and encouragement from editors in the form of awards and “best of” lists published in journals and on Web sites, reviewer scoring feedback, and personal encouragement in the form of praise and letters (we at AJNR do all of these). Unfortunately the intellectual rewards of reviewing may soon lose their initial importance, and we must try other forms of encouragement: payments, discounted subscriptions, discounted fees to annual meetings, discounted or free continuing medical education (CME) activities, and CME credit for reviewing (currently I am aware of 5 imaging journals that use this latter strategy; AJNR will start doing this sometime in the second semester of 2013).¹ Unfortunately, none of these strategies work well in the long run.

The reality is that all journals are facing a reviewer shortage. The total number of scientific articles submitted increases at a rate of 3.3% per year and doubles every 20 years.² More than 1.3 million articles undergo peer review every year. These may even be underestimated if one takes into account the growing number of “international” journals published in China, India, and Eastern Europe. Because less than 45% of all published articles will ever be cited, it seems that most of the reviewing and editing process is being spent on lesser quality science that will never be recognized as important. Reviewers not only contribute to journals but are asked to do similar jobs when they serve as experts on panels of foundations and government agencies. Junior and midlevel individuals end up with no time for activities needed for promotions, and peer review of articles is not one of these. We have found that as the number of review requests has increased, the percentage of individuals who do not respond or decline these requests has remained at about 50% for the past 3 years (Fig 1). Thus, although we have a larger number of reviewers, our success rate at getting them to accept review requests remains static.

Fig 1.

Response to request to review AJNR manuscripts, January 2009–June 2012.

Because of all of the aforementioned situations, it seems natural to tap into our trainees and younger members of our profession to review articles. In 2004, the United Nations Educational, Scientific and Cultural Organization created the International Forum of Young Scientists, and peer review has been addressed repeatedly there.³ The World Academy of Young Scientists agrees that junior members of different specialties should be involved in peer review of the scientific literature and considers it a critical activity in their education process, but medical schools and radiology departments do not give credit for it (at least in the United States).

At our annual meeting of the American Society of Neuroradiology (ASNR), during the luncheon for the Young Professionals Network, I usually ask these individuals for help with the peer-review process. These young volunteers are generally asked to serve as third (or fourth) reviewers; that is, they never review unless an article is being evaluated by 2 or 3 other senior reviewers. During 2010 and 2011, we were able to recruit 36 such collaborators (in 2012 their number was up to 41, but I will not include them here because we still do not have all of the information pertaining to their activities). All except 1 of these young reviewers lived in the United States. Each individual was assigned to 1 Editor, and I had 29 (83%) of them. Two Senior Editors who deal with brain and head and neck issues were responsible for the rest. No trainees volunteered to review articles related to neurointerventional, spine, or advanced imaging techniques. Six of the 36 trainees declined all invitations to review articles. The remaining 30 did 261 total reviews (221 original research, 22 clinical reports, 14 case reports, 3 technical notes, and 1 review article), accounting for 9.3% of all submissions (n = 2826) during the same 2-year period. The time it took for these reviews to be completed was 5–17 days, below our allowed maximum of 21 days and perhaps less than it takes many of our more senior reviewers to complete their tasks. All reviews received by AJNR are scored by using a simple and subjective scale by the editor in charge of the article (1 = review was below average, 2 = review was sufficient, and 3 = review was highly relevant). The average score of reviews done by trainees was 2.3; but as is seen with those assigned to senior reviewers, scores varied significantly from individual to individual.

As with senior reviewers, we do not expect structured evaluations from trainees. It is the philosophy of the editors of AJNR that free-form review allows individuals to better express their opinions and thoughts. However, we do not deny that structured reviews may be useful to more neophyte reviewers. For guidance, we generally refer reviewers to the well-known articles on this topic by Proto⁴ and Provenzale and Stanley,⁵ which are posted on our peer-review Web site. We believe that adequate and continuous feedback leads to the development of well-defined and useful individual styles and results in reviews of high quality; however, this is controversial. In 1 study, neither reviewers with low nor high scores improved their evaluations when feedback was given to them, leading to the conclusion that feedback in this situation is ineffective.⁶

For purposes of assessing the quality of trainee reviewers with respect to more senior ones, it is helpful to compare their decisions. Trainee reviewers gave a “hard” rejection to 114 manuscripts, while senior reviewers gave these same articles the same “hard” rejection in 69 instances and a “reject/resubmit” disposition in 45 cases. Trainees gave 156 articles a decision of “revision needed” (either major or minor revision), and these matched the dispositions given by senior reviewers. One article was accepted without revision per both trainee and senior reviewers. Therefore, there were no significantly discordant decisions between the 2 groups of reviewers, and mild discordances in dispositions were seen in 17% of reviews. I was surprised by this because published studies seem to indicate that there is no reproducibility of peer review in the neurosciences.⁷

After becoming reviewers, 25/36 trainees submitted articles to AJNR either as a principal or coauthor (82 total articles including 59 original research articles, 4 clinical reports, and 4 review articles). Of these, 39 were ultimately accepted, and 43, rejected (which is better than our usual 75% rejection rate). It is thus possible that reviewing made these young individuals better authors. One excellent trainee reviewer was asked to join our Editorial Board at the end of her fellowship. After 2 years, all of our trainee reviewers migrated to the pool of senior reviewers.

How can we improve trainee participation in our review process? Certainly, structured reports are a consideration. Structured review forms tell less-experienced reviewers not only what to look for but also (more important) what to ignore. Some journals that use this type of process have considered eliminating “false cues of quality” from their forms. “Significance tests ” are one such cue because they are generally interpreted as reflecting “quality” in research when this is not always the case.⁸ Currently, the American Journal of Roentgenology and Radiographics are in the process of studying this issue and crafting structured guidelines for their reviewers. Greater communication with neuroradiology program directors could lead to further reviewer recruitment as well as granting young reviewers the time needed for the activity, credit for their work, and encouragement.

In addition, increasing the pool of international trainee reviewers would be desirable. From my experience, it seems that article reviewing is still an important and honorable activity in other countries, especially reviewing for an American journal. Asking members-in-training about their interest in reviewing articles at the time that they are filling out their society applications would simplify the process of identifying interested individuals (ASNR started doing this in 2012). One difficulty that we editors face when assigning articles to trainee reviewers is finding which ones are appropriate for them. In the past when we accepted case reports, these provided us with simple articles that served as a starting point for junior individuals. As the complexity of articles increases, assigning them is more difficult (but maybe I am underestimating the capacity of our trainee reviewers).

Special training programs for individuals may produce fewer, but highly qualified, reviewers and future editors. Currently, there are 3 societies offering such programs. The oldest is the Figley Fellowship from the American Roentgen Ray Society, which is now geared to young individuals practicing in the United States; and the newer Rogers International Editorial Fellowship, which, as its name implies, is available to those residing outside the United States. Fifty-seven individuals (including myself) have participated in one of these fellowships, and 7 have become journal editors (S. Cappitelli, personal communication; December 2012). The Radiological Society of North America (RSNA) and Radiology have offered their Olmsted Fellowship to 7 individuals, but none serve on its Editorial Board (R. Arnold, personal communication; December 2012). The Eyler Fellowship from the RSNA is designed for midcareer individuals, and I have not included it in this discussion. In the Spanish-speaking world, the Spanish Society of Medical Radiology and its journal, Radiologia, are the only ones to offer an editorial fellowship. ASNR and AJNR have recently started a similar endeavor, and information about this can be found on our blog (www.ajnrblog.org) or Web site.

In conclusion, I think that we have had some degree of early, but encouraging, success by using trainees in the peer-review process of AJNR. Their participation has been limited, but the results of their timely reviews do not differ significantly from those of our more senior reviewers. We hope to start implementing new measures that will increase the number of trainee reviewers, standardize their contributions, and recognize their effort.

References

1.
1. Lee K,
2. Bero L
. What authors, editors and reviewers should do to improve peer review. Nature (2006), http://www.nature.com/nature/peerreview/debate/nature05007.html. Accessed May 13, 2013
2.
1. Bauerlein M,
2. Gad-el-Hak M,
3. Grody H,
4. et al
. We must stop the avalanche of low-quality research. The Chronicle of Higher Education, June 13, 2010, http://chronicle.com/article/We-Must-Stop-the-Avalanche-of/65890. Accessed May 13, 2013
3.
1. Mainguy G,
2. Motamedi MR,
3. Mietchen D
. Peer review: the newcomers' perspective. PLoS Biol 2005;3:e326
4.
1. Proto AV
. Radiology 2007: reviewing for Radiology. Radiology 2007;244:7–11
5.
1. Provenzale JM,
2. Stanley RJ
. A systematic guide to reviewing a manuscript. J Nuclear Med Technol 2006;34:92–99
6.
1. Callaham ML,
2. Knopp RK,
3. Gallagher EJ
. Effect of written feedback by editors on quality of reviews: two randomized trials. JAMA 2002;287:2781–83
7.
1. Rothwell PM,
2. Martyn CN
. Reproducibility of peer review in clinical neuroscience: is agreement between reviewers any greater than would be expected by chance alone? Brain 2000;123 (pt 9):1964–69
8.
1. Armstrong JS
. Peer review for journals: evidence on quality control, fairness, and innovation. Sci Eng Ethics 1997;3:63–84

Main menu

User menu

Search

American Journal of Neuroradiology

References

News and Updates

Resources

Opportunities

American Society of Neuroradiology