ENGAGING IN QUALITY TECHNICAL PEER REVIEW AS AN INTERNATIONAL PROFESSIONAL RESPONSIBILITY: THOSE WHO PUBLISH CONFIDENTLY MUST ALSO REVIEW COMPETENTLY^*

David W. Grainger

Departments of Pharmaceutics and Pharmaceutical Chemistry, and Bioengineering University of Utah, Salt Lake City, UT 84112-5820, USA. Correspondence: Department of Pharmaceutics and Pharmaceutical Chemistry, College of Pharmacy, Room 301, 30 South 2000 East, University of Utah, Salt Lake City, UT 84112-5820 USA, Tel: +1 801 581 3715, Fax: +1 801 581 3674, david.grainger@utah.edu.

ABSTRACT

Quality peer-review remains central to current international scientific and technical publishing and proposal assessment methods. As incompetent review and perceived bias remain the most cited problems with peer review processes commonly employed in scientific review of manuscript and proposals, the creation and maintenance of quality pools of engaged, responsive and qualified peer reviewers is essential to scientific publishing and dissemination. An important operational principle for the peer reviewing system is that all who utilize this publishing system should then also review a commensurate load on behalf of the system. This would also imply that those who compose and submit technical manuscripts are competent to assess and levy fair criticism of other's work in their field. Given the large and rapid expansion in numbers of submitted manuscripts from non-traditional sources, including many developing countries, expansion of the peer-reviewing pool to these sources is necessary both to accommodate their respective, newly imposed reviewing burdens on the already over-burdened system, and to engage new communities in the traditional process of vetting and validating scientific and technical works. Effective peer review must enforce the many elements of reviewer technical proficiency, professional conduct, bias and ethics considerations, and responsibility in this process and the competitive international system in which it sits. Reviewers require training, oversight, control, expectations, and continual guidance. Validation of peerreview's overall efficacy requires follow-on policing of published literature to assert its accuracy and content through consensus and experimental reproduction. As former developing countries now contribute increasing numbers of new manuscripts to the technical peer-review system, they should also actively seek to officially train such contributors to also be visible, effective peer-reviewers for international journals, editors and funding agencies. This is not a passive endeavor, requiring expectations, recruitment and training, and the associated resources to make accommodations as rapidly as their contributions are encumbered within the current publishing systems. Collective responsibilities as researchers, contributors, reviewers, readers and enforcers of the integrity and safekeeping of this essential quality control process traditionally rely on individual professional integrity and conscientious effort. Extension of this effort to continually recruit new pools of competent, trained and qualified reviewers in the current publishing era is essential.

KEY WORDS: Peer review, Developing country, Technical publishing, Professional conduct, Quality control, Responsibility, Nontraditional contributors.

La calidad del proceso de evaluación por pares académicos es fundamental en los métodos actuales de publicación científica y técnica, así como en la evaluación de propuestas de investigación. La incompetencia y falta de imparcialidad en la evaluación continúan siendo los problemas más citados sobre el proceso de evaluación por pares académicos. Debido a esto, crear y mantener un grupo de evaluadores comprometidos, responsables y calificados es fundamental para la publicación y diseminación científica. Un principio importante en la mecánica del sistema de evaluación por pares consiste en que aquellos que utilizan el sistema de publicación deberían luego revisar una carga equivalente a su rol como integrantes del mismo. Esto también implicaría que quienes escriben y envían artículos técnicos sean competentes para evaluar y criticar con justicia el trabajo de otros en sus áreas de estudio. Debido al rápido incremento en el número de artículos sometidos por parte de fuentes no tradicionales, incluyendo muchos países en vía de desarrollo, es necesario expandir el grupo de pares académicos al incluir miembros de estas comunidades de modo que sea posible dar respuesta a esta carga adicional impuesta a un sistema ya saturado; asimismo, comprometer nuevas comunidades en el tradicional proceso de evaluación y validación de los trabajos científicos y técnicos. Una evaluación efectiva por pares debe velar por varios elementos que incluyen la habilidad técnica del revisor, la conducta profesional, la imparcialidad, la ética y la responsabilidad por este proceso y por el sistema competitivo en el que éste se desarrolla a nivel internacional. Los pares evaluadores necesitan entrenamiento, supervisión, control, expectativas y guía continua. La validación de la efectividad general del proceso de revisión por pares requiere controles de seguimiento de la literatura publicada para confirmar su precisión y contenido a través de consenso y reproducción experimental. Como, en la actualidad, gran parte de los países en vías de desarrollo contribuyen al sistema de evaluación con un número significativo de artículos, estos países deben buscar activamente entrenar a sus contribuyentes, para que sean pares evaluadores efectivos y reconocidos por revistas internacionales, editores e instituciones financiadoras. Ésta no es una tarea pasiva, ya que requiere definir expectativas, políticas de reclutamiento, entrenamiento y demás elementos asociados, con miras a realizar los ajustes respectivos tan pronto como sus contribuciones sobrecarguen los sistemas de publicación actuales. La responsabilidad colectiva como investigadores, contribuyentes, evaluadores, lectores, y aseguradores de la integridad y protección de este esencial proceso de control de calidad tradicionalmente ha dependido de la integridad y consciencia profesional. La extensión de este esfuerzo por reclutar nuevos grupos de evaluadores competentes, entrenados y calificados, es esencial en la era actual de publicación científica.

PALABRAS CLAVE: Revisión por pares, Países en desarrollo, Publicaciones técnicas, Conducta profesional, Control de calidad, Responsabilidad, Contribuyentes no tradicionales.

(Adapted in part from: Grainger D.W. Peer review as professional responsibility: a quality control system only as good as the participants. Biomaterials, 28:5199-5203, 2007)

I. INTRODUCTION

Effective technical communication and dissemination are essential tasks of any scientist, engineer or medical practitioner. Manuscripts and research proposals comprise the primary body of such communication. Scientific and technical publishing thrives on this essential publishing requirement, much as the body of over 1000 full journal publications emerging daily. Because scientific information that is (1) not published is effectively unknown, and (2) not validated is generally mistrusted, peer review is very important to all who publish scientific information as a common vetting mechanism for validation and acceptance. Therefore, a quality control process is enacted on peer-reviewed scientific transactions, an essential collective peer review professional responsibility [1-3]. Through review or 'refereeing', scholarly products and proposals are subjected to the anonymous or double-blinded scrutiny and critique of peer experts in the field. Journal editors rely on the process to facilitate selection, ranking and prioritization of newly submitted manuscripts for publication; funding agencies use it to prioritize grants for research support. A simple schematic of the dynamic peer reviewing process is depicted in Figure 1. Standards of the discipline, and of science in general, are enforced on the contributors. Despite some emerging evidence for disrespecting peer review among the youngest generation of scientific authors, over 85% of authors believe that peer reviewing improves their submissions [4,5].

Peer reviewing duties have particular significance now for rapidly growing electronic publishing pools, and the increased accessibility that this new mechanism (and associated "open access" venues) provides to the international community for authoring, dissemination and new visibility. This is particularly important for developing countries that traditionally cannot afford many international journals subscriptions. Scientific activities in developing countries now encompass nearly 25% of the world's scientists and engineers but using less than 6% of the global research budget [6]. Research reporting from these countries is increasingly seeking to enter mainstream international publishing venues, but with notable challenges. First, many of the authors of papers from second- and third-world research labs face a choice to publish in their native language but in low visibility, low circulation journals that cater to this priority, or submit to international venues in a language (generally english) that represents their second or even third, non-native language. the number of prestigious journals that represent mainstream science is relatively small [7] compared to all journals, and all of these require written english language proficiency. Journals catering to the scientific periphery either in theme, language, geographical region, or culture, face numerous challenges [6]. international visibility and impact are limited: less than 2% of journals sourced in developing countries are included in major international publishing databases including Web of science, science Citation Index, MEDLINE, Current Contents, SciFinder Index, and PubMed [6]. Additional struggles with these journals' emergence as visible mainstream technical venues include limited submissions, sub-standard manuscript quality, language and communication issues, and poor-quality review processes. this 'vicious cycle' in developing fringe journals for international recognition and readership has been documented [8-10].

Importantly, the proportion of technical manuscripts submitted from researchers in recently developed or developing countries to international journals is growing at a rate larger than that from traditionally developed countries. taiwan, Brazil, turkey, south Korea, india, Mexico and China are notable in this regard. Taiwan, china, india, turkey and south Korea, in particular, increasingly contribute to the international manuscript editorial burden [4]. central and south america doubled their relative fractional contributions to the total international journal pool between 1999 and 2003 [11]. taiwan, Brazil and india have doubled, while turkey and south Korea have tripled, respectively, their annual manuscript contributions to the international peerreviewing pool in the past decade (1999-2009). Based on their overall numbers of scientists and engineers, india and china are now contributing enormous numbers of new manuscripts annually that were simply not present a decade ago [12-14].

A previous opinion from this author outlines the general burden for the peer-reviewing system and recommends a code of conduct and responsibility for all publishing participants to optimize function and quality control within the technical reviewing system [15]. One could argue that there have never been sufficient qualified peer-reviewers; journal editors constantly struggle to find willing, competent placement of manuscripts for proper vetting in their respective communities. Therefore, training and enabling pools of new editors and reviewers, including the critical new fractions of contributors from non-traditional sources of new manuscripts, will remain a challenge. Nonetheless, national professional societies, scientific organizations, governmental agencies, and university training programs should actively engage and develop a consistent message regarding the importance of their quality peer-reviewing responsibilities. This would best include formal new referee training programs, recognition of service and dedication to this task, reward systems for those who participate, and some cognizance of the significance of this duty for personal and national recognition within the international science community. Benefits must not only be explicit: it is no secret that regular peer-review provides the reviewer advanced insight into the latest breaking contributions to the field, that outstanding peer review service can lead to editorial board promotion, and that editorial visibility promotes both the individual, and brings international recognition to their nationality and institution.

Both skepticism or accolades aside, this peer-review quality control process, for better or for worse, is the operative status quo within which practicing scientists operate, and, importantly, upon which the technical community relies for continuous dissemination of high quality and reliable information essential to move fields forward. All scientists and engineers as contributors, academicians, pedagogues, technologists, practitioners, or benefactors, have specific obligations to the peer-reviewing system to make it work. Reasonable professional rules of conduct are occasionally explicitly described with recommendations to recruit or enlist reviewers to ensure quality journal reporting [16-18]. These duties extend equally to three publishing constituencies for coordinated review management: the manuscript contributors, the journal editor, and the scientific readership. Additionally, the International Congress on Biomedical Peer Review and Scientific Publication has held on-going discussions on merits and problems of the scientific peer review system, published in Journal of the American Medical Association in various forms for over a decade [19]. Collective technical community insistence on data reliability, reproducibility, accuracy and communication clarity is critical to publishing integrity. Good peer-review must effectively serve technical and scientific dissemination to ensure accurate informational access, prevent propagation of low-quality scientific literature as a first-pass measure, and eliminate technical "noise" from polluting databases and literature pools. Rapid and reliable identification (sieving) of the most important data and relevant information in each professional's respective fields relies on the presumption of credible scientific quality as a discerning criterion. Because one should not blindly accept everything in print as ultimate truth and accuracy, time spent searching the vast literature bases to selectively locate work of the highest relevance and quality to specific interests relies on continual, cooperative vigilance to the peer-review process that produces it.

Journal publication quality is the collective responsibility of both those who read as users and those who write as contributors -- the same groups from whom peer-review experts are drawn and who then enact the standards for scientific quality and acceptance. Citation, journal impact factor, value and appeal to the field and assessed technical quality are direct functions of published content, topical relevance, readership, exposure, circulation and the resulting influence on visibility and subscriptions. Sadly, responsibility for technical journal quality is often presumed to reside primarily at the editorial level, where the ultimate decision to publish or reject content emanates [2,16]. This dangerous presumption ignores (1) the relatively limited expertise of most editors, (2) resulting inability to adequately judge quality and excellence without quality input from skilled reviewers, and (3) the bias inherent within any system that relies on limited pooling of expertise to make decisions. With on-going expansions in topical breadth, interdisciplinary research and increasing technical methods' sophistication and information content, no editor should be held hostage by their limited knowledge and relative ignorance of a single mind in this complex scientific system. Therefore, the essential importance of collective assessment of the primary body of technical literature using credible peer-review should be readily evident. A critical determinant of any successful journal or technical publication in general is a reliable capability to readily access a talented, adept, accomplished and reliable reviewer pool. Training and recruiting such a pool remains a constant challenge.

Significantly, there are few formal processes beyond the classical graduate-level 'journal club' offerings at most institutions that didactically address the attributes of the effective reviewer/editor/contributor relationship, or that rigorously train doctoral, professional and post-doctoral scientists in methods, expectations, and mechanics of peerreview. More often, the process is simply a professional on-the-job "rite of passage" where many are called upon by editors and initially must act instinctively, often without much experience or formal training, to produce a technical review of manuscripts and proposals. Quality peer-review of a given technical communication or research grant is not simple, easy or quickly performed, but elements can be taught and learned, and published guides exist [16-18,20-22]. Reviewer training courses are labor-intensive and tedious [9], attempting to teach technical writing, research planning, data analysis and interpretation, and elements of technical critique strategy and their communication as peer-review components. These courses have only small impact that appears short-lived [23]. Cursory or poor quality reviews are a tremendous disservice to the community, with profound consequences to science beyond the article in question [24]. Those who read or submit work to journals from the global body of scientists and engineers in universities, government labs, research foundations, or industry must continually re-evaluate their sense of commitment to professional technical reviewing obligations that directly affect journal and technical communication quality.

Professional duty obligates all who are researchactive, who read the scientific literature, or who submit manuscripts or proposals for peer-review, to fulfill, both responsibly and expediently, their share of fair, prompt review of this literature. The value of pooled individual reviewing contributions might be perceived analogously to the value of the individual vote in any election: rights of influence and choice are asserted through individuals in the process, and the collective of referee reports provides the fate and direction of the resulting scientific literature and funded research portfolio. If the peer-review system were perfect, the discussion of problems, alternatives and improvements would not be as active as it is today [2,25-29]. Nonetheless, despite flaws, defects and identified weaknesses, it remains the best system thus far conceived and implemented at global scale.

As an elective system, personal contributions facilitating expediency, credibility, and equity to the scientific review process are a matter of personal choice and management. In assessing these duties, one should seriously consider these guidelines proposed recently [15,17,30]:

1. Every manuscript submission requires the volunteer efforts of at least three 'peer experts' for the referee vetting and eventual publication process: the journalassigned handling editor and at least two anonymous reviewers. Hence, as a quid pro quo, for each manuscript submitted, the author/contributor should review three other manuscripts in return in order to compensate for the burden that the author places on the publishing system. This needs to be considered for the current flood of manuscripts from non-traditional and developing countries' authors.
   Claims that "excessively productive" authors who, due to their prominence, prolific writings and perceived contributions, are exempt from reviewing any others, or are relieved in some elite way from obligations or duty to peer-review represent baseless arrogance. How any contributor of manuscripts or proposals remains beyond the responsibility of contributing to the peer review process is mystifying. Contributions to the literature should be commensurate with peer-reviews of submitted literature based on this "one entitles three" principle [31].
2. Research proposal submission encumbers the same scale of in-kind review duty (and associated issues) as for manuscripts [32]. In many instances, however, imposed reviewing duty for proposals should be even more compelling since panel reviews and study sections involve more reviewer time and numbers of reviewers than single mail-out manuscript reviews. When a proposal review statement is returned to the author/proposer, the number of reviewers involved in the review often can be discerned from the information provided. Hence, the encumbered reviewing burden is also then known and can be expected in return by the author, regardless of the funding outcome, as a necessary professional compensation to the reviewing system. ]]> Many established scientists are now asked to review or shepherd new research proposal-reviewing programs, or young investigator programs being initiated in developing countries, in order to mirror established analogous processes in developed nations. This compels even further obligations on developing nations to restore balance to the international system in their compensating reviewing duties.
3. Given a review request, prompt communication of both (1) the intent to produce a quality review as well as (2) the completed review itself to the editor's office is important. Punctual, reliable communication and reviewing are not only courteous, but relieve the editor's office of significant extra work tracking all pending reviews and reminding those remiss in their submissions. A brief email acknowledgement of receipt, acceptance, and expected review return date, with the manuscript's identification number in the 'Subject' heading assures the editor's assistant that reviewers are on-track with the assigned review. Then, reviewers should make every attempt to get the review back by the assigned deadline. When this is impossible, they should then communicate an expected submission timeline to the editor's office once again with the manuscript number in the email subject line.
   Time-to-publication has become an important measuring stick by which the selection and quality of a journal for one's publication submission is, among other factors, often based. Beyond contributor perceptions, it also affects impact factors and other quality assessments for journals. These "rules of engagement" produce an efficient, effective system, where time-to-publication can be reduced by streamlining the review process through quality performance and responsible communication with the editors.
4. When a specific reviewing request cannot be accommodated due to conflict, other commitments or perhaps poor alignment of expertise, carefully considered recommendations of other qualified reviewers to the editor or program officer are very useful. Listing of the names and full email contact information for three alternative reviewers (and their URLs for their websites) can save an editor substantial time and effort, and readily target review to a qualified, select pool.
5. Actively engage, expand and train new capable, responsible reviewers by formal training of graduate students and colleagues. This should include the expectations, standards, protocols and rules of fair review, and the adverse effects of bias, unethical conduct or poor quality review. Journal clubs and graduate seminars often focus on scientific review and critique of the literature. However, few of these experiences actually practice or formally train students or colleagues to master the mechanics of drafting a realistic, credible journal-type or NIH proposal-type critique or review. Those who submit proposals and papers generally get these reviews back in writing. Review qualities and content that distinguish a good, competent review from a bad one are usually instantly apparent to most authors and form excellent examples for teaching this important contrast. Nevertheless, many of us are not actually taught the elements of style, content, or technical significance that comprise formulation of a good review, nor the elements of what constitutes a fair, objective, unbiased scientific critique.
   The 'rite of passage' in developing 'good reviewers' should not be left to chance or personal self-taught, trial-and-error, or anecdotal experiences. Professional societies and academic programs can facilitate professional training in this area, as well as instill the sense of both duty and necessity, in order to ensure an adequate supply of qualified, capable and reliable scientific reviewers receptive to this need. In principle, the reviewing pool should be as large as the author pool. Yet this is clearly not the case. The current challenge of handling new sources (i.e., developing countries) of non-traditional manuscript burdens is an important case-in-point.
6. Both public and private sector scientists, engineers and medical researchers are all obligated to review. All benefit from reading and use of the scientific literature. However, industrial scientists are often over-looked in the peer-review process as they are presumed to not have a sufficiently vested interest in the outcome, or exhibit a conflict of interest in reviewing confidential information for others, or do not contribute a significant fraction of manuscripts to the literature to understand the reviewing culture. The prudent editor or program manager should be able to utilize both public and private sector scientists alike, and teach them appropriate roles and behaviors. In principle, confidentiality supposedly extends throughout the reviewing process. A skillful editor will be mindful of competitors and attempt to minimize conflict or possible breaches of confidentiality through judicious choices of reviewers. Therefore, a broad pool of scientists and engineers from all walks of professional life should avail themselves to reviewing duties.
   Professional reviewing responsibilities can also be abused for selfish or unethical reasons that produce other difficulties that compromise the integrity of the system. Reviewer anonymity imparted by the current "partially closed" (i.e., author known/reviewer anonymous) system has its issues in this regard [33-35]. Double blind review where neither author nor reviewer identities are revealed is often recognized as the best review system [36,37]. A recent study [38] also indicated diminished gender bias against authors with female first names. Nonetheless, earlier studies indicate that the quality of such review is not perceptibly improved, despite mutual anonymity [39,40]. Additionally, real anonymity is difficult to preserve in the face of self-citation, context and topic, and writing style.

Common reviewer improprieties and misconduct listed by the Center for Science Publishing white paper [17] include:

]]> Deliberate misrepresentation of facts in a review
• Delaying the review process unreasonably for personal strategic gain, or exploiting confidential information to achieve personal or professional gain
• Unfairly criticizing a competitor's work
• Breaching the confidentiality of the review
• Proposing changes that appear to support the reviewer's own work or hypotheses
• Appropriating ideas or text from a manuscript under review
• Including personal or ad hominem criticism of the author(s)
• Failing to disclose a conflict of interest that would have excluded the reviewer from the process.

I respectfully urge each of us to continually (1) assess our own professional reviewing records, and (2) make the necessary service adjustments to accommodate the burden that our own respective proposal or publication productivity places on the peer-reviewing system. When hearing the boisterous claim in a plenary introduction at a meeting that Professor X has over 400 research papers published, my own skepticism tells me that Professor X likely has not provided quality reviews for the 1200 (i.e., 1:3) other research papers necessary to compensate his/ her own imposed peer-reviewing burden! Perhaps this lack of accountability is partly due to inflated recognition and praise earned from technical communities for such outstanding scientific productivity and dissemination, and equal lack of any recognition for commensurate amounts of reviewing service required to review, certify, produce and endorse this productivity in publication form. For the enormous amounts of manuscripts now flowing from new non-traditional sources, this self-assessment is important to determine reviewing responsibilities and maintain the functionality in the existing system.

One alternative currently advocated is to use on-line publications with wider, direct world-wide accessibility to increase technical literature volume and exposure. Many innovative, alternative forms of electronic publishing and "open" alternative forms of peer reviewing are now possible with world-wide, instant on-line access. Some of these are currently in trials or active discussion [41,42]. However, it does not require much thought to ponder problems of poor quality peer-review (or complete lack of any credible review or editorial processes as is occurring in some on-line venues) in wasting significant time and effort in searching, reading and pursuing research of uncertain quality or that lacks any publication standards or validation.

Peer-reviewing is indeed extra work: the average technical manuscript review requires 8-9 hours [43]. But this is extra work that follows necessarily in order to produce and guarantee a useful, reliable technical literature repository and highly valuable resource for all involved in publishing and dissemination. Clearly, the best, most efficient method to ensure science and engineering publishing credibility and technical progress is by advocating consistent peer-reviewing standards across all aspects of the reporting procedures. The technical community should continue to openly endorse and cultivate collective international professional responsibility to actively participate in this system to make it work effectively.

ACKNOWLEDGMENT

Professor David Williams (U.K.) is acknowledged as a mentor in my understanding of peer-review importance and scientific validation.

_____________________________
NOTAS
* This paper was translated from English to Spanish by Felipe García Quiroz, Duke University, North Carolina, USA <felipe.garcia@duke.edu> and the translated version was published in print in Revista Ingeniería Biomedica, 5: 66-74, 2009. The original version of the paper, as presented here, can only be accessed online at http://revistabme.eia.edu.co/.

REFERENCES

[1] Hames I. Peer Review and Manuscript Management in Scientific Journals: Guidelines for Good Practice. Blackwell Press, 2007, Malden, MA, USA.         [ Links ]

[2] Kassirer J.P., Campion E.W. Peer review: crude and understudied, but indispensable. Journal of the Medical American Association, 272:96-97, 1994.         [ Links ]

[3] Davidoff F. Improving peer review: who's responsible? British Medical Journal, 328 657-658, 2004.         [ Links ]

[4] Clark D. Journal Publishing in Astronomy. In: Heck A., Houziaux L. Future Professional Communication in Astronomy. Proceedings of the Colloquium held at the Palace of the Academies, Brussels, 19-13 June, 2007, Bruxelles: Académie Royale de Belgique, 2007.         [ Links ]

[5] Ware M., Monkman M. Peer Review in Scholarly Journals - Perspective of the scholarly community: an international study. Publishing Research Council, 2008.         [ Links ]

[6] Gibbs W.W., Lost science in the third world. Scientific American, 273:76-83, 1995.         [ Links ]

[7] Marusic A., Marusic M. Small scientific journals from small countries: breaking from a vicious circle of inadequacy. Croatian Medical Journal, 40:508-514, 1999.         [ Links ]

[8] Garfield E. The significant scientific literature appears in a small core of journals. The Scientist, 10:13-16, 1996.         [ Links ]

[9] Marusic A., Marusic M. How to help a small journal become a part of the mainstream literature. Science Editor, 23:81-83, 2000.         [ Links ]

[10] Marusic M., Misak A., Kljakovic-Gaspic M., Fister K., Hren D., Marusic A. Producing a scientific journal in a small scientific community: an author-helpful policy. International Microbiology, 7:143-147, 2004.         [ Links ]

[11] Science and Engineering Indicators, 2006, from Thompson ISI, Social Science Index and Science Index.         [ Links ]

[12] Research and Development Statistics 2004-05; UIS, UNESCO 2005.         [ Links ]

[13] Government of India, Ministry of Science and Technology. Last consulted May 16, 2009 at: http://dst.gov.in/admin_finance/ls_9/un_sq3449.htm.         [ Links ]

[14] The World Bank. World Development Indicators 2004/05. Last consulted May 16, 2009 at: http://go.worldbank.org/7C8HQXCS90.         [ Links ]

[15] Grainger D.W. Peer review as professional responsibility: a quality control system only as good as the participants, Biomaterials, 28; 5199-5203, 2007.         [ Links ]

[16] Bailar J.C., Patterson K. Journal peer review: the need for a research agenda. New England Journal of Medicine, 312:654-657, 1985.         [ Links ]

[17] Council of Science Editors. CSE's White Paper on Promoting Integrity in Scientific Journal Publications. Last consulted May 16, 2009, at: http://www.councilscienceeditors.org/editorial_policies/whitepaper/2-3_reviewer.cfm.         [ Links ]

[18] Seals D.R., Tanaka H. Manuscript peer review: a helpful checklist for students and novice referees. Advances in Physiology Education, 23:52-58, 2000.         [ Links ]

[19] Black N., van Rooyen S., Godlee F., Smith R., Evans S. What makes a good reviewer and a good review for a general medical journal? Journal of the American Medical Association, 280:231-233, 1998.         [ Links ]

[20] Benos D.J., Kirk K.L., Hall J.E. How to review a paper. Advances in Physiology Education, 27: 47-52, 2003.         [ Links ]

[21] Provenzale J.M., Stanley R.J. A systematic guide to reviewing a manuscript. Journal of Nuclear Medicine Technology, 34: 92-99, 2006.         [ Links ]

[22] Young Scientists Refereeing Programme. Biomaterials. Last consulted on May, 16, 2009, at: http://www.editorialmanager.com/biomat/.         [ Links ]

[23] Schroter S., Black N., Evans S., Carpenter J., Godlee F., Smith R. Effects of training on quality of peer review: randomized controlled trial. British Journal of Medicine, 328: 673-675, 2004.         [ Links ]

[24] Editorial (anonymous). Bad peer reviewers. Nature, 413: 93, 2001.         [ Links ]

[25] Horrobin D.F. The philosophical basis of peer review and the suppression of innovation. Journal of the American Medical Association, 263:1438-1441, 1990.         [ Links ]

[26] Rennie D. Freedom and Responsibility in Medical Publication: Setting the Balance Right. Journal of the American Medical Association, 280:300-302, 1998.         [ Links ]

[27] Hargens L.L. Variation in journal peer review systems: Possible causes and consequences. Journal of the American Medical Association, 263:1348-1352, 1990.         [ Links ]

[28] Kassirer J.P., Campion E.W. Peer Review: Crude and Understudied, but Indispensable. Journal of the American Medical Association, 272: 96-97, 1994.         [ Links ]

[29] Harnad S. Peer commentary on peer review: A case study in scientific quality control, Cambridge University Press, New York, (1982).         [ Links ]

[30] Zucker R.S. A Peer Review How-To. Science, 318:32, 2008.         [ Links ]

[31] Perrin W.F. In search of Peer Reviewers. Science, 318:32, 2008.         [ Links ]

[32] Marsh H.W., Jayasinghe U.W., Bond N.W. Improving the Peer-Review Process for Grant Applications: Reliability, Validity, Bias, and Generalizability. American Psychologist, 63:160-168, 2008.         [ Links ]

[33] Horrobin D. Anonymity of reviewers. Cardiovascular Research. 28:1141, 1994.         [ Links ]

[34] Lock S. Anonymity of reviewers. Cardiovascular Research. 28:1141, 1994.         [ Links ]

[35] Rennie D. Problems in peer review and fraud: cleave ever to the sunnier side of doubt. In: Balancing act. Essays to honour Stephen Lock. London: Keynes Press, 1991. pp. 9-19.         [ Links ]

[36] Kmietowicz Z. Double blind peer reviews are fairer and more objective, say academics. British Medical Journal. 336:241, 2008.         [ Links ]

[37] Editorial (anonymous). Working double-blind. Nature, 451:605-606, 2008.         [ Links ]

[38] Budden A.E., Tregenza T., Aarssen L.W., Koricheva J., Leimu R., Lortie C.J. Double-blind review favours increased representation of female authors. Trends in Ecology and Evolution, 23:4-6, 2008.         [ Links ]

[39] Justice A.C., Cho M.K., Winker M.A., Berlin J.A., Rennies D, PEER Investigators. Does asking author identity improve peer review quality? A randomized controlled trial. Journal of the American Medical Association, 280: 240-242, 1998.         [ Links ]

[40] Cho M.K., Justice A.C., Winker M.A., Berlin J.A., Waeckerle J.F., Callaham M.L., Rennie D. PEER Investigators. Masking author identity in peer review: what factors influence masking success? Journal of the American Medical Association, 280: 243-245, 1998.         [ Links ]

[41] Harnad S. Implementing peer review on the Net: Scientific quality control in scholarly electronic journals. In: Peek R, Newby G. Scholarly Publication: The Electronic Frontier. MIT Press, Cambridge MA, 1996. pp. 103-108.         [ Links ]

[42] Chew, F., Llewellyn, K., Olsen, K., Electronic publishing in radiology: Beginnings, current status, and expanding horizons. Journal of the American College of Radiology, 1: 741-748, 2004.         [ Links ]

[43] Editorial (anonymous). Who'd be a referee? Nature Nanotechnology. 3:119, 2008.         [ Links ]