Building a Reliable Peer Review Process

Organizing a robust peer review process for an academic journal involves establishing clear policies, managing the editorial board and reviewers effectively, implementing efficient submission handling, overseeing the review process itself, and maintaining ethical standards and transparency.

Define and Document Clear Peer Review Policies and Procedures

Adopt a peer-review process suitable for your journal, field, resources, and systems available.
Ensure that all peer review processes are transparently described and well managed. State on the journal website what is peer reviewed, the model used, and how the process is managed. Journals can use various models, such as single-anonymous, double-anonymous, or open review, and should clarify confidentiality and ownership of the review product based on the model. Double-blind review is not always possible or necessarily better than other models.
Establish and publish policies covering various aspects of peer review, including the number of reviewers used, how reviewers are selected and trained, policies on author-recommended and excluded reviewers, and procedures for handling conflicts of interest involving reviewers.
Outline how decisions about manuscripts are made and who is involved.
Develop processes for handling appeals and disputes that may arise in peer review.
Define policies and procedures regarding the editing of peer reviews.
Implement processes for preventing and detecting manipulation of the peer review process.
Ensure policies on ethical oversight, such as obtaining informed consent and ethical approvals for research, are addressed as part of the review criteria, and reviewers are asked to address these aspects.
Develop policies on data availability and encourage the use of reporting guidelines, which can be reviewed during the process.

Establish and Manage the Editorial Board and Reviewers

Review and confirm the roles and responsibilities of all editors, including those involved in the peer review team. Editorial Board Members (EBMs) can make final decisions, supervise special issues, and identify suitable reviewers.
Build a strong editorial board comprising recognised experts relevant to the journal's aims and scope. Vet new EBMs before onboarding, checking for suspicious affiliations with questionable journals.
Implement processes for selecting, training, and assessing the performance of reviewers. Be aware that reviewers may take on the role without guidance and require training. Journals should have systems for assessing reviewer performance and removing those with unacceptable performance.
Ensure assigned reviewers are qualified, free of conflicts of interest, and have no known ethics or performance concerns.
Manage reviewer activity and have processes for re-engaging or removing inactive members.
If supervisors involve students or junior researchers in peer review for training purposes, they must request permission from the editor, and the student should be acknowledged as the reviewer of record.

Implement Efficient Submission Handling and Screening

Get to grips with the mechanics of the journal's submission system and timelines, preferably in conjunction with the previous editor.
Work with the publisher/owner to determine efficient and appropriate submission processes. Electronic submission systems can help ensure authors provide required information (e.g., authorship, funding) but should avoid being overly complex. Consider requiring all elements to be complete before a manuscript is sent for peer review.
Implement preliminary screening (pre-check) procedures. This involves confirming the manuscript fits the journal's scope, checking for potentially controversial topics, ensuring language quality, verifying compliance with instructions for authors, including ethics, data availability, image quality, and copyright.
Run plagiarism checks and verify that any revisions have addressed overlap.
Verify author affiliations and emails, check authorship consistency and contributions, and look for previously flagged concerns or misconduct history.
Ensure figures, tables, and references are appropriate, well-presented, properly cited, and of sufficient quality. Check for author/journal self-citation rates.

Manage the Peer Review Process

Ensure that material submitted remains confidential while under review. Reviewers must respect the confidentiality of the process. They should not discuss unpublished manuscripts with colleagues or use the information in their own work without permission.
Assign reviewers whose expertise matches the scope of the content.
Provide reviewers with clear instructions on how to perform a review, the time allowed, formatting, and what to consider, including ethical aspects. Reviewers should be objective and constructive, providing a fair, honest, and unbiased assessment.
Monitor the process regularly to ensure peer review is undertaken in a timely fashion and prevent delays.
Ensure review reports are constructive and professional. Provide reviewers with instructions on preparing their report.
Have a process for managing conflicting reviewer decisions, which might involve a double decision process or seeking additional reports.
For resubmissions, confirm that authors have replied to all reviewer comments.
Ensure that editors making final decisions are free of conflicts of interest.
Implement a clear procedure for handling submissions where the editor or a board member is an author, ensuring independent peer review to avoid the appearance of impropriety. Transparency about this process is recommended.

Ensure Ethical Conduct and Transparency

Ensure decisions to accept or reject a paper are based only on the paper's importance, originality, clarity, and relevance to the journal's scope, avoiding bias. Do not link editorial decisions to internal targets or expedite decisions for metric purposes.
Develop guidelines for promptly responding to suspected ethical breaches by authors, reviewers, and editors.
Have clear processes for handling complaints against the journal, staff, editorial board, or publisher.
Be willing and have mechanisms in place for correcting, clarifying, revising, or retracting articles after publication when needed. Prompt retraction of seriously flawed articles is considered a responsible action (implied by the need for effective mechanisms).
Maintain a well-supported, maintained, and secure journal website. The website is the public-facing statement of ethical practices.
Ensure the journal's name is unique and does not mislead.
Clearly state the publishing frequency and indicate content preservation plans.
Follow clear communication practices and do not disclose inappropriate information, such as personal contact details.

Starting with an assessment tool like the COPE Journal Audit can help identify areas needing attention or revision to align with COPE's Core Practices. Adhering to the Principles of Transparency and Best Practice in Scholarly Publishing is also crucial.