A Proportionate Peer Review Service

By SRAI JRA posted 14 days ago

  

Volume LII, Number 2

A Proportionate Peer Review Service

Charles Mackworth-Young
Professor of Practice and Director of Peer Review Service, Imperial College 

Abstract

Background: The peer review of clinical research projects is an essential step in project preparation. While some projects undergo rigorous review by grant-giving organizations, this does not apply to all clinical research. In many cases, peer review, if undertaken at all, is not rigorous, fully independent, or timely. Depending on their department or institution, many researchers do not have easy access to such review. This can result in delay to ethical and institutional approval, and uncertainty about the quality of individual projects.

Methods: After a two-year pilot study the Peer Review Service (PRS) was established at Imperial College, London in 2009. The aim was to provide an easily accessible and quick service for researchers at all the clinical sites associated with Imperial College. A graded system of review levels was designed which used an algorithm to match the independence and robustness of the review to the ethical weight of each project. The levels ranged from 1—internal review by a supervisor to 5—fully independent review by at least two external individuals. A semi-structured response form for reviewers was generated to facilitate the review process and ensure that all relevant aspects were considered. For each reviewed project, the PRS issued a certificate confirming the quality of the review.

Outcomes: There was a gradual increase in use of the service from the 2009 inception. From January to June 2020, 63 projects underwent peer review commissioned by the service. This represented all of the clinical research projects performed at Imperial sites that required review. The mean time from application to delivery of review was 1.73 weeks. Administrators found the algorithm for determining the peer review level easy to use, occasional queries being managed by members of the supervising committee. Audits demonstrated that researchers, reviewers and ethics committees were satisfied with the service.

Conclusion: A proportionate system of peer review for clinical research projects works well. It produces appropriately robust and independent reviews and can be implemented easily by administrative staff. Close association of a peer review service with university research administration ensures that all projects needing peer review receive it. The centralized service assists researchers in obtaining reviews speedily.

This simple model could be used widely by other clinical research centres.

Keywords: peer review, proportionate review, centralized review

Introduction

For clinical research projects to be approved and initiated several regulatory steps are required. One of these is peer review. It is important for a sponsor and/or host organization to be satisfied that a proposed project has sufficient scientific merit. The same applies to the research ethics committee: recruiting participants to a project that is poorly designed is likely to be unethical (NHS Health Research Authority, 2020). The sponsor and ethics committee do not generally have the expertise to determine the scientific validity of a proposed project—hence the requirement for independent peer review.

Many projects are funded by major grant-giving organizations. Examples in the UK are the Medical Research Council, the Wellcome Trust and members of the Association of Medical Research Charities. Such projects undergo robust independent peer review by the funding body— in which case further review is usually unnecessary. Exceptions to this include projects that are proposed under the umbrella of an existing programme grant but which were not specifically considered when the original application was reviewed.

Projects that have funding from other sources may undergo peer review that is not fully independent or robust. Examples include small scale investigator-led research, and studies that are supported by family trusts and similar organizations.

There have been a variety of approaches among organizations to seeking peer review where this is needed (Wood & Wessley, 2003). It is often arranged at department level. However, this can compromise the independence of the reviews, and lead to inconsistency within and between organizations.

At Imperial College, London a more centralized peer review service for clinical research projects was established in 2009 to address these points. It was also recognized that the level of scrutiny required would vary between types of project: for instance, a serological study requiring the donation of a single 10ml blood sample would probably need less extensive review than a two- year interventional drug trial. A centralized, proportionate system of peer review was therefore designed and introduced.

This paper describes the implementation of this system, our experience with it over the first 10 years, and our conclusions regarding its wider adoption. 

Methods

Pilot Scheme

From 2007 to 2009 a pilot peer review service was run at Hammersmith Hospitals, two clinical centres that are part of the Imperial College group of hospitals. The service was offered on a voluntary basis to all researchers proposing clinical research projects. It was based on a proportionate review system (see below). At the end of the pilot period a formalized Peer Review Service (PRS) was set up. Experience from the pilot scheme informed the design of the system used by the PRS. 

Administration

The Peer Review Service (PRS) was established in 2009. It was closely linked to the Joint Research Compliance Office of Imperial College ( JRCO). This manages all the administrative aspects of clinical research at Imperial College and all associated clinical sites. This includes arranging sponsorship and indemnity.

The PRS was designed as an administrator-run service, overseen by a committee. Committee membership includes representatives of major clinical areas (medicine, surgery, women and children’s health, nursing, therapies and pharmacy). The committee designed the processes and continues to oversee and review them. The Committee does not carry out peer review itself, but advises on strategic and operational matters and, where necessary, on queries made by researchers.

Use of the Service

The Peer Review Service (PRS) is available for use by all clinical researchers at Imperial College and its associated hospitals. Use of the service is not compulsory; however, it is strongly recommended unless independent peer review has been obtained elsewhere. Even in this situation, confirmation by the PRS that the review is sufficiently robust and independent is advised.

Use of the PRS is widely promoted at Imperial College via its website and through research managers. All clinical research projects need to be registered with the JRCO. Since 2018, it has been possible for details to be passed on directly to the PRS.

Proportionality

A major feature of the PRS process is proportionality of review in relation to the potential burden of the project. The design addresses the specific requirement of providing evidence to research ethics committees and the JRCO.

Five “levels” of review are used. These are summarized in Table 1, together with examples of the type of project at each level, and the minimum review required. A complete list of examples is given in the Appendix. The levels range from 1, for which no review is required, to 5, for which two reviewers independent of the researcher’s institution are needed. Most projects fall into Levels 1 to 4. However, projects that are supported by the National Institute for Healthcare Research (NIHR) require a greater number of independent reviewers and are therefore allocated to Level 5.


The minimum requirement for Level 4 was set at one external and one internal reviewer. Both reviewers must be fully independent of the project. The reason why this requirement was chosen is that Imperial College with its ten associated clinical centres is a particularly large institution, and it was therefore felt reasonable for one of the reviewers to come from within. However, if an internal but independent reviewer cannot be identified, a second external reviewer is selected. The same rationale was behind the decision to require one internal reviewer for Level 3.

The administrator determines the level of a project using an algorithm. If necessary, advice is sought from a member of the committee. In the case of Level 1b (research already reviewed by a major grant-giving body), the administrator asks the researcher for evidence that the project in question was specifically considered in the peer review. In some instances a project may be proposed under the umbrella of a programme grant, the application for which did not give specific details of the project. In this case, the project is allocated a level according to the algorithm.

Projects at Level 2 are most commonly student projects. They do not require processing by the PRS, although it can issue a certificate to confirm this. The PRS arranges peer review for projects at Levels 3-5.

Process

The PRS receives applications for peer review, or for confirmation that peer review is not required. All documentation and correspondence is performed electronically.

Researchers submit the protocol, participant information sheet and related supporting documents (e.g. questionnaires). The administrator uses an algorithm to determine if peer review is needed, and, if so, at what level. If it is not needed, a certificate is issued confirming this.

For projects that need independent review (Levels 3 to 5) the researchers also submit details of two potential reviewers, using a form that asks about their independence from the researchers and the project, and their expertise in the field. Researchers are also invited to submit the names of up to two individuals who they would prefer not to review their projects.

The administrator then selects potential reviewers. They may be identified from the PRS’ own database, by using the Web of Science or by members of the committee. If needed reviewers suggested by the researchers may be used. Potential reviewers are contacted and invited to complete a form that asks about their independence and expertise. Once these have been returned they are compared with those submitted by the researchers. If the reviewers are confirmed as suitable, they are sent the project documents, together with a review form template (see Review Form below).

When a review form is returned, the administrator checks that all questions have been adequately answered. This is a check on the quality of the review, not the project itself. Once sufficient, adequate reviews have been received, the administrator issues a certificate confirming that the project has undergone appropriate peer review. The certificate is then sent to the researcher, together with the reviews.

The PRS aims for the turnaround time—from receipt of all documents to the issuing of a certificate—to be under four weeks. Members of the committee are available to provide advice to the administrator at any point in the process.

Use of the PRS is free for researchers. Reviewers are not paid but are thanked warmly for their assistance. It is made clear on every certificate—from Level 1 to 5—that it is not a commentary on the quality of the project, but merely confirmation about the status of the project and the quality of the review, if performed.

Review Form

The review form was designed to be easy and quick to use by reviewers. It is semi-structured, with questions requiring yes/no answers, and space for free text. Questions are asked in the following domains: context, research design, sampling, clinical considerations, practicalities, and overall recommendation. It is completed anonymously, although reviewers can ask to have their names revealed to the researchers if they wish.

Quality Assurance

The committee reviews the activity and performance of the PRS biannually. The turnaround time is regularly monitored. Structured forms are used to obtain feedback from researchers and reviewers about the ease of use of the service, and from research ethics committees about the value of the reviews. 

Outcomes

General

The peer review system based on the experience in the pilot scheme appeared to work well. In the first three years establishing visibility of the PRS was assisted by meetings with research managers and heads of department. The service was also promoted on the College intranet.

Proportionality

The proportionate system was easily understood by researchers and managers. The definitions of the peer review levels were adjusted by the committee in the first few years, with more examples of projects being added at each level as new studies were considered. Level 5 was introduced in 2013 to take account of the NIHR requirement. Queries about level allocation or the quality of reviews were generally dealt with by committee members outside formal meetings.

Because of the size of Imperial College and its linked hospitals, independent internal reviewers were almost always easy to find for projects at Levels 3 and 4. On some occasions was this not possible, and it became necessary to find an external reviewer.

Activity

In the first year of activity of the PRS the total number of certificates issued at all levels was 34. Thereafter until 2015 between 50 and 70 certificates were issued per year. Following this there was a gradual increase in activity, shown in Table 2. This appeared to be mainly due to the closer connection of the PRS with the JRCO, so that by late 2019 all projects that had not already had adequate peer review were referred directly to the PRS. Also contributory were increasing visibility of the service within Imperial College, and the growing experience of researchers that the PRS process was quick and easy.


Table 3 shows activity at the various levels from January to June 2020, measured by the issuing of certificates. Certificates were issued for 115 projects. Of these, 63 underwent peer review commissioned by the service. Cross-referencing with the JRCO database indicated that all of the clinical projects registered by the JRCO in that period which required independent peer review were processed by the PRS.


Turnaround

Table 3 includes the turnaround times for issuing certificates from the six months from January to June 2020. The results indicate that the target of turnaround of less than 4 weeks was met in almost all cases.

The consistent turnaround time of less than one week for projects at Levels 1 and 2 reflects the fact that no review was commissioned for them by the PRS.

The mean turnaround time for Levels 3 to 5 was 1.73 weeks. Turnaround times greater than two weeks for these levels were all due to reviewers being slow to respond. Nevertheless, only in one case was the target time of less than 4 weeks exceeded. In most cases at Levels 4 and 5 more external reviewers were requested than the minimum required as an insurance against delay. The review process generally occurred in parallel with other regulatory processes and so did not usually delay the progression of the projects. 

Quality of Reviews

Because of the design of the review form, the quality of each review could usually be assessed by the administrator. Occasionally the advice of a member of the committee was sought. The vast majority of reviews were found to be sufficient. In 2016-2020 only two reviews were deemed to be inadequate. In these cases, other reviewers were sought.

Quality of Projects

The commissioned reviews gave a full range of opinions, from unconditional support to severe criticism. The majority gave helpful suggestions. Researchers were expected to submit these to the JRCO and to the relevant ethics committees, with an explanation of how these suggestions were addressed.

Feedback

A questionnaire sent to 20 consecutive researchers with projects at Levels 3-5 indicated universal satisfaction with ease-of-access of the PRS, and speed of obtaining reviews. One researcher queried the level allocation if his project. Otherwise there were no adverse comments.

A questionnaire sent to 20 consecutive independent reviewers found satisfaction with the process and the review form. There were no adverse comments.

Questioning ethics committees was more difficult, since it required the researcher to indicate which ethics committee would be considering the project, and then asking the administrator of that committee to request that the committee provide an opinion. Thirty approaches to ethics committees were made using this route; only five responses were obtained. In all cases the reviews supplied were felt by the ethics committee to be of good quality and sufficient for their purposes. 

Discussion

Peer review plays an important role in clinical research, notably in the selection of projects for funding and in the assessment of manuscripts for publication (Wood & Wessley, 2003). It is also needed by host organizations and by research ethics committees for confirmation that proposed projects are of sufficient merit. The Imperial College Peer Review Service was set up to address this specific requirement. This paper describes the implementation of the service and its development over the first 10 years.

We found that for an organization with multiple academic sites a centralized system worked well. Researchers had easy access to it, and the turnaround was fast. By the end of the first 10 years all projects within the Imperial group requiring independent review were being processed by the service. Because the review process was commonly performed in parallel with other regulatory processes, it did not usually delay the progress of projects.

A novel proportionate system was used. This was easily understood by all involved. Feedback from researchers, reviewers and ethics committees was good. The minimum requirements for Level 3 (one independent internal reviewer) and Level 4 (one internal and one external reviewer, both independent) were determined by the size of Imperial College and its associated clinical centres. This appeared to work well and could be replicated in other institutions. Some modifications might be needed: for instance, in smaller organizations it would probably be necessary to raise the requirements, e.g., to one and two external reviewers, respectively.

At its best, peer review can be constructive. Helpful suggestions from reviewers can improve project proposals, and thus raise the overall standard of research (Huisman & Smits, 2017). The short turnaround time of our system meant that this could happen without significantly delaying the approval process.

Independent peer review is a powerful tool that commands great respect. But it is also imperfect (Smith, 2006; Neff & Olden, 2006). Its many drawbacks have been well described. They include the competitive nature of some research, the difficulty in finding truly independent “peers,” and the disinclination of many potential reviewers to perform such unpaid work. Nevertheless, most clinical researchers would accept that, when performed to a high standard, it is probably as good a system as we can devise within the limitations of funds and personnel ( Jefferson et al., 2002).

Our system uses some reviewers that are suggested by the researchers. Ideally all reviewers would be identified independently of the researchers. However, in some fields this can be difficult. The use of researcher-suggested reviewers is an established practice for many institutions and journals, as is the avoidance of using particular reviewers at the request of researchers (Earnshaw et al., 2007). All potential reviewers are added to our database, and we aim to reduce the use of researcher-suggested reviewers over time.

Most peer review systems are dependent on the willingness of researchers to review the work of their peers without reward. There may be small advantages to being a reviewer, such as maintaining one’s profile and learning about the research plans of other units; but most researchers feel a moral imperative to contribute to this aspect of science, from which they also benefit when submitting their own projects. In setting up our system we have been dependent on this goodwill, and are grateful for it. 

Conclusion

The Imperial College Peer Review Service offers a proportionate certification method and rapid access to robust, independent peer review. Our experience suggests that this type of centralised system could be replicated or adapted at other centres. This would be of benefit to researchers, host institutions, ethics committees, the quality of research, and therefore, ultimately, patients. 

Authors’ Note

I would like to thank the members past and present of the Peer Review Committee for their contributions to designing the Imperial College peer review system, and for their work in overseeing it. Thanks are also due to Tanja Schiotz, who was central to setting up the pilot study, and to her successors as administrators at the PRS, notably Gule Hanid, who helped with providing the data for this paper.

Charles Mackworth-Young
Professor of Practice and Director of Peer Review Service
Imperial College, Charing Cross Hospital
London W6 8RF, UK

References 

Earnshaw, J. J., Farndon, J. R., Guillou, P. J., Johnson, C. D., Murie, J.A., & Murray, G. D. (2000). A comparison of reports from referees chosen by authors or journal editors in the peer review process. Annals of the Royal College of Surgeons of England, 82(4 Suppl), 133-135. 

Huisman, J. & Smits, J. (2017). Duration and quality of the peer review process: The author’s perspective. Scientometrics, 113(1), 633-650. https://doi.org/10.1007/s11192-017-2310-5 

Jefferson, T., Wager, E., & Davidoff, F. (2002). Measuring the quality of editorial peer review. JAMA, 287(21), 2786-2790. https://doi.org/10.1001/jama. 287.21.2786 

Neff, B. D., & Olden, J.D. (2006). Is peer review a game of chance? BioScience, 56(4), 333- 340. http://dx.doi.org/10.1641/0006-68(2006)56[333:IPRAGO] 2.0.CO;2 

NHS Health Research Authority. (2020, March 26). Governance arrangements for research ethics committees: 2020 edition. https://s3.eu-west-2.amazonaws.com/www.hra.nhs.uk/ media/documents/GAfREC_Final_v2.0_26.03.2020.pdf 

Smith, R. (2006). Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4), 178-182. https://dx.doi. org/10.1258%2Fjrsm.99.4.178 

Wood, F., & Wessley, S. (2003). Peer review of grant applications: A systematic review. In T. Jefferson & E. Godlee, Eds., Peer review in health sciences (pp. 14-44). BMJ Books.

APPENDIX: PEER REVIEW LEVELS

Review requirements and types of project, with examples

Level 1a

Minimum level of review required: None required

Types of project: Studies that involve minimal risk to participants

Examples of projects or procedures:

  • Use of data from medical notes by clinician looking after patient
  • Short questionnaire studies for use among hospital staff or GPs
  • Questionnaires asking participants about the quality of hospital services, or requesting other non-personal data, taking up to 10 minutes for a patient, or 20 minutes for a healthy volunteer

Level 1b

Minimum level of review required: None required

Types of project: Studies that have been specifically peer reviewed by a major grant-giving body or similar organization. These include: UK Research Councils (including the Medical Research Council); the National Institute for Health Research; and Members of the Association of Medical Charities (including the Wellcome Trust and a large number of specialist or disease- specific charities). This exemption does not include projects that are part of a program grant but which have not been specifically considered by the grant-giving body.

Level 2

Minimum level of review required: Departmental colleague or student project supervisor

Types of project: Low-risk projects with minimal patient involvement. Student projects that involve either no patient/participant involvement or only minor involvement

Examples of projects or procedures:

  • Routine history taking
  • Projects using existing stored data
  • Administration of simple questionnaires that do not involve "sensitive" (e.g. psychiatric, sexual, drug or end of life-related) information, unless that information is part of normal clinical practice for the condition under study
  • Non-intimate physical examination g. joint examination, blood pressure measurement
  • Photography if participant is not identifiable
  • Venesection involving a single skin puncture: up to 50mls total from healthy volunteers, 20mls total from patients (or pro rata for children)
  • Taking of blood via existing cannula or at same time as venesection which is part of normal patient cars: in single or multiple samples, total volumes as above
  • Spirometry
  • The obtaining or analysis of non-invasive samples, g. urine, saliva, faeces
  • Histological studies on existing/historical specimens
  • Standard MRI scanning

    Level 3

    Minimum level of review required: Individual within Imperial College or the applicant's hospital trust

    Types of project: Involving minor patient or participant risk

    Examples of projects or procedures:

    • Single-arm study of a drug or device not affecting patient care decisions
    • Clinical intervention study or controlled trial with low risk to participants (e.g. a study of an oral nutritional supplement, low vitamin doses, or dietary intervention)
    • New acquisition of personal data that are not part of the normal clinical history
    • Administration of questionnaires involving "sensitive" information outside normal clinical practice
    • Intimate physical examination when appropriate to clinical context
    • Photography/recording if participant is identifiable
    • Taking of up to two blood samples of no more than 100mls in total from healthy volunteers, 50mls from patients (or pro rata for children)
    • Taking of extra biopsies during biopsy procedure that is part of normal care
    • A minor lengthening of an invasive procedure (less than 5 minutes or 10% added to a procedure that is part of patient care, whichever is the shorter), with little or no extra risk associated with either the investigation or the lengthening of the procedure
    • Investigation that involves a minimal risk procedure (e.g. arterial blood gas analysis)
    • DNA analysis with no clinical implication for the participant 

    Level 4

    Minimum level of peer required: Two reviewers, including one individual outside IC or the applicant's hospital trust

    Types of project: Projects with greater than minor risk to participants

    Examples of projects or procedures:

    • Phase I, II and III drug or device trials
    • Randomized trials of drugs or devices within their licensed use
    • Use of radiation
    • Intimate physical examination outside appropriate clinical context
    • DNA analysis with potential clinical implication, g. for new diagnosis
    • Studies involving embryos

    Level 5

    Minimum level of review required: Two reviewers, both outside Imperial College or the applicant's hospital trust

    Types of project: those for which NIHR adoption is sought

    0 comments
    0 views

    Permalink