This paper presents three distinctly different approaches to the rapid review of evidence to address a pre-defined research question within a limited time frame. The approaches included a brief summary and grouping of results for the sexual health review [12], meta-analysis incorporating data from existing reviews and new RCTs for the premature ejaculation review [13] and reporting of a narrative summary of significant outcomes for each intervention/comparator pair in the cannabis cessation review [14]. All three reviews included other rapid review methods such as the use of a focussed research question, partial double sifting of titles and abstracts and partial double data extraction. Only numerical data was double data extracted in the cannabis and premature ejaculation reviews. Narrative synthesis was used, as opposed to full qualitative synthesis of evidence, in two of the reviews (sexual health interventions [12] and cannabis cessation [14]). Cameron et al. [16] suggest that rapid reviews may benefit from the rigour of external peer review. All three rapid reviews included in this study had expert advisory panels and were peer reviewed by a minimum of two independent external reviewers. They were also deemed to be of relatively high quality using the AMSTAR checklist with scores of 10/11, 9/11 and 9/11.

Methods chosen for the rapid review of evidence must be both feasible and appropriate taking into account the requirements of the commissioners, the quantity and nature of the evidence and the time and resources available to do the review. The methods chosen for these three reviews were deemed acceptable to the commissioners as they approved the protocols where the methods were described in detail. The reasons for the commissioning of the reviews varied and this fed into the choice of review methods. For the sexual health review, the majority of evidence was from North America and it was not known how transferable this might be to the UK. The aim of the review was to inform the design of a UK based trial. The aim of the premature ejaculation review was to identify the most effective treatment option or combinations of treatment and the aim of the cannabis cessation review was to evaluate interventions for cannabis use and to identify important evidence gaps that might require further research. In all three reviews alternative methods could have been chosen such as the inclusion of fewer interventions, comparators or fewer outcomes, restriction of study design or the updating of existing systematic reviews. However, all of these options would have meant that either the needs of the commissioners were not met or that the high standards of the NIHR were not adhered to.

Policy makers require evidence to make decisions in a timely manner therefore choices need to be made as to how to select, analyse and present the evidence required. We chose the approaches presented here in part because other approaches were not acceptable to the policy makers (limiting the number of interventions considered). Research by Cameron et al. [16] comparing the findings from rapid vs full reviews found no difference in the essential conclusions reached by the reviews. Traditional systematic reviews are more likely to provide greater depth of information than rapid reviews [1] although rapid reviews have been found to meet the needs of knowledge users [8]. It is however important, as suggested by Schünemann and Moja [17], to ensure that guidelines for review conduct and reporting are adhered to. Boundaries between rapid and full systematic reviews are often blurred and many published systematic reviews use rapid methods.

The key factors identified in this study for consideration when selecting rapid review approaches include: an assessment of the evidence base, consideration of how to present the evidence, understanding the needs of the policy makers and adequate reporting of methods and their strengths and limitations. The range of reviewing options can then be considered by the reviewer once the size of the evidence base is ascertained including limiting the scope of the review [5] and streamlining processes for full text review and data extraction [1]. The incorporation of good quality existing systematic reviews is potentially very useful [6] [8] and can save valuable time and resources.

Previous research has also highlighted the importance of reporting and communication with policy makers. Without a full description of methods the direction and magnitude of any risk of bias cannot be fully assessed [1]. This has real implications for policy makers when making decisions based on rapid reviews and it is essential that limitations are clearly described. Varker et al. [18] advocate the use of a reporting template in rapid reviews in order to ensure a consistent approach. Reviews must be “fit for purpose” so that they reflect the knowledge needs of the commissioning body [10]. Hartling et al. [19] also stress the range of methods employed are both driven and supported by close and ongoing communication between the producer of the review and the end user, which is a very different context from most standard systematic reviews. Feedback from policy makers is crucial to ensure that the use of rapid review methods did not hinder decision making and to determine what approaches are useful to them and which are not.

There are several limitations to this research study. Only three rapid reviews were assessed, all from the same institution involving an overlap of reviewers. All three reviews were undertaken for the same HTA programme in the UK. Other programmes may have had other requirements and meant that other approaches were needed. Other reviewers from other organisations may have chosen different review methods. This limits the findings of the research. The authors of this paper were also the reviewers for the three case studies and therefore had in depth knowledge of the reasoning behind choices but other data extractors may have come to different conclusions compromising the replicability of this study. We deliberately looked at our own reviews so that we could provide an in depth description as to how and why decisions were made about which rapid review method to use, and what we found to be the strengths and limitations of these methods.

More research is needed in this area to provide guidance to reviewers to enable them to choose the most appropriate rapid review methods. One possible approach would be to select a sample of rapid reviews produced by a range of groups that used the general approaches outlined in these case studies and then use this sample to compare both within and between the three case approaches to help establish best practice in this area. Another possible future research approach would be to compare the rapid review methods used with a full systematic review in the same area. We have recently published a paper comparing our chosen rapid review methods with a full review for one of the interventions from the premature ejaculation review [20]. We found that in the topic area primary outcome data were the same whether the de novo rapid review method or a full review method were employed. However, due to limited reporting across reviews, quality assessment of all RCTs could only be undertaken as part of the full systematic review. Finally, future research could explore the comparison of different rapid review approaches on the same topic, bearing in mind that not all approaches will be relevant and feasible for every topic.

We did not receive feedback on whether or not our reviews met the needs of the commissioners, although this was requested. We have suggested this become part of the NIHR peer review process in future. Future research is needed on how best to incorporate feedback from commissioners and policymakers as to how useful the reviews were for decision making as well as potential limitations due to the chosen review methods. There is now a considerable amount of literature available on the use of rapid reviews with details of the methods available and limitations associated with these. There is little guidance on how to choose the most appropriate method for the evidence base identified. It is crucial that commissioners and policy makers have sufficient information to make a judgement on whether or not the chosen review approaches may be considered appropriate and robust. They must also be made aware of what approaches are feasible, bearing in mind the quantity and nature of the evidence as well as the time and resource constraints of the review. Details of the strengths and the limitations of the methods chosen must also be presented to commissioners and policy makers and their potential impact on decision-making. This research goes some way in exploring possible approaches suitable in this context.