Challenges in identifying and quantifying country-level participation in multi-country randomised controlled trials involving Ireland as a collaborating partner
Department of General Practice, RCSI University of Medicine and Health Sciences, Dublin, Ireland
School of Pharmacy and Biomolecular Sciences, RCSI University of Medicine and Health Sciences, Dublin, Ireland
Department of Public Health and Epidemiology, School of Population Health, RCSI University
*Larkinja@tcd.ieAbstract
Background and Objective
Randomised controlled trials (RCTs) provide vital information about healthcare interventions. Accurate reporting is vital for effective RCT dissemination. This study aimed to assess the reporting quality of multi-country RCTs, using Ireland as a case study, examining trial characteristics, adherence to reporting standards and the reporting of participation from Ireland.
Study Design and Setting
This is a secondary analysis of RCTs identified in a previous observational study of RCTs where ≥80% of participants were recruited in Ireland. This current study focuses on multi-country RCTs with Ireland as a participating country. The current study involved an additional screening process according to these inclusion criteria: RCTs conducted on humans in a healthcare setting with centres based in Ireland, and <80% of participants recruited in Ireland. The primary outcome variables were trial characteristics and reporting rates for: trial registration, use of standardised reporting guidelines, number of Irish centres and number of participants recruited in Ireland. Descriptive statistics were used for analysis.
Results
Overall, 239 RCTs were included. The most common intervention was a drug (74.9% of RCTs). The most common setting was an ambulatory setting (74.1% of RCTs). The most common clinical domain was the cardiovascular system (18.0% of RCTs). Among RCTs published after the CONSORT reporting guideline was published (1996), 8.3% referred to a standardised reporting guideline. Among RCTs published after the International Committee of Medical Journal Editors mandated clinical trial registration (2005), 81.8% provided registration numbers. Number of Irish centres was reported in 75.3% (N=180) of RCTs. Number of participants recruited in Ireland was reported in 27.2% (N=65) of RCTs.
Conclusion
Our findings show deficits in reporting quality for multi-country RCTs, particularly in referring to reporting guidelines and reporting number of participants for the examined country. Institutions should create policies to ensure transparent RCT dissemination.
Article notes
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
No specific funding was obtained for this work; however, JL's position was partly funded by a research grant from the Health Research Board in Ireland (grant ILP-HSR-2019-006), and for part of the study's conduct FM's position was funded by a research grant from the Health Research Board in Ireland (grant HRC/2014/01).
1.Introduction
Randomised controlled trials (RCTs) are essential for healthcare systems, in providing evidence about the safety and efficacy of interventions [1, 2]. Accurate reporting of RCTs is needed to facilitate critical appraisal, and for evidence to be acted upon. To encourage standardisation of reporting of RCTs, in 1996, the Consolidated Standards of Reporting in Trials (CONSORT) reporting guideline was published [3]. CONSORT is “a checklist of essential items that should be included in reports of RCTs,” [4] and is required by hundreds of journals [5, 6]. This includes items ranging from specifying outcomes of interest, approaches to randomisation, allocation concealment, and the setting(s) where the trial was conducted. Transparently reporting these items facilitates assessment of an RCT’s internal and external validity [5, 6]. For example, a clinician can evaluate whether the findings may be generalisable to their patient population, or whether the standard of care in the setting of the trial could impact the results. This particularly applies to multi-centre RCTs, which are increasingly ubiquitous in health research, however deficits in reporting of aspects of the multi-centre design are common [7, 8], prompting the development of multi-centre trial extensions to existing reporting guidelines [9].
Further measures that ensure transparent RCT reporting include protocol publication and pre-registering RCTs on a public trials registry. The latter, pre-specifying methods and all outcomes for an RCT and registering those details on a publicly accessible site prior to commencing patient enrolment [10], has been a publication requirement of the International Committee of Medical Journal Editors (ICMJE) since 2005, and is endorsed by various organisations including the WHO [11]. Using reporting guidelines and pre-registering RCTs aims to address several issues in RCT publications including publication bias and outcome switching [10]. However, despite these initiatives, reporting and registration issues remain [12, 13].
As well as enabling critical appraisal, clear reporting of the setting of RCTs in publications and registrations can also allow trial activity in different countries to be ascertained. This study uses Ireland as a case study to examine multi-country RCTs. RCTs are a central focus of health research in Ireland [14]. However, it is unclear whether the extent of a country’s involvement in international RCTs can be ascertained, for example, the number of participants recruited or study centres within a specific country. A 2020 study examined RCTs published before 2019 where it was reported that at least 80% of participants were recruited in Ireland [15]. Among 752 unique RCTs, it found a range of issues such as limited reference to standardised reporting guidelines, though it highlighted that the quality of reporting was improving over time [15]. That study did not examine RCTs where less than 80% of participants were recruited in Ireland.
Therefore, the aim of this study is to evaluate the reporting quality of multi-country, patient or population health-oriented RCTs. A single country is used as a case study, the study focuses on RCTs involving Ireland, examining their characteristics, their adherence to reporting standards and the level of Irish participation.
2.Methods
This is a secondary analysis, examining RCTs identified in a previous observational study evaluating quality and scope of RCTs in health research conducted in Ireland [15]. This previous study included RCTs where 80% or more of participants were recruited in Ireland [15]. The study drew on systematic review methods, searching six databases (e.g. PubMed), from inception to December 2018, and title/abstracts were reviewed followed by full-texts [15].
As part of the screening process for the previous observational study, RCTs were identified that had an indication of Irish participation (i.e. participants/centres/investigators from Ireland, or an author with an affiliation in Ireland) but did not clearly report at least 80% of participants recruited in Ireland. That screening process identified 428 RCTs, which were screened for inclusion in the present study based on criteria shown in table 1.
In assessing the population, any information in the publication (including in supplementary materials) indicating participants, study centres, or investigators from Ireland were taken as meeting this eligibility criterion (unless it was explicitly stated that no participants were recruited in Ireland or all recruited from Ireland). Where a trial registry number was included, the trial registry was checked for details of any centres in Ireland. Where conflicting information was presented in the publication and the registry, information from the publication was used. Publications were excluded where the only indication of participation from Ireland was an author with an affiliation at an Irish institution, but no further information that indicated participants recruited in Ireland. If the publication found was a study protocol, or interim, secondary, or sub-group analysis, a supplementary search was conducted (details in Appendix A eBox 1) to identify the main publication.
2.1Study inclusion and data extraction
Titles and abstracts for the 428 RCTs identified by Clyne and colleagues [15] were reviewed independently by two reviewers (EF or FM) who excluded any studies that clearly did not meet inclusion criteria. Full-texts for remaining publications were reviewed for eligibility by one reviewer (MC, VRN, UA, ST) according to the inclusion criteria. Data relating to participation from Ireland was checked by a second author (JL) in all cases. Any instances where it was unclear whether there was Irish participation were discussed with FM or TF.
A data extraction form, in Microsoft Excel, was piloted by two reviewers (FM and TF) and modified before the main data extraction phase. The extraction form included the following variables: publication information, eligibility, study documentation (e.g. ethics approval), study characteristics, methods, participants, and results. A standardised operating procedure for data extraction (details in Appendix A, eTable 1) was developed and read by reviewers before commencing, and this was iteratively updated with further guidance as queries arose. Data was extracted by one reviewer (MC, VRN, UA, ST, or JL), with 10% also reviewed and extracted by a second reviewer (FM) to assess consistency.
2.2Variables
There were several variables extracted, across the following categories: 1) Study characteristics and methods, 2) Study documentation and reporting, and 3) Countries, centres and participants (see Appendix A, eTable 2).
2.2.1Study characteristics and methods
Data on study design, unit of randomisation, setting, population, intervention and comparator type, funding source, and primary outcome. Primary outcome was categorised using the core areas within the outcome taxonomy developed by Dodd and colleagues: 1) physiological/clinical, 2) life impact, 3) death, 4) adverse events 5) resource use [16] and a sixth category that we added: other. Clinical domain was categorised using the UK Clinical Research Collaboration’s Health Research Classification System [17]. Other variables included, whether baseline characteristics were presented and whether the primary outcome result was positive or negative.
2.2.2Study documentation and reporting
The study documentation variables included whether there was reference to 1) ethical approval, 2) trial registration number, 3) a standardised reporting guideline, 4) publication of protocol, and 5) a sample size calculation (Appendix A, eTable 2).
2.2.3Countries, centres and participants
The main outcomes for Irish participation in the included RCTs were assessed as quantification of reporting of each of the following:
- Centres in Ireland among the total number of included centres
- Participants recruited from Ireland among the total number of participants
2.3Analysis
The study characteristics and methods of included RCTs were summarised using frequencies and percentages. Also, median (interquartile range) number of participants recruited overall and in Ireland for each characteristic was calculated. Study documentation and reporting was summarised as the number and percentage of RCTs referring to ethics approval, a standardised reporting guideline, published protocol or trial registration number, and where applicable, restricting to years since this provision was introduced. Percentage of RCTs with each was also plotted over time. For countries, centres, and participants, the number and percentage of RCTs reporting Irish participant at each level was summarised. The median (IQR) number and percentage of Irish centres and participants were also calculated, and violin plots generated to visualise the distribution of these. For countries, a network analysis was conducted to examine the pattern of countries participating in RCTs with Ireland. Lastly, the level of Irish participation reported in RCTs was summarised based on presence of key study documentation and funding source.
3.Results
From the 428 RCTs reviewed, 168 were excluded (47 at title/abstract screening, and 134 at full-text review), most often as secondary analyses, duplicates or not reporting participants recruited in Ireland (see Figure 1). A further 21 RCTs that had a co-author based in an Irish institution but no other indication of Irish involvement were excluded.
Overall, 239 studies met the inclusion criteria. For two included studies, the primary publication could not be retrieved and therefore associated publications (e.g. secondary analysis) were used to extract relevant information. However, it was therefore not possible to make conclusions on several variables for these RCTs (details in appendix A, eTable 3).
3.1Study characteristics and methods
RCTs were mainly parallel design (92.1%, N=220), evaluated drugs (74.9 %, N=179) among adults (74.9%, N=179) in an ambulatory (outpatient) setting (74.1%, N=177) (Table 2). The most common funding source was industry (54.8%, N=131), randomisation primarily occurred at the individual level (96.2%, N=230), the most common type of blinding was double-blinding (49.8%, N=119), and the primary outcome measure in 56.1% (N=134) of RCTs was categorised as physiological/clinical.
The baseline characteristics of participants were presented in 90.0% (N=215) of RCTs. In 25.1% (N=60) of RCTs the comparison group used was placebo drug or sham intervention (full details in Appendix A, eTable 4). The result for the primary outcome was positive in 54.0% (N=129) of RCTs, negative in 34.3% (N=82) and for 11.7% (N=28) of RCTs insufficient information was provided to make a conclusion. The cardiovascular system was the focus of 18.0% (N=43) of RCTs (Figure 2).
3.2Study documentation and standardised reporting
A published protocol was referred to in 27.3% (N=65) of RCTs. Of the 204 RCTs published after 1996, when the CONSORT statement was published, 8.3% (N=17) referred to a standardised reporting guideline. Of the 132 RCTs that were published after 2005, when the ICMJE mandated clinical trial registration, 81.8% (N=108) provided a trial registration number. It was stated that the trial had ethical approval in 92.9% (N=221) of RCTs. Figure 3 provides detail of reference to trial registration, protocol publication, reporting guidelines, and ethical approval, across years.
3.3Countries, centres and participants
Information on the number of countries that participants were recruited from was provided in 100% (N=239) of RCTs. Source of information on countries is in Appendix A, eBox 1. Across included RCTs, there were 90 different countries (excluding Ireland) that recruited participants (full details in Appendix A, eTable 5). The median number of countries other than Ireland was 4 (IQR:1-11). Ireland made up a median of 20% (IQR:8.3%-50%) of countries involved in RCTs. The United Kingdom was the country that was included in the most RCTs (81.2%, N=194), followed by Germany (39.3%, N=94). Figure 4 illustrates participation between the top 20 countries.
Information was provided on the total number of centres in 92.1% (N=220) of RCTs. The median number of centres was 33 (IQR:13-101). The number of Irish centres was reported in 75.4% (N=180) of RCTs. The median number of Irish centres was 1 (IQR:1-3). For RCTs that reported both the overall number of centres and the number of Irish centres, 74.9% (N=179) of RCTs, the median percentage of centres in Ireland was 6.0% (IQR:2.2%-15.4%). RCTs reported number of Irish centres (75.4%, N=180) more often than they reported list of investigators involved in the trial (68.2%, N=163).
All included RCTs provided information on the number of participants randomised; the median was 417 (IQR:223-1,004). Information was provided on the number of participants recruited in Ireland in 27.2% (N=65) of RCTs. The levels of reporting for countries, centres and participants is summarised in Figure 5. The median number of participants recruited in Ireland was 49 (IQR:12-79). The median percentage of participants that were recruited in Ireland was 5.3% (IQR:1.7%-17.1%). The distribution of the proportion of Irish centres and participants recruited in Ireland is in Appendix A, eFigure 1.
Studies that provided details of ethical approval, trial registration number, reporting guideline used or a published protocol had higher percentages of reporting number of Irish centres and number of participants recruited in Ireland (Table 4).
4.Discussion
This observational study of multi-country RCTs, focusing on those involving Ireland as a case study, identified 239 unique RCTs published between 1966-2019. Our findings suggest that both the number of multi-country RCTs and the reporting quality of multi-country RCT articles have increased steadily over time. However, deficits in reporting quality remain. Using our case study, Ireland, as an example, we found that reporting to quantify country involvement was very limited; only 27% of RCTs provided the number of participants recruited in Ireland and the number of Irish centres was reported in 75% of RCTs. Information on the total number of centres was provided in 92% of RCTs, comparable to an analysis of 82 RCTs published in five major journals in 2015 which found that 94% of RCTs stated the number of centres, suggesting the sample of RCTs in the present study is comparable to the wider literature [18]. Overall, this research is part of a growing focus on regional/country analyses of trial conduct and reporting [19, 20]. There were other important reporting issues: only 8% of RCTs published after 1996 (when the CONSORT reporting guideline was published) explicitly referred to a standardised reporting guideline, which is low given that most major journals require adherence to such guidelines. This is not improving over time; only 8% of the 13 RCTs published in 2018 referred to a reporting guideline. This compares unfavourably to the previous Clyne study of RCTs where 80% or more of participants were recruited from Irish centres (mainly RCTs led by investigators based in Ireland), in which 16% of RCTs referenced a reporting guideline [15].
Trial registration levels in the current study are relatively high; 82% of RCTs published after 2005 referred to a trial registry number. Also, trial registration improved over time, with 100% trial registration in several recent years. This compares well to the Clyne study, which found that registration only occurred for 32% of included RCTs [15]. A 2021 survey of 2,844 multicentre RCTs, found that that the registration number was provide in <50% of RCTs [8]. The Irish investigator-led RCTs compare poorly with regard to published protocols too, with only 10% of RCTs referring to a published protocol [15] compared to 27% in the current study. The 2021 survey of multicentre RCTs found that details of a full protocol were available in <50% of RCTs [8].
The RCTs examined in this study took place across 91 countries. A previous study identified 154 countries contributing to almost 40,000 RCTs, and the countries contributing most were similar to those found in our study of Irish RCTs (e.g. UK and Germany) [21]. In relation to funding, industry funded 55% of the included RCTs. A study of the most cited RCTs in the years 2019-2022 found that the majority of included RCTs were funded by industry [22]. This is concerning, as industry funded RCTs are more likely to report results and conclusions that favour the study’s intervention [23, 24].
4.1Recommendations and Future Research
Funders, institutions, regulators, industry and journal editors should ensure adequate governance measures are in place to ensure transparent reporting of RCTs. This could include institutions creating a policy of mandated results dissemination, and structure to support this, as well as investing in transparency guidance and training [25, 26]. Journals should consider interventions shown to be effective for improving reporting quality such as conducting an additional peer review where the focus is on reporting quality [27, 28]. Another initiative with potential to address issues outlined in this study is the development of a CONSORT Extension for multicentre RCTs [9] which will aid reporting of multicentre RCTs.
4.2Strengths and limitations
This study’s primary strength is that, along with the Clyne study [15], it provides a comprehensive overview of RCTs with centres based in Ireland. It is novel in being one of the first studies to assess the extent to which participation of centres/individuals from specific countries is quantified in trial publications. The primary limitation is that some relevant studies may not have been identified. However, many of these are likely due to poor reporting. Another limitation is that the years used to assess trial preregistration and standardised reporting may be overly strict, however we also present yearly rates (Figure 2). Also, the use of a reporting guideline, publication of a protocol or registration do not necessarily mean that these documents were completed comprehensively or adhered to [5]. However, the availability of these documents does facilitate the appraisal of the trial, though this is beyond the scope of our study. Also, some studies may adhere to a reporting guideline without explicitly referring to one. Other relevant limitations are outlined by Clyne and colleagues [15].
4.3Conclusion
This study of multi-country RCTs adds further evidence that reporting and registration of RCTs is sub-optimal, though it is improving. Quantifying the participation of countries in RCTs at centre or participant-level is limited by poor reporting in multi-country RCT publications. Further improvements could be potentially made through interventions by institutions, funders and journals.
Supporting information
Funding
No specific funding was obtained for this work; however, JL’s position was partly funded by a research grant from the Health Research Board in Ireland (grant ILP-HSR-2019-006), and for part of the study’s conduct FM’s position was funded by a research grant from the Health Research Board in Ireland (grant HRC/2014/01).
Competing interests
The authors declare that they have no competing interests.
Data availability
The relevant data set is available at https://doi.org/10.5281/zenodo.10653711.
Acknowledgements
We are thankful to Dr. Alan Barry for his assistance with record management.