The IPC EBOD-RAT, designed to measure IPC practices in HFs, underwent a validation process that demonstrated its feasibility and reliability. The median completion time per facility was 62 minutes, indicating its practicality in real-world settings. This tool underwent a robust validation process combining quantitative and qualitative methods. The process controlled for variance attributable to HFs while underscoring inter-rater reliability.

Few IPC-related tools have undergone rigorous validation processes. The WHO Hand Hygiene Self-Assessment Framework (WHO HHSAF), a validated tool assessing hand hygiene resources and practices, demonstrated reliability across 41 facilities in 16 countries and has tracked global progress since 2011. Similarly, the WHO Surgical Safety Checklist, developed using psychometric evaluation and a modified Delphi method, showed acceptable reliability across 110 patient cases in developed countries, based solely on quantitative methods [47, 48], unlike the mixed-method approach used for IPC EBOD-RAT. The WHO Infection Prevention and Control Assessment Framework (WHO IPCAF) was assessed using qualitative methods in Germany and Liberia before broader quantitative validation.[49] However, compared to two local IPC tools, WHO IPCAF required greater professional expertise to implement [50]. While some tools have been validated, many remain unevaluated for reliability or usability. The PROHIBIT study in Europe assessed hospital IPC via staffing, surveillance, education, and communication [51, 52], while the Netherlands’ Infection

Risk Scan (IRIS) visualized ward-level IPC indicators [53]. The CDC developed US-based tools addressing IPC infrastructure, hand hygiene, PPE, healthcare-associated infections, and competency gaps, with several national surveys supporting IPC monitoring [54, 55]. Additionally, qualitative interviews on the Catheter-Associated Urinary Tract Infections Guide to Patient Safety revealed differing healthcare worker perspectives but lacked quantitative reliability data [42].

In resource-limited settings, IPC assessments have primarily focused on vertical programs like tuberculosis control, incorporating cross-sectional surveys and diverse administrative, clinical practice, policy, infrastructure, and occupational health measures [19, 20, 24, 56-60]. Despite some alignment with WHO IPC guidelines, these tools lacked formal validation.

To further examine the reliability of the IPC EBOD-RAT, it is crucial to assess the statistical and methodological factors influencing inter-rater agreement. Fleiss’ Kappa, commonly used in structured health assessments, quantifies agreement among multiple raters while accounting for chance [44, 61]. However, certain checklist items or criteria can exhibit unexpectedly low kappa values, like the case of “Staff training” related criteria, necessitating a nuanced interpretation of reliability scores. The following discussion explores key statistical artifacts and methodological challenges that affect IPC EBOD-RAT assessments and provides insights into addressing these limitations.

The validation of the IPC EBOD-RAT identified some gaps and components with low reliability, especially when a criterion comprised multiple items that all needed to be met. This all-or-nothing principle could lead to low agreement among assessors and inconsistency in the assessment process. This issue was particularly prevalent in criteria related to staff training, isolation settings and equipment, PPE, and environmental cleaning and disinfection. Modifications improved the tool’s accuracy despite poor reliability in some indicators, as shown in past studies, [49] To address this, the study proposed potential solutions such as subdividing the criteria, employing a different scoring method that does not rely on the all-or-nothing principle, or reducing the number of items within each criterion,[43].

Ultimately, inter-rater reliability, Fleiss’ Kappa, for structured assessment tools is shaped by both statistical and methodological influences. One statistical challenge is the prevalence effect, where extreme distributions of responses skew agreement calculations. When an item is overwhelmingly rated in a single category, such as “absent” or “present”, the expected chance agreement increases, paradoxically reducing kappa despite high raw agreement [61, 62]. This prevalence paradox is evident in “Staff use one needle/syringe/intravenous cannula per patient for each injection” and PPE criteria, where nearly all observations matched [45, 46, 62]. Although actual rater agreement was high, kappa values remained low due to the inflated expected agreement. Similarly, bias in rater tendencies can distort kappa estimates, particularly when some assessors consistently apply higher ratings than others. For instance, if one observer marks an IPC practice or items as present “Yes” more frequently than another, disagreements may not stem from item ambiguity but rather they judge same situations differently. Such variation affects marginal probabilities, causing kappa to underestimate or, in some cases, overestimate actual consensus [63, 64]. With Fleiss’ Kappa (unweighted) for multiple raters, any difference among raters on a subject counts as disagreement. If an item has subtle gradations (for example, rating “Facility has required PPE to manage case” like using all-or-nothing principle as mentioned above), Fleiss’ Kappa treats any nonidentical ratings as discordant (“Gloves,” “Masks,” “Apron,” “Gown or coverall,” and “Eye protection”). This can be misleading in cases where differences are minor. An item might have mostly “moderate” vs “high” ratings given (adjacent categories), indicating some consistency, but kappa will penalize those as full disagreements. Moreover, with many raters, if even one out of five observers gives a slightly different score, the item is marked as not in complete agreement for that subject. As a result, kappa tends to decrease as the number of raters or categories increases, given the greater opportunity for any divergence. A “fair” Fleiss’ Kappa might hide the fact that, say, 2 out of 3 raters agreed perfectly and only one differed by a small margin [64-68].

Low variability in ratings also impacts Fleiss’ Kappa, making agreement difficult to quantify. When an item exhibits minimal variation, such as “Staff discard needles, syringes, and intravenous waste into appropriate biohazardous waste streams”, “Water Supply and Storage”, “Waste segregation”, or “Waste elimination” criteria, which were consistently rated identically, the absence of differentiation among raters leads to disproportionately lower kappa values. Decision-makers or tool developers might incorrectly conclude that an item lacks reliability or that raters are inconsistent, even though they are actually in strong agreement. This can undermine confidence in the tool or lead to unnecessary revisions. Items like “Waste segregation” or “Water supply” may be inherently clear and consistently implemented, leading to unanimous ratings. Low kappa values in such cases can mislead evaluators into thinking these items are problematic when, in fact, they may be well-understood and universally applied, potentially wasting resources on well-functioning items instead of addressing truly ambiguous or inconsistent ones. This is because kappa adjusts for chance agreement, which becomes inflated when most raters choose the same category.

On a positive note, the IPC EBOD-RAT was designed to rely mostly on “Yes” or “No” dichotomous alternatives, which alleviate the ambiguity inherent to complex question types like multiple choice, ranking, or Likert scale, etc. Even when the overall agreement is high, kappa cannot register meaningful variation, making it a suboptimal measure in those cases. Furthermore, the complexity of rating scales affects inter-rater reliability; a greater number of categories increases the likelihood of disagreement, which penalizes kappa calculations [69].

Similarly, the small sample size and the number of raters contribute to instability in kappa estimates. The study used the minimal required sample size, which bore inherent limitations. “Inpatient surveillance and management” thematic instances the issue, where a small number of HF responded to the question, resulting in wide confidence intervals and fluctuations in reliability measurements, making it “difficult to establish inter-rater reliability due to limited data. Conversely, with many raters, the chance of any disagreement increases. Thirteen raters participated in the 51 assessments; three were randomly selected per HF. Fleiss’ Kappa aggregates agreement across all raters, so an item may get penalized if even one or two raters differ from the others on a few cases [44, 61, 69, 70]. These statistical considerations mean that kappa should be interpreted in the context of sample size and category prevalence – a low kappa alone does not always imply the criterion is unreliable without examining these factors. The intra-rater agreement was not assessed during the study due to the outbreak context, as mentioned in the limitations.

Beyond statistical factors, methodological and contextual factors also reduce inter-rater reliability, as emanating from FGD. Ambiguous or subjectively worded items or translations present a significant challenge, as raters may interpret descriptions differently [71, 72]. For example, some criteria contained terms such as “Autoclave in use”, “necessary material for cleaning”, or “functional hand hygiene station,” which lacked concrete operational definitions. Without precise criteria, one rater might classify a scenario as compliant while another deems it inadequate, reducing agreement, as observed and highlighted during FGD. Standardization of language and criteria is essential to prevent discrepancies and improve reliability [71-73].

Inconsistent rater training and protocol application further contribute to variability in assessments [72] and played a role during the tool deployment in the context of the Ugandan active SVD outbreak response [74]. FGD suggested how variations in professional background, experience, and adherence to protocol affected scoring [74]. A more experienced observer might identify subtleties that others miss, while untrained raters might misapply evaluation criteria. This issue is particularly pronounced in emergency health assessments, where observers may range from clinicians to field epidemiologists, each emphasizing different aspects of IPC practices. Without standardized training sessions and reference assessments, Fleiss’ Kappa suffers from inconsistent rater expertise rather than true item unreliability.

Observer subjectivity and human factors play a role in inter-rater disagreement, particularly in outbreak settings where assessments are conducted under stressful conditions. Items M, N, and O revealed how fatigue, stress, and perceptual bias influenced reliability. Additionally, contextual challenges unique to outbreak environments complicate structured assessments [19, 20, 75], for which chaotic settings, rapid workflows, and limited visibility hinder synchronized observations.

These findings emphasize the necessity of considering external factors when interpreting Fleiss’ Kappa. The IPC EBOD-RAT validation revealed that low kappa values may not always indicate poor reliability but rather methodological constraints inherent to outbreak assessments. Future studies should explore alternate scoring methods, refined assessment criteria, and enhanced training protocols to improve inter-rater consistency in IPC evaluations. Of note, one component of the IPC EBOD-RAT, “Inpatient surveillance and management,” was not validated by the assessors due to the timing of the validation process, not the data quality, as there were no further cases of Sudan virus at the time of the study. Other components that involved observing the infrastructure had moderate or higher agreements among the assessors.