Our analysis suggests that the majority of design methods used to design quality checklists were not collaborative in nature (e.g., involve the cooperation of multiple parties working together), with most methods only involving one researcher and one stakeholder (e.g., interviews, surveys/questionnaires). Studies that used collaborative methods such as Delphi consensus, consensus workshops, and focus groups noted some logistical challenges, such as drop-out observed across consensus meeting phases, virtual-only Delphi meetings hindering productive face-to-face discussion, funding constraints, lack of speaking up in mixed groups, and limitations on participant type and locations (38,44,51,52,65). This may suggest that studies favored more feasible design methods over more collaborative yet logistically-challenging ones.

As anticipated, a wide variety of different design methods were identified among the 29 studies, and several of the studies used multiple design methods in the development of the quality checklists. A third of the studies (n=10/29) started with a form of literature review before developing their quality checklist. This trend likely stems from the diverse existing literature on checklists that can be used to inform the design and development of new quality checklists. However, several studies noted that the quality checklists discussed were part of a quality improvement project in an organization, which may arise from a local need, local evidence base, or existing system process that needs improvement, thereby not prompting a desire for a full literature review to inform the project.

Twelve of the twenty-nine studies (41%) also included a form of interviews in order to gather insights from users and stakeholders prior to checklist development and gather feedback for checklist iterations. Interviews were predominantly described as semi-structured interviews and were done with one to two research team members. All studies used interviews in conjunction with another design method (e.g., observation sessions and interviews, interviews and focus groups). This was anticipated given the growing use of mixed methods to develop a comprehensive understanding of a topic (triangulation) particularly in qualitative research (82,83). Interview methods varied by channel (video, telephone, in person), length (between 15 minutes to 1 hour), and some described using a prepared interview guide. Notably, several studies described multiple rounds of interviews for different purposes – for example, as part of contextual inquiries for data collection and for value specification, between different checklist prototypes with different focuses, or different interview purposes depending on the type of stakeholder (43,48,49,54). Across the studies using interviews, a variety of primary and occasionally secondary stakeholders were included, in line with checklist development and implementation among high reliability organizations (16). Our findings suggest that while interviews, particularly semi-structured interviews, continue to be the most widely used data collection strategy, organizations adapt interviews as a design method to fit the purposes and needs in the development of quality checklists.

Simulated sessions or scenarios were also frequently used in the design, development, and evaluation of the quality checklist (n=9/29, 31%). These studies predominantly used more than one simulated case and varied by task, environment, situation, and fidelity; for example, one study ran usability testing with multiple, simple scenarios (50), while another used near-live simulation sessions to evaluate features of the checklist (56). Given the healthcare settings of these studies, simulated sessions provide an effective way of performing testing and iterative evaluation of a quality checklist without risking harm to stakeholders such as patients in an emergency care setting (57). Some studies acknowledged that a lack of objective observations in actual settings impacted the validity and limitations of the study (47,54,64). As the focus of this scoping review was on the design and evaluation and not the implementation of quality checklists, many included studies remained in the simulated scenario phase and did not address the transition from simulation to actual (e.g., clinic) settings, although they discussed the intent to apply the checklists in real-world settings in the future.

Other design methods used (Table 4) include 11 different methods that were used in only 1 of the 29 studies. Many of these methods include modeling methods that support conceptual designs or allow the research team to visualize users and their needs, such as personas, storyboarding, and process mapping. These methods were also used in conjunction with other, more commonly used design methods. This may be because the majority of studies had a clear understanding or direct participation of checklist users to qualify data and provide feedback that make these creative modeling methods noncompulsory for research teams. These results indicate opportunities in the design methods of quality checklists that need further exploration. For example, methods such as storyboarding and touchpoint videos have been used with interviews and focus groups to expand the contextual knowledge gathered in the design of electronic patient reported outcomes (84,85). Further research is needed to examine the feasibility and potential added benefit of modeling and creative design methods in conjunction with more established design methods such interviews, focus groups, and simulated sessions/scenarios.