This study demonstrated that adaptive eLearning can increase reach to providers not reached by conventional training, and the use of in-person support and an escalating nudging strategy increased initial learning completion. Further, refresher assignment completion was low, and our implementation strategy needs to be revised to solidify knowledge gained through initial learning. Finally, provider awareness of knowledge gaps was low at baseline and lower baseline knowledge is associated with decreased completion or learning activity.

aESNC reached 85% of providers in this study. The critical mass needed for collective behavior change for providers in facilities is unknown, but using estimates from current social behavioral science suggest that 10-40% of providers are needed to be engaged to drive behavior change.^57–59 As knowledge is only one component that drives behavior change and we know not all providers would complete all modules, we will examine thresholds for reach required to achieve sustained changes in quality of care.

aESNC was able to reach providers that conventional newborn training had not reached. We found that 30% of providers had received Essential Newborn Care training, 40% had received Helping Babies Breathe, but only 23% had received both. The impact of this limited reach may be underestimated if this reported training was years ago. A reason for conventional education’s limited reach may be due to high turnover of front-line providers at academic centers, which may be able sustain training but not sufficient to address its training gaps. Furthermore, community health centers may have limited support to hold local training or send staff for training to meet this need. This conclusion from this initial study should be tempered as our observed reach, as well as our enrollment rates, were higher than anticipated, and don’t correlate with staffing estimates reported from health system leadership (Table 1). This may be due to a combination of incomplete lists by facility leadership and early adopter phenomenon. Future studies will bear this out.

Our initial efficacy of 41% was consistent with our previous conventional studies that demonstrated knowledge efficacy rates of 10-25% after pediatric acute care training using current training methods^18–,21, as well as other newborn and pediatric conventional in-service education studies.^{5, 22} It is important to note that our use of formative assessments at baseline and during the educational session, are relatively new measures of educational efficacy. In our pilot, the difference between latest conscious competence and baseline conscious competence was roughly 10 points higher than between pre-post testing.⁴⁹

The average progress of providers for initial learning was 78%, and 56% of providers completed all initial learning assignments, higher than we observed in our initial pilot of PACE. This may be due to the use of in-person support and an escalating nudging strategy increased initial learning completion. The completion rate in this study is comparable to previous studies in high income countries: much higher than massive open online courses progress rates of 7-15%,^{60, 61} and comparable to longitudinal e-learning studies of healthcare providers (44-50%).^{62, 63} A few shorter courses (< 3 months) with more in-person support report completion rates as high as 75-80% without incentives,^{64, 65} and as high at 70-96% completion when incentive of $200-$750 was provided.^{66, 67}

In this study, we observed more time is needed for providers to complete initial learning than anticipated. We had estimated providers needing 5 hours over 1 month to complete initial learning. While median time to complete initial learning was shorter than what we targeted (4.5 hours), the median days to completion of initial learning was 61 days, twice what we had estimated. Interestingly, the retention curve demonstrated no significant drop-off that had been seen previously. Inadequate time for education, not technical issues or lack of internet access was the main barrier.

Other reasons, besides the in-person support and escalating nudging strategy, may have contributed to these findings. We did not use a designated order determined by subject matter experts as we did in our initial pilot, as we observed a significant drop off with disability and exposure assessment. This choice of module completion may have allowed better alignment of provider learning interest and increased engagement during initial usage, and familiarity with adaptive learning may have facilitated usage of completion of additional content. Additionally, there may have been increased peer support for adaptive learning due to a higher percentage of providers participating in PACE at each facility. Whether this improved average completion is due to our implementation strategy components (escalating nudging strategy, program manager, reports to facility), changes to PACE itself (self-selection of topics), or environmental (unmeasured peer support) is unclear.

Our implementation strategy needs further revision to increase refresher assignment completion. Less than a third of participants participated in refresher assignments, and the average progress through refresher assignments was <10%. Improving established provider retention and completion of refresher assignments is key to optimizing education effectiveness. In-service provider education is designed to build upon foundational knowledge gained in pre-professional training, as well as solidify it through relevance to a provider’s clinical experience. Unfortunately, many continuing professional education programs are short courses without longitudinal follow-up. The use of in-person skills training, tailored to provider needs using peers as “clinical champions”, may enhance refresher assignment completion. These clinical champions may add a point of engagement and facilitate normalization of adaptive eLearning use to facilitate education over time. Additionally, as our nudging strategy was triggered only to complete all initial learning assignments, providers may believe they have no more learning to do. Further refinements with our nudging strategy and sensitization meetings with facility providers and leadership to emphasize importance of refresher training may also help normalize adaptive e-learning for providers.

Adverse events occur in 18% of inpatient admission in Africa and continuous learning is needed to improve patient safety.^{68, 69} Although baseline knowledge of guidelines was consistent with previous studies, we identified that providers are often unaware of knowledge gaps. In addition to demonstrating that providers know about half of newborn knowledge at baseline, adaptive e-learning allows us to see providers’ awareness of their knowledge and misconceptions. Unconscious incompetence is a latent threat to health systems, and as they are not realized, may mask workforce knowledge gaps. Unconscious incompetence is a particular danger clinically, as providers are confident but incorrect in what care they believe should be delivered.

While aESNC increased provider knowledge, our exploratory analysis identified that baseline lower knowledge may be associated with inactivity or non-completion of aESNC. While one of the strengths of adaptive learning is using formative assessments to facilitate learning, providers whose baseline knowledge is too low may suffer cognitive overload. Our level of unconscious incompetence surpassed Area9’s best practice guidelines, which suggest keeping it between 20 and 30%. It was also higher than what we observed in our initial pilot. Development of a strategy of focused support to engage with those at risk may be needed.