Explaining Machine Learning Clinical Decision Support: Influence of Nurse Numeracy and Graphical Literacy on Nurse Satisfaction with Machine Learning Clinical Decision Support Explanations

Purpose: To examine if nurse numeracy and graphical literacy influence the strength of associations between machine learning (ML) clinical decision support (CDS) explanatory information display strategies and nurse satisfaction with ML CDS explanatory information.

Background/significance: ML is emerging as a promising technology to drive improved CDS systems and provide insights to improve patient care and safety. One challenge of leveraging ML CDS in clinical practice is explaining to clinicians how complex ML algorithms produce a given output. This explainability challenge is further compounded by the variable competencies of clinicians to interpret data. Previous research has demonstrated that levels of numeracy and graphical literacy vary among nurses and this variation impacts nurse effective use of CDS. Determining whether nurse numeracy and graphical literacy impact nurse satisfaction with ML CDS explanatory information could inform design improvements and foster more successful adoption of these tools into nurse practice settings.

Methods: A cross-sectional study was conducted using a web-based factorial survey among registered nurses working in a midwestern health system. Participants were presented with one of two clinical scenarios and randomized to different ML CDS explanatory display formats (tabular vs. graphical). Participant satisfaction with the explanatory information was measured using the explanation satisfaction scale. Participant numeracy was measured via the subjective numeracy scale (SNS) and graphical literacy via the long graph literacy scale (GLS). Participant numeracy and graphical literacy were dichotomized into low and high groups so that the median value and any values higher than the median were included in the high category. Multiple linear regression models and margins analysis were employed to assess the associations between variables as well as interaction effects.

Results: The sample consisted of 223 nurses, predominantly female (88.8%), age less than 39 (70.4%), bachelor’s educated (81.6%) with an average of 10 years (ranged 0-43 years) of nursing experience. Most participants had either high numeracy (56.1%) or high graph literacy (70.9%) and almost half had both high numeracy and high graph literacy (43.9%). The results suggested that nurse numeracy significantly moderated (β = 0.430, p=0.05) the relationship between explanatory display format and nurse satisfaction with explanatory displays, with participants in the low numeracy category reporting higher satisfaction with tabular display formats. The interaction between graphical literacy and display format did not significantly impact nurse satisfaction with explanatory displays in our study.

Conclusions/implications: These findings highlight the importance of considering nurse characteristics, such as numeracy, when designing explanatory displays for ML CDS. Tailoring the presentation format to nurses' numeracy and graphical literacy levels may enhance their satisfaction of the system's output, fostering adoption. Further research is needed to explore additional factors that may influence nurse satisfaction, as well as implications for adoption and use. This study provides valuable insights into the associations between nurse characteristics and satisfaction with explanatory information for ML CDS, contributing to knowledge to inform improvements in the design of ML CDS.