Background: Registered nurses are the core communicators of patient status within inpatient settings, channeling and receiving critical information across complex care teams. Secure messaging through electronic health records (EHR) allows nurses to communicate directly with care team members via instant messaging without external tools such as pagers. However, its growing popularity may increase clinician cognitive burden from interruptions, multitasking, and generally higher volumes of messages. Artificial intelligence (AI) tools may help in the prioritization of urgent messages as well as provide a way to identify categories of free-text messages sent within an organization.

Methods: In September 2023, initial messages sent from nurses to providers via secure chat were extracted and de-identified. Each message was reviewed and labeled by pairs of physicians and pairs of nurses based on their urgency. The first round of review contained 150 messages that were labeled based on the desired response time for that message (<1min: emergent, <5min: urgent, >5min: non-urgent). Disagreement within pairs was resolved through group consensus, with the more urgent label chosen if discordance between doctor and nurse pairs existed. The labels were used to generate a prompt for GPT4 and themes for each message class. Model performance was refined through iterative rounds of human labeling and its final prompt was used to update the themes found within each message type.

Results: In round one, doctor-nurse agreement was 70%. After the first round of labeling, a prompt was developed to use ChatGPT to label messages. When this prompt was tested on the first round of human labels, it initially achieved 90% accuracy on emergent messages. After human voting on discordant labels, accuracy increased to 96% (17/42 human labels were overturned in favor of GPT4). The model was re-tested after prompt revision, resulting in 98% accuracy labeling emergent messages from the original group. The updated model classified a new set of 400 human-labeled messages on a separate testing set, achieving 93% accuracy on labeling emergent messages. Additional analysis phases are underway.

Discussion: Nurses and physicians agreed on the urgency of messages for a majority of the instances of review. As the GPT4 model was tuned, its ability to label emergent messages similarly to clinicians improved. By learning to classify messages as emergent, urgent, and non-urgent, future technologies could help clinical teams in a myriad of ways. Secure message analysis may provide clinical informaticists valuable information on EHR tool utilization and highlight improvement areas. It may aid nursing leaders and bedside staff in the reinforcement of appropriate communication methods for various clinical issues. Future research could explore real-time AI analysis of secure messages to suggest priority routing or alternative communication methods like phone calls.

Conclusion: This project demonstrated that GPT4 can effectively classify secure message urgency comparable to human clinicians. The use of AI categorization in secure chat analysis could help discover patterns of usage, areas for improvement, and training opportunities. Future research may establish ways to improve communication efficiency and reduce cognitive burden on healthcare providers.

Lisa L. Groom PhD, RN

Appearances