Supporting the quality of SafeUT: The evaluation of text-based crisis counseling with machine learning.

Suicide is a leading cause of death among people ages 10-34. The suicide rate in Utah is higher than the national average, with 600 deaths and 4500 attempts each year. SafeUT is an app that provides 24/7 access to text-based crisis support from trained counselors to 734,587 students (81% of all Utah K-12 schools and Universities). With over 20,000 crisis chats on SafeUT in the last year, their volume places a burden on counselors, their supervisors, and systems to train and support the quality of crisis chats; the scale of unstructured text from crisis chats means that it is not feasible to evaluate most encounters. Supervisors must make arbitrary or reactive decisions in regards to what chats to review. Thus, scalable systems to support quality text-based interventions are essential to the impact of SafeUT and other similar text-based counseling platforms.

This proposal will support the creation of a multidisciplinary team to evaluate machine learning tools that assess the quality of text-mediated crisis counseling in real-time.

AIM 1: Use the text of crisis chats to build natural language processing (NLP) models that can predict continued user engagement (i.e. if a user responds to counselor messages).

AIM 2: Utilize human ratings of chat quality to train and evaluate the performance of NLP models that automatically rate counselor messages, and highlight specific interactions where the counselor could benefit from further consultation.

Current Status

2021-09-15
Abstract:
Suicide is a leading cause of death among people ages 10-34. The suicide rate in Utah is higher than the
national average, with 600 deaths and 4500 attempts each year. SafeUT is an app that provides 24/7 access
to text-based crisis support from trained counselors to 734,587 students. With over 20,000 crisis chats on
SafeUT in the last year, the scale of unstructured text from crisis chats means that it is not feasible to evaluate
most encounters. This 1U4U project supported the creation of a multidisciplinary team to evaluate machine
learning tools that assess the quality of text-mediated crisis counseling in real-time. We trained a human
labeling team that evaluated over 400 chats for both texter suicide risk and counselor assessment of risk. We
used these ratings to train initial machine learning models that can automatically evaluate texter risk and
counselor assessment of risk. Based on these results we submitted a NIMH R34 grant that was scored and are
preparing a revision. In addition, we submitted a presentation to the Association of Behavioral and Cognitive
Therapies that was accepted and will be presented this Fall.