# Reflect-a-bot

### Therapist training chatbot to encourage reflection

#### Hobson Lane & Taylor Kirk

#### **Sponsored by:** Job Crowder

#### **Based on:**

1. "PAIR:...Counselor Reflection Scoring..." Min et.al.[1](#links)
2. "Building a Motivational Interviewing Dataset" Pérez-Rosas et.al.[2](#links)

---

## Agenda

1. Purpose
2. Theory of impact
3. Measure progress
4. Measure the measurer
5. Data
6. Architecture
7. Data science process
4. Preliminary results
5. Arcagent (monkeypatched iPython)

---

## Purpose

Improve quality of counseling sessions

#### Improve counselor training
- Professors are overworked
- Students do not get enough practice
- Practice is rote memorization?

---

## Theory of impact

Better training == better counselors

#### Automation
- Reduce instructor workload
- Allow students to practice more
- Motivate students to practice more
- Create more diverse practice sessions

---

## Metric

Scientific method

#### Summative assessments

- Before/after case studies
- Data **Science**

---

## Real world

Product design

#### Formative assessments

- Practice in a simulation: [Runestone.Academy](Runestone.Academy)
- Spaced repetition: [Anki](https://app.ankiweb.net)
- Make it fun! **Gamification**: [LibreLingo](https://librelingo.app/)

---

## Reflect-a-bot

Identify a positive class

- Good: **Reflection**
- Bad: **Complex** reflection (TMI)
- Bad: **Other** (everything else)

---

## Grade the grader

<a alt="Wooden rulers side by side with a metal ruler in between. Metal ruler gives a sixteenth inch shorter measurement after 4 inches of length (2%)" href="rulers.png"><img src="rulers.png" height="300" /></a>
 by Marcin Wichary on flickr.com

#### Model evaluation

- Accuracy
- Recall (False Negative Rate)
- Precision (False Positive Rate)
- F1-Score
- AUC - "Area Under the Curve"

---

## Academic data

7141 utterance pairs: counselor prompt -> client response

| Category  | Count |
|-----------|-------|
| Complex.0 | 993   |
| Complex.1 | 318   |
| Other     | 1077  |
| Simple    | 4761  |

---

## Real world data

---

## NLP Architecture

---

## Reflect-a-bot

---

### Links

1. ["PAIR: ... Counselor Reflection Scoring in Motivational Interviewing", Min et. al.](https://aclanthology.org/2022.emnlp-main.11.pdf)
2. ["Building a Motivational Interviewing Dataset.pdf", Pérez-Rosas et. al.](https://aclanthology.org/W16-0305.pdf)

----