Knows When to Stop
AI Research
90 synthetic healthcare administrative cases that test whether an agent finishes the work when it can and stops to escalate when it can't.
Key features
- 90 cases across 30 workflow families, ten each for prior authorization, appeals, and equipment
- Each family has a case to finish, a twin with one decisive field changed that should be stopped, and a finishable case with a harmless warning
- The dataset and a pre-run commitment were published before any of the four model runs
- 360 episodes over four models, with every run's evidence in the repo
- The strongest model passed 90 of 90 tasks and handed off 29 of 30 stops fully correctly. The weakest passed 75 and handed off 14
- The evaluator uses only the Python standard library
An agent doing administrative work has two ways to fail. It can stop on a case it could have finished, or it can push through a case it should have handed to a person. This dataset measures both.
How the cases are built
There are 30 workflow families in prior authorization, appeals, and medical equipment. Each family has three cases. One can be finished. One is its twin with a single field changed so that finishing would be wrong. One can be finished but carries an old or unrelated warning, to catch an agent that stops at anything that looks alarming.
The three cases in a family are related, so 90 cases are not 90 independent problems. The write-up says so.
What a good stop looks like
Stopping isn’t enough. A fully correct handoff stops in time, gives the right reason and urgency, and cites the evidence. Scores for that are reported apart from the plain task score.
I built it for Grayhaven. The dataset is on Hugging Face under Apache 2.0, and the source and run evidence are on GitHub.