Model scale cost
hosted-large minus local
- Precision changes by -3.1 percentage points.
- Recall changes by +32.7 percentage points and F1 by +24.8 percentage points.
- Cost-weighted error changes by +1.55 per thread.
Results
Measured dev split results for hosted-large and local extractor configs at IoU 0.5.
Results cover the dev split only, with two configurations reported: hosted large and local. A single prompt version was used with no iteration against dev results. The held out test split has not been run. The hosted redacted run was attempted and did not complete.
Attempted, Incomplete
80 threads attempted, 67 provider failures, 70 parse failures, 10 threads with parsed loops. Run abandoned to free-tier rate limits, to be re-run.
An open loop is an outstanding commitment between two people: something one person owes the other, agreed to in passing in the middle of a conversation and filed nowhere. Finding the promise is the easy half. The hard half is deciding whether it is still live, because a commitment that was cancelled, delegated, or overtaken four messages later reads exactly like one that still stands. The promise text does not change when the obligation dies; only the rest of the thread says otherwise. An extractor that reads the commitment and stops there cannot tell a live loop from a dead one, and it will report both with the same confidence.
Two configurations are reported below. Hosted large is a 70B model reached over a hosted API, so every thread leaves the user's device to be read. Local is a 7B model running on the user's own hardware, where nothing leaves. The gap between those two rows is what a privacy-first deployment costs today, and it is not a uniform discount. The local run is marginally the more precise of the two on the loops it does report, but it recovers about a third as many of them, gets the direction wrong in more than half the pairs it matches, and reports three of every four dead commitments as still live. Reading the columns rather than the averages is the point: keeping the data on the device does not degrade performance evenly, it removes the specific capabilities that make the output safe to act on.
| Config | Loop precision | Loop recall | Superseded as open | Direction accuracy | Span tightness | Deadline quote found | Resolved date exact |
|---|---|---|---|---|---|---|---|
| Hosted largellama-3.3-70b-versatile | 51.5% | 49.5% | 14.3% | 88.7% | 0.85 | 72.7% | 8.6% |
| Localqwen2.5:7b | 54.5% | 16.8% | 75.0% | 44.4% | 0.94 | 25.0% | 0.0% |
Headline Number
14.3%
The hosted-large run reports 2 of 14 matched superseded commitments as still open.
In plain terms: roughly one in seven dead commitments is reported as live. The user is handed a task that no longer exists and goes chasing it, which usually means a message to a counterparty who has already moved on. It is measured on matched pairs only, so it is not softened by the loops the model never found at all.
On the hosted large run the model finds the deadline phrase in the text 72.7% of the time and resolves it to a calendar date 8.6% of the time.
The failure concentrates in the code-mixed half of the corpus, where deadlines are not written as dates. Phrases like kal tak, parso, naaliki and weekend tak are ordinary usage that names a day without naming a date. The model quotes them back accurately and resolves almost none of them.
This matters because a proactive system has to rank. Its whole job is deciding what to surface now and what can wait until Thursday, and that ordering is by urgency, which in practice means by date. A phrase cannot be ranked. Two loops carrying kal tak and weekend tak cannot be ordered against each other, or against a loop due on the 12th, so the model has located the deadline and then discarded the one property that would let anything be done with it.
Read on its own, the quote rate looks like competence at deadline handling. The distance between the two numbers is the distance between reading a thread and acting on one, which is why deadline quote found and resolved date exact are reported as separate columns rather than as a single deadline score.
hosted-large minus local
| Config | Cost-weighted error | Run description |
|---|---|---|
| Hosted large | 5.04 | Groq llama-3.3-70b-versatile on full text. |
| Local | 3.49 | Local qwen2.5:7b run on full text. |
| Error | Weight |
|---|---|
| false_positive.blocked_on_you | 3 |
| false_positive.blocked_on_them | 8 |
| false_positive.mutual | 3 |
| false_negative | 1 |
| superseded_as_open | 8 |
| direction_inverted | 8 |