Averta

The question

What this set out to test, and what it found.

A coding agent either resolves its task or it doesn't. A session heading nowhere keeps spending tokens that produce nothing, so detecting that early is worth doing. But stopping a session that would have recovered is worse than letting it run. That asymmetry is the whole problem: the value is in catching doomed sessions, the cost is in killing salvageable ones.

Fail-Fast, Restart-Smart (Wang et al., 2026) approaches this with a 0.6B neural monitor. Averta asks whether engineered trajectory features and a cheap classifier can do useful work at a fraction of that inference cost.

Every number on this page was measured before it was written. Criteria were committed to source control before any model existed, so they could not be relaxed to fit a disappointing result.

Data

One corpus, because only one had labels.

Training data comes from SWE-Gym/OpenHands-Sampled-Trajectories. A survey of six public trajectory corpora found it to be the only one carrying both outcome classes, the others are supervised fine-tuning sets that store the agent's patch but never whether it worked.

Two properties shape everything downstream. The positive class is rare, so metrics that tolerate imbalance are required. And session length barely separates the classes: resolved sessions average 39.86 turns against 39.66 for unresolved. Prior work describes failing agent runs as tending to run longer; that does not reproduce here.

The source carries no declared license, so this project does not redistribute it. Raw trajectories are downloaded locally and excluded from version control; only derived statistics, figures and model artifacts are published.

Method

Three constraints, each guarding against a specific way of fooling yourself.

Prefix-only features

A feature computed at turn t reads turns[0:t] and nothing else, never the total length, never the outcome. A test suite shuffles, truncates and extends the unseen tail and asserts the feature vector is byte-identical, and checks that features at turn 6 are the same whether the session runs to 12 turns or 200.

Absolute turn indices

Evaluating at “40% through the session” requires knowing the total length, which is unavailable while a session is running. Evaluation uses turns 3, 5, 10, 20 and 40. Each cut has a different population, since only sessions that reached that turn appear, and the resolve rate is reported per cut so the shift stays visible.

Repository-grouped splits

Every row from a repository lands in one fold. Rows sharing a codebase are not independent, so splitting by row would let a model memorise a repository and be rewarded for it. Confidence intervals resample repositories rather than rows for the same reason, row-level bootstrap reports intervals that are too narrow under clustering.

Polarity

The positive class is failure, so the false positive rate means “sessions that would have resolved but were flagged” - the quantity worth constraining. Average precision is reported on the minority class instead, with its polarity named. Accuracy is never reported: at a failure rate of 88.79%, always predicting failure scores 0.89 and is useless.

Results

Turn 10 · 4,381 sessions · 88.8% failure rate · 5 repository-grouped folds · out-of-fold predictions pooled

Against the pre-registered criteria

The shortfall is robust: it holds at every cut point tested and under any model-selection rule. The best recall at a 5% false-positive budget anywhere on the curve is 0.191.

Two attempts

Both reported. Identical criteria for each.

Attempt 1 used 23 aggregate counts over the prefix. AUROC 0.668, recall 0.155. Permutation importance showed one feature dominating, n_repeated_calls, a flat count of tool calls reissued with byte-identical arguments, while every error-based feature was negligible.

That pointed at ordering as the missing ingredient. A count knows an action recurred; it cannot express that the agent is cycling A-B-A-B, or how far back it reached to repeat itself.

Attempt 2 added 10 sequence-structure features. Recall rose from 0.155 to 0.191 and AUROC from 0.668 to 0.677, real movement on exactly the failing criterion, but not enough to clear it.

n_repeated_calls collapsed once ordering was represented properly, it had been a proxy for structure it could not express. The two strongest features now both measure declining action novelty.

Modelling stopped after two attempts. The gains were real but shrinking, the mechanism is understood, and a third round of feature invention against a bar that had already failed twice would be threshold-shopping under another name.

What drives it

The expectation was wrong, and measurement is what corrected it.

The prior expectation was that a recurring error signature would dominate. It is close to worthless in isolation, AUROC 0.521 alone. Failure is predicted by declining action novelty: the share of actions that are distinct, and the share of recent actions never issued before. A session heading nowhere is one that has stopped trying new things, which is a different phenomenon from one that keeps hitting the same wall.

Inference cost

Single-row latency, because the monitor scores one live session per turn.

The most accurate model is also the smallest and fastest by a wide margin. Plausible cause: with 33 features, 491 positives and repository-grouped evaluation, trees fit repository-specific thresholds that do not survive the group boundary.

Calibration

A bug the plot caught, kept visible because the before-curve is the lesson.

Every model is fitted with class weighting, which is correct for ranking but leaves the output on a re-balanced scale. A raw score of 0.35 corresponded to an observed failure rate near 0.82. Rank-based metrics are unaffected, but any number shown to a person has to mean what it says.

Isotonic regression fitted on out-of-fold scores corrects it, moving the Brier score from 0.2268 to 0.0948. Fitting the calibrator on training predictions would have learned the model's own overconfidence and reported it back as calibrated.

Savings against harm

What intervention buys, and what it costs.

At a comparable budget, 4.7% against the paper's 5% target, this reaches 8.4% estimated token savings where the 0.6B neural monitor reports 14.6–20.4%. Roughly half the value, at a fraction of the size.

Tokens are estimated from content length; the corpus records no token counts. Savings count only what would have been spent after the cut. Sessions wrongly terminated are never netted off the savings, the two are not commensurable and a single “net” figure would let the harm disappear into an aggregate.

Cross-scaffold transfer

Planned as an AUROC comparison. Not reported, the sample cannot support one.

The plan was to train on OpenHands trajectories and test on hand-labelled Claude Code sessions. Only 3 local sessions are long enough and none carry outcome labels, so a transfer AUROC would be noise presented as a result.

What is measurable without labels is whether the features compute comparably at all, a prerequisite for transfer rather than a substitute for measuring it.

The 6 largest shifts all exceed 1.7 standard deviations, and 2 of 33 features fall outside the training range entirely. Claude Code emits more assistant turns per tool call than OpenHands, because thinking blocks become their own turns, a “turn” is not the same unit across scaffolds. Cross-scaffold accuracy is therefore unmeasured, and the tool labels its output indicative in every code path.

This check caught a real bug. Two features reported exactly 0.00 on sessions full of edits, because edit detection was keyed to one tool vocabulary. Not a slightly worse score, a confident zero for something that happened dozens of times.

Limitations

Stated plainly rather than buried.

• The operating point is not reachable here. Two attempts, the second targeting the diagnosed gap, both fell short. Closing it likely needs a genuine sequence model over the action stream rather than more engineered summaries of it.
• One corpus, one agent scaffold, 11 repositories. Grouped folds mean 11 grouping units, which widens every interval.
• Token savings are estimated, not counted. The corpus records no per-message token usage.
• Cross-scaffold accuracy is unmeasured. The feature comparison above suggests it would be poor.
• The comparison to the paper is not controlled. Different corpus, different scaffold, estimated rather than counted tokens. It is the closest like-for-like available, not a replication.

Using it

Runs entirely locally. No API keys, no network at inference time, no session data leaves the machine.

git clone <repo> && cd averta
python3 -m venv .venv
.venv/bin/python -m pip install -e .

averta explain      # analyse your most recent coding session
averta sessions     # list local sessions
averta site         # rebuild this page

Those work immediately, the trained model is committed. Reproducing the study needs the corpus, which is a 12-minute download:

averta ingest && averta features
averta train        # cross-validate and apply the gate
averta diagnose     # importance and inference cost
averta savings      # savings against harm
averta figures      # the plots on this page

averta explain separates what is measured, recurring errors, reissued calls, clustered repetition, exact token counts, from what is estimated by a model that did not clear its gate. The measured half needs no model and is as reliable as the transcript. An MCP server exposes the same analysis to a coding agent over stdio.

macOS users need brew install libomp for xgboost. On Linux, libgomp1.