Typed Decisions

A benchmark for typed probabilistic decisions. A model gets one piece of unstructured state and answers five typed questions about it at once, and every answer is a probability distribution, not a single label.

The schema follows the System One primitives (noul, choice, score) used by TypeSafe AI, so a row replays against any API with that shape. The benchmark is independent: it is not affiliated with TypeSafe and does not reproduce their Jev model.

Leaderboard

test split: 400 cases, 2,000 decisions. This table lists general models scored zero-shot: they have never seen these workflows or their question schemas. Models that were fitted or fine-tuned on train are in the second table below; the two tables are not comparable.

#	Model	Kind	Accuracy ↑	KL from gold ↓	Brier ↓	ECE ↓	p50 latency	Price / 1M input
1	meraGPT Decider 1 (sd-1)	general, zero-shot	0.768	0.096	0.052	0.180	526 ms	$0.03
2	Amos 1 medium (9B) § (#31)	general, zero-shot, open weights	0.7645	0.151	0.080	0.103	118 ms‡ (H100)	open weights
3	Amos 1 large (27B) § (#31)	general, zero-shot, open weights	0.761	0.126	0.069	0.097	357 ms‡ (H100)	open weights
4	pplx-decider-v1.1-27b (27B) ◆ (#37)	general, zero-shot, open weights	0.753	0.441	0.160	0.060	–	open weights
5	d3 (27B) ◆ (#37)	general, zero-shot, open weights	0.749	0.409	0.146	0.049	–	open weights
6	Darwin-27B-ZTC (27B) § (#17)	general, zero-shot, open weights	0.743	0.204	0.097	–	–

Every row except those marked ♠ was scored by sending the whole case (state plus all five questions) in one request. Request shape matters: in a third-party run Jev's yes/no accuracy was 0.843 with one question per request and 0.788 alongside the others.

Hosted rows are p50 end to end from a client, one request at a time. † Measured on the same machine as the model (M3 Max), not comparable with hosted latency. ‡ Reported by the submitter on their own hardware (a different GPU and a different unit for each), not comparable with the other rows. § Self-reported by the submitter in the linked discussion and not re-run by us. Metric definitions follow the submitter's report; ECE in particular is computed differently by different submitters. ◆ Scored by a third party (SaylorTwift, discussion #37) on Hugging Face Jobs, 2026-10-16 to 10-19, with one public scorer for every row and not re-run by us. Raw predictions and scripts are in the results bucket. Its accuracy counts a tie in the gold as correct and its ECE uses 10 bins, so small differences from the other rows can come from the scorer. Latency was measured on mixed Jobs hardware and is left out. d3-turbo was served through its own system_one() runtime because vLLM does not load it yet. ♠ Scored through the model's native POST /v1/decide, one decision per request, not one request per case.

Fitted or fine-tuned on train

These models were trained on this benchmark's train split, so they are not zero-shot and their scores are not comparable with the table above. All were self-reported in the linked discussions and were not re-run by us, except the two specialists marked ours and the three Bekko rows, which we scored ourselves, and the rows marked ◆, scored by a third party (see the note under the first table). Scores well above the 0.735 teacher self-agreement mean a model is learning the teacher's quirks.

#	Model	Kind	Accuracy ↑	KL from gold ↓	Brier ↓	ECE ↓	Reported latency
1	Decider-4B (Qwen3.5-4B + LoRA) (#36)	general, then fine-tuned on train and other decision data, temperature-calibrated	0.8205	0.301	0.132	0.031¶	117 ms‡ (A800)
2	Arbiter-4B (from NeoHorse-Jev-4B + LoRA) (#34)	general, then fine-tuned on train and other decision data, temperature-calibrated	0.817	0.092	0.048	0.184¶	133 ms‡ (A800)
3	Darwin-27B-ZTC-v5 (27B, full fine-tune) (#33)	general, then fine-tuned on train	0.812	–	–	–	–
4	ministral3-3b-reasoning-typed-decisions (Ministral 3 3B Reasoning, full fine-tune) (#30)	specialist	0.808	0.077	0.042	0.182	373 ms‡ (GB10)
5	Amos 1 large (27B), per-question priors fitted on train (#31)	general, then fitted on train (one log-prior bias per question and option; no weights changed)	0.8025	0.122	–	–	357 ms‡
6

¶ The submitter's own ECE definition, which does not match the one used above.

§§ ProKope reports a multi-class vector Brier score, which is not the per-decision Brier used in the other rows, so the two are not comparable.

◇ Clean re-score by the submitter after removing 500 training pairs derived from model predictions on test rows (discussion #25). The originally submitted 0.906 was measured on a split the model had partly seen, so it is not listed; KL, Brier and ECE were only reported for that run. Do not train on test rows, test labels, or anything derived from predictions on them.

meraGPT Decider 1 is state of the art among zero-shot models on this benchmark. It leads Jev on every question type (noul 0.840 vs 0.775, choice 0.733 vs 0.720, score 0.739 vs 0.696), its distributions sit far closer to the gold (KL 0.096 vs 1.442), it is faster end to end, and it costs less per token. It answers POST /v1/systemone at meragpt.com, so the typesafe-sdk works against it by setting TYPESAFE_BASE_URL.

To add a model, score it on test with the full distributions and open a discussion with the numbers and the mode (specialist or general) it used.

Benchmarking a model with HF Jobs

One command scores any model that serves vLLM's /v1/systemone (any model without a native server works this way; serve it with a current nightly). It runs one request per case (state + all five questions) over the test split and prints accuracy, KL from gold, Brier and ECE:

hf jobs run --flavor a100-large --timeout 2h \\
  --env MODEL_ID=your-org/your-model \\
  --secrets HF_TOKEN \\
  --volume hf://datasets/LocalLLaMA/typed-decisions:/repo:ro \\
  pytorch/pytorch:2.9.1-cuda12.8-cudnn9-devel \\
  bash /repo/bench/score_with_jobs.sh

• Pick the --flavor for the model's size: l4x1 up to ~2B, l40sx1 or a100-large for ~27B, h200 above.
• bench/score_with_jobs.sh installs vLLM nightly, serves the model, scores all 400 cases, and writes results.json.
• The run finishes by creating the eval YAML and opening a PR on the model repo with the four scores, so they show on the Hub leaderboard once merged. Add --env MODE=specialist if you fitted the model on train.

Notes on the rows

• Liquid AI d1 was measured on 2026-09-30 through Liquid's API (https://api.liquid.ai/v1/systemone, model: d1:free), with the same client code as the Jev row: all 2,000 decisions, zero errors. It ties Decider 1 on noul (0.840) and choice (0.732 vs 0.733) and trails on score (0.677 vs 0.739), and it beats Jev on accuracy and KL. Liquid lists no per-token price yet, so the row shows the free tier.
• Jev 1.13.0 was measured on 2026-09-18 through TypeSafe's API (jev-latest, which reported jev-1.13.0): all 2,000 decisions, zero errors, $0.016 in total. Its accuracy is near the 0.735 ceiling, but it puts nearly all its probability on one answer, which is where the KL gap comes from. Its confidence is not badly calibrated (overconfidence +0.023); the gold is a three-sample spread it does not reproduce.
• Jeff (firelex/jeff, Apache-2.0) was scored on 2026-09-29 through the same client code as Jev, against Jeff's own jeff-serve (commit 2c1bfce) with the calibration each checkpoint ships. All three clear the Prior on accuracy but not on KL. Jeff's own 83.1% comes from a different five-benchmark panel. If there is a better way to serve them, open a discussion and we will rescore.
• Self-reported rows (§ above, and the second table) come from discussions #5 (prima-ratio), #7 (Bongard-mini), #8 (OpenDecider), #4 (od1), #3 (soft-decider), #2 (Laya), #34 (Arbiter-4B), #35 (linne-oak 1.0), #36 (Decider-4B) and #39 (SafeDecide, which scores each question on its own rather than all five in one request). Each submitter states the mode; we have not re-run them. If a number looks wrong, say so in the discussion and we will correct it.

Reading a score

Gold is the mean of three samples from a teacher of roughly 4B-class capability, so a score measures agreement with that teacher, not correctness. A better model can score lower wherever the teacher is wrong (it missed a duplicate invoice whose ID matched an earlier one).

Reference	Accuracy	What it is
Prior	0.470	the floor: label frequencies, ignoring the input
Perfect factor recovery	0.704	a model fitted to the latent factors that generated each case
Teacher self-agreement	0.735	a fresh teacher sample against gold built from the others

Scores well above 0.735 mean a model is learning the teacher's quirks. Per-question ceilings vary from 0.560 (agent_trace/urgency) to 0.937 (customer_service/category), so read scores per question as well as on average.

Specialist and general numbers are not comparable. A specialist is fitted on train for these four workflows and cannot answer anything else. A general model takes any question schema at request time and has never seen these. Say which mode you used; the gap between them is the price of generality, not a quality ranking.

The data

Type	Answer	Shape
noul	yes/no	probability that the statement is true
choice	one of N labels	distribution over labels, plus confidence
score	an ordered rubric	distribution over levels, plus an expected score

Every option carries a written description in criteria, and the descriptions are part of the input.

Workflow	Decision	Train	Test
agent_trace_observability	Does an agent run need human review, and how urgently?	300	100
customer_service	The right response and action for a customer thread and account.	300	100
invoice_processing	Pay, hold or reject a vendor bill against its order and delivery.	300	100
security_incidents	Close, investigate or contain a security alert, given machine history.	300	100

state and questions together are exactly the body of a POST /v1/systemone request. gold holds the full gold distributions, and flat <question>__label / __probabilities / __score / __probability_true columns hold the same answers for convenience. factors and label_agreement describe how the case was built and are not model input.

from datasets import load_dataset
import json

ds = load_dataset("LocalLLaMA/typed-decisions", "customer_service", split="test")
row = ds[0]
state, questions, gold = (json.loads(row[k]) for k in ("state", "questions", "gold"))
print(gold["urgency"]["probabilities"])   # score against the full distribution

Report KL or log loss and Brier next to accuracy; calibration is the point.

How it was built

Each case starts from independently sampled latent factors (topic, tone, severity, discrepancy type and so on), which a model renders into free text where the artefact is textual and keeps structured where it is structured. A teacher labels each case three times at temperature 0.7, and the gold is the mean of those distributions, so it stays soft where a decision is genuinely ambiguous. Before release, an audit checks state diversity, label balance, and that the gold actually tracks the input. train comes from a separate run at a different seed; packaging refuses to build if any case id or state appears in both splits.

The write-up behind the benchmark, including the two bugs it exposed in the classifier library: Typed Decisions on Latent Node.