A compact typed-decision encoder for agents and pipelines. Version 2, distilled from Crivo-400M.
Crivo-150M reads a state and a set of typed questions (choice, noul, score) and returns a probability for
every option in a single encoder pass. It does not generate text. It is built for the classify, route, gate and verify
decisions that agents and pipelines make thousands of times, where a generative model is slow and expensive.
This is version 2. It has the same architecture and size as version 1, but it was retrained with the recipe of the
newer Crivo models: the training data of Crivo-400M, with Crivo-400M
as its teacher. Version 1 (board Full 22.21, public 36.39) stays available at revision 7a6c745; see
Versions for what changed and why.
| Architecture | Ettin / ModernBERT encoder, 22 layers, hidden 768, [OPT] readout + MLP scorer |
| Parameters | 149.6M (149,610,241) |
| Base model | cross-encoder/ettin-reranker-150m-v1 |
| Teacher | tardellirs/crivo-400m (knowledge distillation) |
| Decision Index 0.3 | public index 34.38 (self-run, 140,178 / 140,178 requests answered); Full score pending (maintainer-run) |
| Latency, 1× A100 40 GB, bf16 (one request at a time) | median 18.0 ms · mean 26.1 ms · p95 44.4 ms per request |
| Context | trained up to 8,192 tokens; reads longer inputs whole, up to 32,768, without truncation |
| Language | English |
| License | CC BY-NC 4.0 (see License) |
Crivo follows the typed-decision interface popularized by TypeSafe's Jev (a state plus typed questions in, a probability for every option out) and runs unmodified in the open Decision Index kit. It is an independent open model, not affiliated with or endorsed by TypeSafe, and not trained on Jev outputs. Crivo is Portuguese for "sieve"; passar pelo crivo, to pass through the sieve, means to be judged and sorted.
Esta é a versão 2 do Crivo-150M, um encoder de decisões tipadas com 149M de parâmetros (Ettin/ModernBERT). Ele lê um
state e perguntas choice, noul ou score e devolve uma probabilidade para cada opção numa única passada, sem
gerar texto. A v2 tem a mesma arquitetura da v1, mas foi retreinada com os dados do Crivo-400M e destilada dele. No
índice público do Decision Index 0.3 marca 34,38 (run completo publicado), contra 36,39 da v1: a v2 usa bem menos
dados no formato exato das fontes da suíte e mais famílias de tarefas novas, e nos nossos painéis internos generaliza
melhor (domínios novos e mesmas skills). O Full score da v2, com os testes privados dos mantenedores, ainda não saiu; a
v1 (Full 22,21) continua disponível na revisão 7a6c745. Treinado só com dados abertos, com verificação de
contaminação contra a suíte. Funciona em inglês. Licença CC BY-NC 4.0 (uso não comercial).
Run with the official kit (apolinario/decision-index, commit
9eb2dbe, edition 0.3) on the complete public suite, with the engine shipped in this repository: every one of the
140,178 scoreable requests answered, none unsupported, nothing truncated. Scores are chance-corrected skill (0 = random
guessing, 100 = perfect), weighted as the board does. Full results:
tardellirs/decision-index-results → runs/crivo-150m-v2.
The board's main ranking (Full score) is 20% this public index, 50% private tests of the same skills and 30% private tasks from new domains, run by the maintainers. The Full score of version 2 will be added here once it is published.
| Version | Public index | Raw | Knowledge & Reasoning | Language | Retrieval & Classification | Tools & Automation | Arts & Human Taste |
|---|---|---|---|---|---|---|---|
v1 (7a6c745) | 36.39 | 52.45 | 11.8 | 41.8 | 52.2 | 60.7 | 10.1 |
| v2 (this revision) | 34.38 | 51.32 | 8.3 | 34.5 | 48.2 | 62.8 | 21.8 |
★ = gold benchmark (weight 1.2 within its area). The index uses the native metric listed, chance-corrected.
| Benchmark | Native metric | v1 | v2 |
|---|---|---|---|
| Knowledge & Reasoning | |||
| GPQA Diamond ★ | accuracy | 0.0 | 1.4 |
| GSM8K | accuracy | 2.0 | 1.2 |
| ChessBench | accuracy | 0.0 | 0.0 |
| MuSR | accuracy | 26.7 | 21.1 |
| SATA-Bench | case exact accuracy | 0.0 | 2.6 |
| CRUXEval | accuracy | 7.3 | 14.0 |
| CLadder | accuracy | 61.0 | 32.1 |
| HLE ★ | accuracy | 0.0 | 0.4 |
| MMLU-Pro ★ | accuracy | 3.4 | 4.3 |
| BBH fixed-option tasks ★ | accuracy | 11.6 | 8.2 |
| WinoGrande ★ | accuracy | 21.9 | 9.4 |
| Language Understanding | |||
| ContractNLI | macro-F1 | 74.7 | 71.4 |
| ANLI ★ | macro-F1 | 19.0 | 7.6 |
| HellaSwag ★ | accuracy | 48.9 | 39.0 |
| ACOS | per-review F1 | 38.9 | 19.6 |
| FinEntity | macro-F1 | 76.7 | 78.5 |
| iSarcasmEval | sarcasm F1, track A English |
A comparison with other models of similar size will be added together with the Full score, since the public index alone does not rank models by Full score.
Checkpoints were chosen on our own panels, never on the board and never on the complete public suite (it was run only on the final package). Two of the panels, run on v1 and v2 with the same engine, show what the new recipe changed. These are internal proxies, not substitutes for the board's private tests, and no Full score is estimated here.
| Panel | What it measures | v1 | v2 |
|---|---|---|---|
| ND-proxy | mean chance-corrected skill on 21 held-out decision families from outside the suite (written rules and policies with exceptions, schedules, dates, propositional logic, code outputs and tests, arithmetic in context, rewritten items, product-domain decisions, and others), chosen because they track the board's new-domain scores across 26 models | 0.150 | 0.224 |
| SS-core | mean chance-corrected skill on our own development sets for 8 of the board's skills (BFCL, BANKING77, CLINC150, When2Call, NLI4CT, ANLI, Amazon ESCI, smart-home control), not built from suite rows | 0.459 | 0.556 |
| Akita v1 (originals) | accuracy on the 130 original items of the Akita benchmark | 43.8% | 38.5% |
v2 is better on both proxies of the private parts of the board and worse on Akita. One caveat on the ND-proxy: the v2 training data includes code-generated decision families of related kinds (policies with exceptions, schedules, dates, logic, executed code). They were filtered against the panel rows, but the panel partly measures transfer from them.
The repository holds two versions in its git history. main is version 2.
| Version | Revision | Recipe | Public index (self-run) | Full score (board) |
|---|---|---|---|---|
| v2 (current) | a5e21d8067c5c37ba5f88cd498306ae5abfd625d | data of Crivo-400M, distilled from Crivo-400M, end of stage 3 | 34.38 | pending |
| v1 | 7a6c745 | own data mix, no teacher, average of stages 2 and 3 | 36.39 | 22.21 (private same skills 27.74, new domains 9.77) |
v1 is the board entry submitted in October 2026 (public 36.39, Full 22.21, on the Pareto frontier at 150M parameters at the time). It is also the teacher of Crivo-17M and of Crivo-32M v1. To load it:
from huggingface_hub import snapshot_download
path = snapshot_download("tardellirs/crivo-150m", revision="7a6c7453c06be576e1b48aeb70429318402791d7")
What changed in v2, and why. v1 scored well on private tests of familiar skills (27.74) but poorly on tasks from new domains (9.77): it had learned the suite's request layouts more than the skills behind them. v2 trades some of that fit for breadth:
The result is a lower public index (36.39 → 34.38), with losses concentrated on benchmarks v1 had seen in exact layout (CLadder, RAGTruth, ACOS, smart-home control, BFCL), and higher scores on both internal proxies of the private parts (ND-proxy 0.150 → 0.224, SS-core 0.459 → 0.556). Whether this turns into a higher Full score is what the maintainers' run will show. If your task is close to one of the benchmarks where v1 is clearly ahead, v1 may serve you better; pin its revision.
from huggingface_hub import snapshot_download
import sys; path = snapshot_download("tardellirs/crivo-150m"); sys.path.insert(0, path)
from crivo import Crivo
model = Crivo(path) # CUDA if available, else CPU
out = model.answer(
state={"ticket": "My card was charged twice for the same order."},
questions={
"intent": {"type": "choice", "instructions": "What does the customer want?",
"criteria": {"refund": "Money back for the duplicate charge",
"cancel": "Cancel the order",
"info": "Just asking for information"}},
"urgent": {"type": "noul", "instructions": "The customer needs a fast response."},
})
print(out["intent"]["choice"], out["intent"]["probabilities"]) # refund {'refund': 0.69, 'cancel': 0.08, 'info': 0.23}
print(out["urgent"]["noul"]) # 0.64
Question types:
| type | criteria | answer |
|---|---|---|
choice | {key: description or null}, 2+ options (no upper limit) | choice + probabilities over every key |
noul | optional {"true": ..., "false": ...} describing both poles | noul = P(statement in instructions is true) |
score | ordered list of level descriptions, lowest first | score = expected level + probabilities |
state and instructions can be strings or JSON objects (rendered as compact JSON with sorted keys, exactly as in
training). Pin a revision (snapshot_download(..., revision=...)) if you need a fixed version.
pip install "decision-index[transformers] @ git+https://github.com/apolinario/decision-index@9eb2dbe"
hf download tardellirs/crivo-150m --local-dir crivo-150m && cd crivo-150m
python -m decision_index pipeline --engine decision_index_engine:DecisionEngine --edition 0.3 --out runs/crivo-150m-v2
The engine defaults are the settings of the reported run: one option order (tta=1), fp32 scoring head, SDPA
attention, batch_size=32, token_budget=65536, max_length=32768, bf16 autocast on CUDA. The state is tokenized
once per request and all questions are batched. It raises Unsupported only for a question above max_length tokens,
and there is none in the 0.3 suite.
Input. Each question is one sequence:
[CLS] instructions [SEP] state [SEP] [OPT] option_1 [OPT] option_2 … [OPT] option_n
where option_i is key: description (or just key). noul is two options (false, true); score levels are
options 0: …, 1: …, …. A special token [OPT] (initialized from [SEP]) marks each option.
Readout. The encoder's hidden state at each [OPT] marker, plus a learned question-type embedding, goes through a
small MLP scorer (LayerNorm → Linear → GELU → Linear, run in fp32) to one logit per option; a softmax over the options
of the question gives the distribution. All questions of a request are batched together; options of a question see each
other (full bidirectional attention), and option order was randomized during training so the model does not rely on
position.
Context. ModernBERT/Ettin uses rotary position embeddings with alternating local (128-token) and global attention. Training used sequences up to 8,192 tokens. Longer inputs (in the 0.3 suite, mostly ToolRet requests carrying tool documents of ~23k tokens) are read whole with rotary positions up to 32,768 rather than refused or truncated.
Files. model.safetensors (fp32 weights), config.json (ModernBERT config + decision block), tokenizer files,
model.py (architecture, rendering, tokenization), crivo.py (standalone API), decision_index_engine.py (kit engine).
Full fine-tune of cross-encoder/ettin-reranker-150m-v1 in three stages, on exactly the training files of Crivo-400M,
with Crivo-400M as the teacher in stages 1 and 2. Every example is one question with a target distribution over its
options. The released weights are the end of stage 3.
| Stage | Questions | Content |
|---|---|---|
| 1 | 638,306 | Bekko System One dataset v0 (≤1,200 per subset) · code-generated decision families with exact labels (policies with exceptions and precedence, opaque label rules, schedules with constraints, ordinal severity, dates, propositional logic, executed code and tests, link-based phishing, multi-select) · format-invariance augmentations (opaque keys, key-only options, distractors, "none of the above") · 328 task families from tasksource/tasksource-jev-typed-decisions · sibling sources of the suite's skills (argument quality, SEntFiN, RAGBench, Hermes function calling, xLAM irrelevance, bitext banking and e-commerce intents, CUAD contracts) · train splits of the public benchmarks' sources, in the original layout and rewritten (≤5,000 per source) · tasksource/procedural-typed-decisions · original-vs-perturbed public-domain text · open teacher labels from Mapika/decider (open Qwen models) · CFPB complaint routing · commit-message matching (CommitPackFT) |
| 2 | 169,437 | tool selection (ToolACE, glaive), CLadder variants not used by the suite, SEntFiN entity sentiment, synthetic smart-home cases, 6,500 synthetic "which API next" dialogues with catalogs of ~50 APIs (gold calls from Hermes function calling), + replay of stage 1 |
| 3 | 24,635 | long context up to 8,192 tokens: ContractNLI, RAGTruth, NLI4CT and other long inputs, plus 5,000 "which API next" dialogues with long catalogs (~6k tokens) |
40 tasksource families were held out of training entirely and kept as a reserved check. No data labeled by Jev or any other proprietary decision API was used.
Crivo-400M's logits were computed once over the stage-1 and stage-2 training rows (it agrees with the gold label on 82.7% and 87.8% of them); the same logits were used for Crivo-68M and Crivo-17M. The loss is
α · (CE + 0.5 · Brier) + (1 − α) · τ² · KL(teacher ‖ student), with α = 0.5,
τ = 2 for questions with more than two options and τ = 1 otherwise. Sources where the teacher agrees with the gold label on fewer than 60% of their training rows (mostly math, logic and spatial reasoning, human-preference ratings and many tasksource families), plus smart-home control, CLadder and RAGTruth, train on the gold label only: 163 sources in stage 1 and 46 in stage 2. Stage 3 (long context) trains on the gold label only.
| Stage | Init | Steps | LR (encoder / head) | Max length | Tokens per micro-batch × accumulation | Teacher |
|---|---|---|---|---|---|---|
| 1 | Ettin reranker 150M | 5,941 (1 epoch) | 3e-5 / 3e-4 | 2,048 | 8,192 × 4 | Crivo-400M |
| 2 | stage 1 (final weights) | 2,406 | 2e-5 / 2e-4 | 2,048 | 8,192 × 4 | Crivo-400M |
| 3 | stage 2 | 2,589 | 1.5e-5 / 1e-4 | 8,192 | 8,192 × 4 | none (gold) |
| Final | — | end of stage 3 |
AdamW (weight decay 0.01), linear warmup (5% stage 1, 3% after) and linear decay, bf16 autocast, option order shuffled
for every choice example. A weight EMA (0.999) was kept during stage 1; the final stage-1 weights scored slightly
better than the EMA and initialized stage 2. Every checkpoint choice (stage-1 final vs EMA; stage 2, stage 3 or their
average; interpolations of 10% and 20% toward the base model, WiSE-FT) was made on our own panels: held-out new-domain
decision families, a development set of the board's skills, the Akita benchmark and a public-suite sample. None of them
used the board, and the complete public suite was run only on the final package. Stage 3 was best overall (SS-core 0.480 →
0.556 over stage 2, while the ND-proxy dipped from 0.242 to 0.224); the stage-2/3 average and both WiSE interpolations
were worse.
Compute. One NVIDIA A100 40 GB (Lambda Cloud), shared with the training of Crivo-32M v2. Training, panels, packaging and the complete public suite of both models took about 5 h 15 min of wall time; the teacher logits were reused from the Crivo-68M / Crivo-17M run. A reproducibility check on the package (500 questions, batch size 32 vs 4) gave the same answer on every question (max probability difference 0).
The training files are the ones of Crivo-400M, checked at three levels:
The API-Bank train split was not used: its dialogues and API catalogs repeat those of the test pool. The API-Bank skill comes only from the synthetic dialogues above, built from Hermes function-calling data and filtered against the suite. Benchmarks whose sources have no clean train split were used for evaluation only. The teacher only labeled these same training rows; it never saw suite items.
Fast, local, probabilistic decisions over text or JSON state, as a cheap first pass or a component inside an agent or pipeline:
7a6c745).key: description) works better than bare keys.CC BY-NC 4.0, the same as its teacher, Crivo-400M, and as v1: several training sources are licensed for non-commercial use only (for example ANLI, CC BY-NC 4.0, and some Bekko dataset subsets), so the weights are released for non-commercial use. Upstream dataset licenses and terms continue to apply to the data. The base model (Ettin reranker 150M) is Apache-2.0.
[OPT] readout design this model follows.@misc{crivo150m2026,
title = {Crivo-150M: a small typed-decision encoder},
author = {Tardelli Stekel},
year = {2026},
note = {Version 2, distilled from Crivo-400M},
howpublished = {Hugging Face model card},
url = {https://huggingface.co/tardellirs/crivo-150m}
}
| 0.0 |
| 0.0 |
| VAST | macro-F1 | 61.4 | 58.5 |
| NLI4CT | macro-F1 | 15.1 | 17.1 |
| RAGTruth | F1 on hallucinated class | 44.3 | 23.5 |
| Retrieval & Classification |
| BANKING77 ★ | macro-F1 | 90.6 | 85.7 |
| CLINC150+OOS ★ | macro-F1 | 91.8 | 89.5 |
| BRIGHT ★ | nDCG@10 | 18.6 | 21.9 |
| Amazon ESCI | macro-F1 | 29.8 | 21.5 |
| PhishNChips | accuracy (verdict) | 4.0 | 0.1 |
| HoVer | accuracy | 69.4 | 60.3 |
| Tools & Automation |
| BFCL ★ | case exact accuracy | 79.0 | 68.0 |
| ToolRet | nDCG@10 | 55.7 | 57.0 |
| API-Bank ★ | accuracy | 29.4 | 70.1 |
| Home appliance simulator | case exact accuracy | 79.5 | 63.6 |
| When2Call | accuracy | 62.5 | 52.6 |
| Arts & Human Taste |
| BPoMP | accuracy | 0.0 | 73.0 |
| Humicroedit | accuracy | 4.5 | 0.9 |
| POP909-CL | accuracy | 1.2 | 1.1 |
| cfcolor | accuracy | 0.6 | 0.4 |
| Habermas Machine | accuracy | 10.1 | 11.7 |
| New Yorker caption matching | accuracy | 44.1 | 43.4 |