nari-decision-4b
A Korean + English decision model. Give it a state (text or JSON: a conversation, a document, a tool result…) and typed questions — yes/no (noul), pick-one (choice), or ordinal (score) — and it returns a calibrated probability for every option in one forward pass, without generating text. Use the confidence to route low-certainty cases to a person or a larger model (selective prediction). From the same pass it reads two answers per question — a trained decision head and the backbone's own answer letter — and averages them.
한국어·영어 결정 모델입니다. 상태(대화·문서·도구 결과 등)와 타입이 정해진 질문(예/아니오, 하나 고르기, 순서 척도)을 받아, 글을 생성하지 않고 한 번의 forward로 선택지마다 보정된 확률을 냅니다. 라우팅, 가드레일(그대로 보내도 되나), RAG 관련성 판단, 에이전트 분기, 데이터 라벨링에 씁니다. 같은 forward에서 질문마다 학습한 결정 헤드의 답과 백본이 직접 고른 글자 답을 함께 읽어 평균합니다.
Highlights
- Most accurate in both languages. Against every model below, on the same questions, it leads every column: held-out 85.9% Korean / 85.3% English, System One eval set 82.1% / 76.9% — 5.1–15.5 points above the best other model in each.
- Reads the instruction, negation included. Asked about the same text and options for the statement that matches and for the one that does not (1,000 held-out KLUE-NLI pairs, instruction wording not seen in training), it gets both right in 98.3% of pairs (v2: 98.9%).
- Fast: all questions in one forward pass. 32 ms per request on one NVIDIA A100 80GB PCIe at batch 1 with
enable_cuda_graphs() (63 ms without; mean of 5 requests) — on par with laya (English) at 30 ms, 56% faster than clef-flash (9B) (73 ms), 72% faster than Prompted Qwen3.5-4B (115 ms) and 77% faster than JEV-9B (141 ms). Longer states and more questions add little (25 → 44 ms from simple to complex requests); models that send one request per question grow with each (Prompted Qwen3.5-4B 94 → 146 ms, JEV-9B 86 → 225 ms). Only laya-multilingual is faster (23 ms), at 30–36% accuracy on the eval set.
- Calibrated confidence. One temperature per question type, fitted on a separate calibration split: held-out ECE 0.012, eval set 0.041. Send low-confidence cases to a person or a larger model.
| Model | Size | Held-out ko | Held-out en | Eval set ko | Eval set en | ms / request |
|---|
| nari-decision-4b (this model) | 4B | 85.9 | 85.3 | 82.1 | 76.9 | 32 |
| Prompted Qwen3.5-4B | 4B | 68.8 | 62.6 | 67.8 | 64.7 | 115 |
| clef-flash (9B) | 9B | 75.2 | 69.8 | 73.8 | 71.8 | 73 |
| JEV-9B | 9B | 67.3 | 53.2 | 76.0 | 67.3 | 141 |
| laya-multilingual | 322M | 51.0 | 49.9 | 36.4 | 30.1 | 23 |
| laya (English) | 395M | — | 52.1 | — | 36.5 | 30 |
Accuracy (%) after temperature calibration, same questions for every model (on held-out, Prompted Qwen3.5-4B and JEV-9B skip questions with more options than they can take). ms / request: mean of 5 requests at batch 1 on one NVIDIA A100 80GB PCIe (this model with enable_cuda_graphs()). Details under Evaluation.
요약
- 두 언어 모두 가장 정확합니다. 같은 문항으로 비교한 아래 모델들보다 모든 칸에서 높습니다: held-out 한국어 85.9% / 영어 85.3%, System One 평가셋 82.1% / 76.9% (칸마다 그다음 모델보다 5.1–15.5%p 높음).
- 지시문을 읽습니다 (부정형 포함). 같은 글·같은 보기에 맞는 것과 맞지 않는 것을 각각 물으면 (KLUE-NLI 검증 1,000쌍, 학습에 없던 지시문) 두 질문을 모두 맞히는 비율이 98.3%입니다 (v2: 98.9%).
- 빠릅니다. 질문 여러 개를 한 번의 forward로 판단해 NVIDIA A100 80GB PCIe 한 장, 배치 1에서 요청당 32ms입니다 (
enable_cuda_graphs()를 켠 값이고 끄면 63ms, 5건 평균). laya (English) 30ms와 비슷하고, clef-flash (9B) 73ms보다 56%, Prompted Qwen3.5-4B 115ms보다 72%, JEV-9B 141ms보다 77% 빠릅니다. 질문이 늘고 상태가 길어져도 25ms → 44ms로 조금만 늘어납니다. 이보다 빠른 건 23ms인 laya-multilingual뿐이고, 평가셋 정확도는 30–36%입니다.
- 확률이 보정되어 있습니다. 질문 타입별 temperatures를 따로 떼어 둔 보정 분할로 맞춰 held-out ECE 0.012, 평가셋 0.041입니다. 확신이 낮은 건은 사람이나 더 큰 모델로 넘기면 됩니다.
Usage
from transformers import AutoModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained("blackmallangcow/nari-decision-4b", trust_remote_code=True)
model = AutoModel.from_pretrained("blackmallangcow/nari-decision-4b", trust_remote_code=True).eval()
# On a GPU, for one request at a time: record the forward once per input-length bucket (about 5 s), then reuse it
# model.to("cuda").enable_cuda_graphs()
out = model.decide(
tok,
state="고객: 지난주 주문이 두 번 결제됐어요. 한 건은 취소해 주세요.",
questions={
"refund": {"type": "noul", "instructions": "환불·취소 요청인가요?"},
"team": {"type": "choice", "instructions": "담당 팀을 고르세요.",
"options": ["billing", "shipping", "account", "none"]},
"urgency": {"type": "score", "instructions": "긴급도", "options": ["1", "2", "3", "4", "5"]},
},
)
# {"answers": {"refund": {"answer": True, "p_true": ..., "probs": {...}},
# "team": {"answer": "billing", "confidence": ..., "probs": {...}},
# "urgency": {"answer": "3", "confidence": ..., "expected": ..., "probs": {...}}},
# "truncated": False}
instructions is optional; the question name and options alone also work. Korean or English.- If state + questions exceed 2,048 tokens the state is cut in the middle, keeping its first quarter and last three quarters of the room (
truncated tells you). Questions and options are never cut.
- Several questions can be asked about the same state in one call.
- Each answer also has
paths: the decision head's and the backbone letter's probabilities before averaging and calibration. Questions with more than 26 options use the head only.
- Needs a recent
transformers with Qwen3.5 support. flash-linear-attention and causal-conv1d make it faster on GPU but are optional.
Guidelines
Writing questions
-
Give each question a name that says what it asks. The name is written into the input, and without instructions it is the whole question:
### 질문 1: q1 [예/아니오 (true/false)] <- the model cannot tell what is asked
### 질문 1: contains_pii [예/아니오 (true/false)] <- the name alone asks the question
q1 → contains_pii (does the text expose personal information?)flag → safe_to_send (can this reply go to the user as is?)cat → support_team (which team should handle it?)level → urgency (how urgent, 1–5?)
-
Add a none option when none of the options may apply.
-
Keep options mutually exclusive.
-
Put everything the decision depends on into the state, including settings and earlier turns. The model is more likely to call information missing when it appears only indirectly: in settings, in an earlier turn, or loosely ("any time this week").
-
For an ordinal question ("type": "score"), list its options from lowest to highest, e.g. ["1", "2", "3", "4", "5"] or ["low", "medium", "high"]. The model reads this order as the scale, and the answer's expected is the probability-weighted position (1 = the first option).
-
Phrase yes/no (noul) questions so that "yes" is the case you want to catch; p_true is its probability.
-
With up to 26 options both readouts answer; with more, only the decision head does.
Act on the confidence
Use the probability, not only the top answer: act automatically above a threshold and send the rest to a person or a larger model.
Example from the System One eval set: 50 guardrail responses (26 unsafe, 24 safe), sending a response only when p_true of safe_to_send is at least the threshold.
| Threshold | Unsafe responses sent | Safe responses held for a person |
|---|
| 0.5 | 2 / 26 | 1 / 24 |
| 0.8 | 1 / 26 | 1 / 24 |
| 0.9 | 1 / 26 | 4 / 24 |
The sample is small. Start near these values and set the threshold on a few hundred labeled examples of your own.
For questions that decide whether to ask the user back (is information missing, does the request need clarification), ask back only above a higher threshold. Example from the System One eval set: 51 clarification questions, 21 where the information is given and 30 where it is missing, counting an answer as "missing" only when p_true is at least the threshold.
| Threshold | Information given but called missing | Missing information not caught |
|---|
| 0.5 | 8 / 21 | 2 / 30 |
| 0.7 | 5 / 21 | 3 / 30 |
| 0.8 | 4 / 21 | 4 / 30 |
Again a small sample: tune the threshold on your own requests.
Multi-step tool calls
Let the model choose the next tool, but enforce prerequisite steps such as authentication or looking up an ID in your agent code. The model tends to pick the tool for the user's goal rather than the step that has to come first: on the API-Bank tool-selection task of the Decision Index public suite, when the correct next call was the authentication step (GetUserToken), it chose it in 2 of 108 cases (256 of 508 correct overall).
For ordinal (score) questions, read expected or the whole distribution rather than only the top option: on held-out the top option is exactly right 57.7% of the time, but within one step 91.9%.
Pin the version
Without revision, from_pretrained follows main, which moves with each release. In production, pin one: revision="v3.2".
Re-fit the temperatures
Probabilities are scaled by temperatures fitted on the calibration split, kept in model.config.temperatures. On data unlike the training mix, re-fit them: set model.config.temperatures = {"*": 1.0} to get unscaled probabilities, collect them for a few hundred labeled questions of your own, find for each key the value T that minimizes the log loss of softmax(log p / T), and write the values back.
Keys are type:options (noul:2, choice:3-5, choice:6+, choice:2, score:3-5, choice:27+, score:6+, where choice:27+ is for head-only questions); * is used for any missing key.
Intended use
- Decisions over text you already have: routing, guardrail checks, RAG relevance, agent branching, ticket triage, moderation queues and labeling, with low-confidence cases sent to a person or a larger model.
Out of scope
- Decisions with legal, medical, hiring or credit consequences without human review.
- A general content-safety filter: the guardrail questions cover PII, secrets and prompt injection.
- Questions that need broad knowledge or long reasoning (see Limitations), and generating text.
Bias
- Part of the training data is hate-speech and social-bias data (K-MHaS, Korean HateSpeech, KoSBi, DynaHate, Civil Comments). Judgments about groups of people can still be biased. On KoBBQ (Korean bias QA, never trained on) it answers 87.5% correctly.
Hardware
- The weights are bf16: 4.2B parameters, 8.46 GB (text model only; the backbone's vision part is left out). Plan on GPU memory beyond that for the input;
enable_cuda_graphs() keeps extra buffers for its recordings.
enable_cuda_graphs() needs a CUDA GPU. CPU and Apple Silicon (MPS) are untested.- There is no quantized version: quantizing would change the scores the
temperatures were fitted on.
decide() takes one state per call; for bulk labeling, loop over the records (about 31 requests per second on one NVIDIA A100 80GB PCIe with enable_cuda_graphs()).
사용 가이드 요약
- 질문 이름은 무엇을 묻는지 드러나게 짓습니다. 이름이 입력에
### 질문 1: contains_pii [예/아니오 (true/false)]처럼 그대로 들어가서, 지시문이 없으면 이름이 곧 질문입니다. 예: q1 대신 contains_pii(개인정보가 드러나 있나), flag 대신 safe_to_send(그대로 보내도 되나), cat 대신 support_team(어느 팀이 맡나), level 대신 urgency(얼마나 급한가, 1–5).
- 맞는 선택지가 없을 수 있으면 선택지에
none을 넣습니다.
- 선택지끼리 뜻이 겹치지 않게 씁니다.
- 판단에 필요한 설정·앞선 대화는 상태에 직접 넣습니다. 정보가 설정이나 앞선 대화에만 있거나 느슨하게 주어지면 ("이번 주 중 아무 때나") 빠졌다고 판정하기 쉽습니다.
- 순서 척도 질문(
"type": "score")은 선택지를 낮은 쪽에서 높은 쪽 순서로 적습니다. 예: ["1", "2", "3", "4", "5"], ["낮음", "보통", "높음"]. 모델은 이 순서를 척도로 읽고, 답의 expected도 이 순서의 순번(첫 선택지 = 1)으로 계산합니다.
- 사용자에게 되물을지 정하는 질문(정보가 빠졌나, 확인이 필요한가)은 더 높은 확신에서만 되묻습니다. 평가셋 clarify 51문항에서
p_true 0.5 이상을 빠짐으로 보면 정보가 있는 21문항 중 8개를 빠졌다고 하고, 0.7 이상이면 5개로 줄어듭니다 (정말 빠진 30문항 중 놓치는 수는 2 → 3).
- 여러 단계 도구 호출에서 인증·ID 조회처럼 먼저 불러야 하는 단계는 에이전트 코드에서 순서를 지킵니다. 모델은 사용자 목적의 도구를 바로 고르는 경향이 있어, 공개 스위트 API-Bank에서 정답이 인증 단계인 108문항 중 2개만 골랐습니다.
- 순서 척도 질문은 1등 선택지만 보지 말고
expected나 확률 분포를 봅니다. held-out에서 1등 선택지가 정확히 맞는 건 57.7%지만, 한 칸 차이 안에는 91.9%가 들어옵니다.
- 운영에서는
revision="v3.2"로 버전을 고정합니다. 지정하지 않으면 새 버전이 나올 때 바뀝니다.
- 학습 분포와 다른 데이터에서는
temperatures를 자기 데이터 수백 건으로 다시 맞춥니다 (model.config.temperatures).
- 법률·의료·채용·대출처럼 결과가 큰 결정에는 사람 검토 없이 쓰지 않습니다. 일반 유해 콘텐츠 필터가 아니고, 넓은 지식이나 긴 추론이 필요한 질문과 글 생성에는 맞지 않습니다.
- 가중치는 bf16 8.46GB이고 그보다 많은 GPU 메모리가 필요합니다. CUDA 그래프는 CUDA GPU에서만 되고, 양자화본은 없습니다.
Versions
- v3.2 (2026-10-18, this version) — more decisions for agents and RAG: claim verification, checking a response against its sources, phishing and spam, choosing a tool or intent, commonsense and math. Each question family's answers are also balanced in the loss, so one task's yes/no habit does not leak into another. Continued from v2 (40,553 steps in total)
- Decision Index 0.3, public suite (run locally, public part only): 38.49 → 44.80 (see the table under Evaluation)
- New training data: HoVer, VitaminC, Fool Me Twice, FaithDial, MASSIVE-Agents, STAR, Taskmaster-3, BANKING77, Amazon ESCI, WinoGrande, CommonsenseQA, CSQA2, Social IQa, QASC, StrategyQA, AQuA-RAT, GSM8K, BLiMP, the Nazario and CLAIR phishing corpora, UCI SMS Spam, Korean public text messages, and new synthetic sets (response choice, RAG response checks, phishing). Training rows that overlap any Decision Index public benchmark were removed
- System One eval set: 80.2% → 80.5% (Korean 82.6% → 82.1%, English 74.4% → 76.9%), ECE 0.034 → 0.041
- Held-out, same 79,688 questions: 86.3% → 85.6%, ECE 0.012 → 0.012
- MMMLU-ko, all 13,912 questions: 58.3% → 58.5%; 2026 bar and legal ethics exams (188 questions): 43.6% → 43.1%
- Polarity pairs, both right: 98.9% → 98.3%
- Batch-1 latency with
enable_cuda_graphs() on one A100 80GB PCIe: 32 → 32 ms
- Weaker than v2: Belebele-ko 87.1% → 84.6%. On the eval set's clarification items (80.4%, v2 86.3%) it more often judges information as missing when it is given only in settings, in earlier turns or loosely ("any time this week"); routing 72.1% (v2 73.8%)
- 에이전트·RAG용 판단을 늘렸습니다: 주장 검증, 근거 대조, 피싱·스팸, 도구·의도 고르기, 상식·수학. 질문 묶음마다 정답 비율을 손실에서 맞춰, 한 과제의 예/아니오 습관이 다른 과제로 옮겨 가지 않게 했습니다. Decision Index 공개 지수 38.49 → 44.80
- v2 (2026-10-15) — a second answer from the same forward pass: the backbone's own answer letter, averaged with the decision head. Trained from scratch for 1 epoch, then continued for 1 more at a lower learning rate (17,082 steps)
- Input: each question is its own chat turn ending in "Answer with one letter."; the answer slot is the start of the assistant turn
- The letter readout is trained with cross-entropy and kept close to the original backbone with a KL term, so the backbone's knowledge stays usable
- New training data: Korean bar exam (1st–13th) and legal ethics exam (1st–15th) multiple-choice questions from the Ministry of Justice (KOGL Type 1)
- MMMLU-ko, all 13,912 questions: 49.4% → 58.3% (the prompted backbone: 58.6%)
- 2026 bar and legal ethics exams (188 questions, after every round used for training): 19.1% → 43.6% (the prompted backbone: 29.3%)
- Belebele-ko: 79.6% → 87.1%
- System One eval set: 79.8% → 80.2% (Korean 80.7% → 82.6%, English 77.6% → 74.4%), ECE 0.049 → 0.034
Evaluation
Held-out (79,688 questions)
Official validation/test splits of the training sources plus evaluation-only benchmarks (APEACH, Belebele 한국어, KoBBQ, MMMLU 한국어), at most 1,000 records per task. Never used for training or calibration.
| Questions | Accuracy (%) | ECE (raw → calibrated) | NLL | Brier | Selective acc @80% (%) |
|---|
| All | 79,688 | 85.6 | 0.009 → 0.012 | 0.412 | 0.199 | 92.7 |
| Korean | 43,730 | 85.9 | 0.022 → 0.025 | 0.407 | 0.209 | 92.3 |
| English | 35,958 | 85.3 | 0.012 → 0.009 | 0.418 | 0.187 | 93.3 |
| noul | 45,371 | 90.6 | 0.005 → 0.009 | 0.277 | 0.140 | 94.8 |
| choice | 27,907 | 83.9 | 0.028 → 0.020 | 0.490 | 0.226 | 91.5 |
| score | 6,410 | 57.7 | 0.028 → 0.026 | 1.025 | 0.502 | 62.7 |
The numbers above are from the evaluated model (LoRA kept separate). The released weights merge the LoRA into bf16; on a sample, accuracy of evaluated → released: eval set: 79.83 → 79.56 % (362 questions, answers agree 99.5%); held-out: 84.40 → 84.40 % (2,602 questions, answers agree 99.7%).
System One eval set (519 questions, 200 items: Korean 140 / English 60)
Hand-checked scenarios close to real use: routing, guardrails, RAG relevance gates, agent branching, clarification. Written with an open-weights model and reviewed item by item; no overlap with the training data (MinHash, Jaccard 0.5).
| Questions | Accuracy (%) | ECE (raw → calibrated) | NLL | Brier | Selective acc @80% (%) |
|---|
| All | 519 | 80.5 | 0.045 → 0.041 | 0.521 | 0.278 | 88.2 |
| Korean | 363 | 82.1 | 0.044 → 0.048 | 0.490 | 0.262 | 89.3 |
| English | 156 | 76.9 | 0.095 → 0.075 | 0.593 | 0.315 | 86.4 |
| noul | 220 | 85.5 | 0.050 → 0.050 | 0.361 | 0.221 | 90.9 |
| choice | 200 | 86.5 | 0.061 → 0.049 | 0.415 | 0.195 | 94.4 |
| score | 99 | 57.6 | 0.153 → 0.118 | 1.090 | 0.572 | 60.0 |
| Category | Accuracy (%) | ECE |
|---|
| routing | 72.1 | 0.097 |
| guardrail | 83.0 | 0.057 |
| rag_gate | 83.5 | 0.066 |
| agent | 84.2 | 0.090 |
| clarify | 80.4 | 0.114 |
Ordinal (score) questions
Exact accuracy is strict for ordinal labels, and most misses are one step off. expected is the probability-weighted position of the answer (1 = the first option); its error is counted in option steps.
| Set | Questions | Exact (%) | Within one step (%) | Mean error of expected (steps) | ECE |
|---|
| Held-out | 6,410 | 57.7 | 91.9 | 0.62 | 0.026 |
| ↳ HelpSteer2 response ratings, 0–4 | 5,000 | 57.7 | 93.2 | 0.61 | 0.038 |
| ↳ KLUE-STS similarity, 0–5 | 519 | 56.6 | 97.1 | 0.53 | 0.075 |
| ↳ Korean HateSpeech level (none < offensive < hate) | 471 | 64.8 | 97.5 | 0.40 | 0.125 |
| ↳ HelpSteer2 preference strength, −3 to 3 | 420 | 51.4 | 64.3 | 1.14 | 0.062 |
| System One eval set | 99 | 57.6 | 87.9 | 0.62 | 0.118 |
Two readouts from one pass
Each question is answered twice from the same forward pass: by the decision head and by the backbone's own answer letter. The released model averages the two (log-probabilities, 0.5 / 0.5) and then calibrates. Accuracy (%) / ECE, each readout calibrated on its own, on the questions that have an answer letter (at most 26 options): 75,688 held-out questions.
| Set | Decision head | Backbone letter | Averaged (this model) |
|---|
| Held-out | 85.6 / 0.016 | 83.8 / 0.030 | 85.7 / 0.011 |
| System One eval set | 80.9 / 0.059 | 77.8 / 0.041 | 80.5 / 0.041 |
Knowledge, reading and exams (never trained on)
Accuracy (%). The prompted backbone is Qwen/Qwen3.5-4B without training, asked one question per request with the options as letters (first-token letter probabilities). MMMLU-ko is OpenAI's professional Korean translation of MMLU (MIT); Belebele-ko is Meta's reading benchmark (CC BY-SA 4.0); the exams are the Ministry of Justice's 2026 rounds, after every round used for training (KOGL Type 1, source: 법무부).
| Benchmark | Questions | This model | Prompted Qwen3.5-4B | v2 |
|---|
| MMMLU-ko, all questions (knowledge) | 13,912 | 58.5 | 58.6 | 58.3 |
| Belebele-ko (reading) | 898 | 84.6 | 82.5 | 87.1 |
| Korean bar exam (15th) + legal ethics exam (17th), 2026 | 188 | 43.1 | 29.3 | — |
Decision Index 0.3, public suite
Chance-corrected skill (0 = random guessing, 100 = perfect) on the public benchmark panel of the Decision Index leaderboard (edition 0.3, 140,178 scored requests), run locally with the leaderboard's own kit; unanswered requests count as wrong. The index is a fixed-weight mean of five areas. This is the public part only: the board's Full score adds private tests that its maintainers run. These are our own runs, not numbers taken from the leaderboard.
| v2 | This model |
|---|
| Decision Index (public) | 38.49 | 44.80 |
| Knowledge & Reasoning | 24.6 | 28.8 |
| Language Understanding | 42.0 | 44.5 |
| Retrieval & Classification | 40.4 | 59.8 |
| Tools & Automation | 60.6 | 62.1 |
| Arts & Human Taste | 21.3 | 25.4 |
Baselines (zero-shot, same questions)
- Prompted Qwen3.5-4B: the same backbone without training; options shown as letters, first-token letter probabilities (questions with more than 26 options skipped).
- clef-flash (9B): Cloudflare's open decision model (Qwen3.5-9B based), released weights and code.
- JEV-9B: AutoTrust's open decision model (Qwen3.5-9B + decision LoRA/head), System 1 via vLLM as in its model card (choice ≤ 16 options, score ≤ 6 levels).
- laya-multilingual: Convai's open decision model (mmBERT-base, 322M), released weights via the laya package.
- laya (English): Convai's English decision model (ModernBERT-large), English records only.
- All baselines are temperature-calibrated the same way on the calibration split. Cells are accuracy (%) / ECE.
| Set | This model | Prompted Qwen3.5-4B | clef-flash (9B) | JEV-9B | laya-multilingual | laya (English) |
|---|
| Held-out | 85.6 / 0.012 | 66.0 / 0.038 | 72.8 / 0.014 | 61.1 / 0.040 | 50.5 / 0.025 | — |
| Held-out, Korean | 85.9 / 0.025 | 68.8 / 0.040 | 75.2 / 0.021 | 67.3 / 0.028 | 51.0 / 0.029 | — |
| Held-out, English | 85.3 / 0.009 | 62.6 / 0.047 | 69.8 / 0.027 | 53.2 / 0.059 | 49.9 / 0.040 | 52.1 / 0.030 |
| System One eval set | 80.5 / 0.041 | 66.9 / 0.052 | 73.2 / 0.077 | 73.4 / 0.088 | 34.5 / 0.119 | — |
| Eval set, Korean | 82.1 / 0.048 | 67.8 / 0.061 | 73.8 / 0.077 | 76.0 / 0.115 | 36.4 / 0.110 | — |
| Eval set, English | 76.9 / 0.075 | 64.7 / 0.058 | 71.8 / 0.098 | 67.3 / 0.072 | 30.1 / 0.173 | 36.5 / 0.153 |
Speed (batch 1, one NVIDIA A100 80GB PCIe)
5 requests from the System One eval set: 3 simple (2 questions, 43–97-character state; 2 Korean, 1 English) and 2 complex (3 questions, 552–900-character state; 1 Korean, 1 English). Each number is the mean over those requests of the median of 10 runs (after 3 warm-up runs). The other models were measured on one A100 80GB PCIe on 2026-10-15; this model (the released weights) was measured the same way on an A100 80GB PCIe on 2026-10-18. The computation is the same as v2's: 63.4 ms eager here, 64.3 ms for v2 on the first machine. Models run through transformers are eager (no compilation or CUDA graphs) unless the row says enable_cuda_graphs(); the vLLM servers use vLLM's default CUDA graphs.
| Model | Mean (ms) | Simple (ms) | Complex (ms) | How it runs |
|---|
| laya-multilingual | 22.7 | 22.1 | 23.5 | laya package (encoder), all questions in one pass |
| laya (English) | 30.4 | 27.0 | 35.5 | laya package (encoder), all questions in one pass |
| nari-decision-4b (this model) | 32.2 | 24.7 | 43.5 | transformers + enable_cuda_graphs(), all questions in one forward pass |
| nari-decision-4b (this model) | 63.4 | 62.7 | 64.5 | transformers, all questions in one forward pass |
| clef-flash (9B) | 73.4 | 66.4 | 83.8 | transformers, all questions in one forward pass |
| Prompted Qwen3.5-4B | 115.1 | 94.4 | 146.1 | local vLLM server, one request per question |
| JEV-9B | 141.4 | 85.7 | 224.9 | local vLLM server, one request per question |
Training
- Backbone:
Qwen/Qwen3.5-4B (text model only), frozen, with LoRA on every linear layer (attention, Gated DeltaNet projections, MLP). The LoRA is merged into the released weights.
- Decision head: the hidden state at the end of each option line is the query; 2 layers of self-attention across options and cross-attention over the whole sequence, plus a linear prior; softmax per question.
- Input: the backbone's chat template with one user turn per question, ending in "Answer with one letter."; the answer slot is the start of the assistant turn (earlier questions get an empty one).
- Backbone letter: the hidden state at the answer slot times the backbone's own (tied) embedding rows for the option letters A, B, …. The answer averages the head's and the letters' log-probabilities (0.5 / 0.5), then applies the temperature.
- Loss: log loss + Brier score (+ ranked probability score for ordinal questions) on the head, all strictly proper scoring rules, plus cross-entropy on the answer letter and a KL term that keeps the letter distribution close to the original backbone (adapter off, on a quarter of each batch). Option order is shuffled every epoch (except ordinal).
- Calibration: one temperature per (question type, number of options), and one for head-only questions with more than 26 options, fitted on a separate calibration split:
{"*": 1.0, "noul:2": 0.9278, "choice:3-5": 1.1189, "choice:6+": 0.9632, "choice:2": 1.0, "score:3-5": 1.0382, "choice:27+": 1.1189, "score:6+": 0.8937}
- Data: 427,158 labeled questions (Korean 52%, English 48%) from 54 sources. 40,553 steps on one A100 80GB.
- Answer balance: in 70 of 110 question families the loss weights each answer so the weighted answer mix is close to uniform (yes/no questions: 41.4% → 48.7% yes), so the model does not carry one task's answer prior into another. Families whose natural answer is "none" or "no" keep their mix, and 2 small families are up-weighted. Calibration and evaluation use the natural mix.
- No closed-model outputs anywhere in the training data. The synthetic part (D4) was generated by Qwen3.8-27B and kept only where Gemma-4-31B-it independently agreed, then spot-checked by a person.
Training data
| Source | Questions | License | Attribution |
|---|
| KLUE | 52,122 | CC BY-SA 4.0 | KLUE (Park et al., 2021), CC BY-SA 4.0 |
| KoSBi·SQuARe (NAVER) | 32,001 | MIT | KoSBi·SQuARe (Lee et al., 2023), NAVER, MIT |
| nari-decision synthetic decisions (D4) | 27,216 | Apache-2.0 (synthetic, generated and cross-checked by Apache-2.0 models) | Synthetic decisions generated with Qwen3.8-27B and cross-checked with Gemma-4-31B-it (Apache-2.0) |
| ABCD (Action-Based Conversations Dataset) | 16,000 | MIT | ABCD (Chen et al., 2021), ASAPP Research, MIT |
| MIRACL 한국어 | 15,110 | Apache-2.0 | MIRACL (Zhang et al., 2023), Apache-2.0 |
| Schema-Guided Dialogue (SGD) | 14,000 | CC BY-SA 4.0 | Schema-Guided Dialogue (Rastogi et al., 2020), Google, CC BY-SA 4.0 |
| KorNLI 학습 세트 (SNLI·MNLI 기계번역) | 14,000 | CC BY-SA 4.0 | KorNLI (Ham et al., 2020), Kakao Brain, CC BY-SA 4.0 |
| K-MHaS | 12,006 | CC BY-SA 4.0 | K-MHaS (Lee et al., 2022), CC BY-SA 4.0 |
| NoMIRACL 한국어 | 12,003 | Apache-2.0 | NoMIRACL (Thakur et al., 2024), Apache-2.0 |
| MASSIVE 1.1 한국어 (ko-KR) | 12,000 | CC BY 4.0 | MASSIVE (FitzGerald et al., 2022), Amazon, CC BY 4.0 |
| HelpSteer2 (NVIDIA) | 12,000 | CC BY 4.0 | HelpSteer2 (Wang et al., 2024), NVIDIA, CC BY 4.0 |
| KoBEST (SKT) | 11,485 | CC BY-SA 4.0 | KoBEST (Kim et al., 2022), SKT, CC BY-SA 4.0 |
| MKQA 한국어 | 10,000 | CC BY-SA 3.0 | MKQA (Longpre et al., 2021), Apple, CC BY-SA 3.0 |
| VitaminC | 10,000 | CC BY-SA 3.0 | VitaminC (Schuster et al., 2021), CC BY-SA 3.0 |
| CLINC150 (oos plus) | 10,000 | CC BY 3.0 | CLINC150 (Larson et al., 2019), CC BY 3.0 |
Limitations
- Calibration holds on data like the training mix. On a different distribution, re-fit the temperatures on a few hundred labeled examples of your own.
- Korean and English only. Other languages are untested.
- Ordinal (
score) questions are the weakest type: 57.7% exactly right on held-out (91.9% within one step). Most of them are human ratings (HelpSteer2), where people also differ by a step. On the System One eval set they are poorly calibrated (ECE 0.118 on 99 questions).
- It judges what is in the state; it has no access to tools, the web or current facts.
- The guardrail questions cover PII, secrets and prompt injection; this is not a general content-safety classifier.
- Knowledge-heavy questions and long reading still favor larger models (same held-out questions): MMMLU-ko (1,000 questions) clef-flash (9B) 77.1%, JEV-9B 65.4% vs 55.7% here; Belebele-ko (898 questions) JEV-9B 90.0%, clef-flash (9B) 89.9% vs 84.6% here. Use the confidence to send such inputs to a larger model.
- On general-knowledge questions the confidence runs high: MMMLU-ko ECE 0.091 (mean confidence 67.6% at 58.5% accuracy). Re-fit the temperatures if that is your main use.
- The Ministry of Justice exams used for training (bar exam 1st–13th, legal ethics 1st–15th) are partly memorized (81.0% there vs 43.1% on the 2026 rounds); do not use them to evaluate this model.
- Weaker spots on held-out: Korean paraphrase detection (PAWS-X) 82.7% vs English (PAWS) 94.3%; knowledge (MMMLU-ko) 55.7%.
License
Apache-2.0 for the weights and code. The training data includes CC BY and CC BY-SA sources; their attribution is listed in the table above.
The Korean national exam questions are public works of the Ministry of Justice released under KOGL Type 1 (공공누리 제1유형, attribution required); the attribution is in the table above.
