Model
| Total Parameters | 4B |
| Architecture | Looped (latent-recurrent) normalized-GPT with interleaved sliding-window attention (SWA) and full-attention layers |
| Precision | BF16 |
| Context Length | Up to 256K tokens |
| Supported Languages | English |
| Best For | General-purpose reasoning on a small footprint |
| License | OpenMDW License Agreement, version 1.1 |
| Release Date | October 16, 2026 |
NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 is a looped (latent-recurrent) large language model (LLM) trained by NVIDIA. The model employs a normalized-GPT architecture with interleaved hybrid sliding window attention (SWA) and full attention layers. By repeatedly applying a shared block of layers, Dory1.0-4B can spend more compute per token allowing it to scale intelligence. This model is ready for commercial or non-commercial use.
Governing Download Terms: Use of this model is governed by the OpenMDW-1.1 model license.
Note: Dory support in vLLM is under review in vllm-project/vllm#60149. Until it is included in a vLLM release, install vLLM from that PR:
VLLM_USE_PRECOMPILED=1 pip install "git+https://github.com/vllm-project/vllm.git@refs/pull/60149/head"
VLLM_USE_PRECOMPILED=1 reuses compatible prebuilt vLLM kernels, avoiding local CUDA compilation. This section will be updated once a vLLM release supports Dory.
vllm serve --model nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 \
--hf-overrides '{"n_recurrent_loops": 2, "recurrent_kv_cache_mode": "per_loop"}' \
--max-model-len 262144 \
--max-num-seqs 128 \
--enable-prefix-caching \
--async-scheduling \
--reasoning-parser nemotron_v3 \
--tool-call-parser qwen3_coder \
--enable-auto-tool-choice
Validated context: 256K tokens. Lower --max-model-len if you are memory-constrained or want more KV-cache headroom at higher concurrency.
We recommend two loops as the default starting point for a good balance between accuracy and compute. Most of the accuracy gain from looping comes from the second loop, with smaller gains beyond that (see the loop-count comparison).
Use one loop when minimizing compute per token is the priority. Consider four loops when additional accuracy matters more, particularly for agentic tasks such as Terminal-Bench and SWE-bench (see benchmark results). Because higher loop counts can also reduce the number of generated tokens, evaluate the end-to-end latency and cost on your workload when choosing a setting.
To use a different loop count, update n_recurrent_loops in --hf-overrides in the command above.
The command above uses per-loop caching, which stores a separate KV cache for each recurrent loop, so KV cache memory grows with the loop count. Last-loop KV caching instead keeps a single cache per recurrent attention layer where each loop overwrites the previous loop's entries, so the cache holds only the final loop's keys and values. This keeps the KV cache footprint at the one-loop size for any loop count, leaving more capacity for longer contexts or concurrent requests.
To enable it, set recurrent_kv_cache_mode to "last_loop". In our experiments, the accuracy with last-loop caching stays close to the per-loop mode with nearly unchanged output lengths.
Note: Dory support in llama.cpp is available in ggml-org/llama.cpp#30306. Until it is included in a llama.cpp release, build from the Dory-enabled source below. This example pins revision 98a74c97ea09ac72f6ceda1559808fc0a2f447ce, used by the local macOS reference setup, rather than following the moving PR head.
These commands target native ARM64 macOS on Apple Silicon, not Intel or Rosetta. Install Homebrew and Apple's Command Line Tools (xcode-select --install) first.
brew install cmake openssl@3
git init llama.cpp-dory
cd llama.cpp-dory
git remote add origin https://github.com/brianchmielNV/llama.cpp.git
git fetch --depth 1 origin 98a74c97ea09ac72f6ceda1559808fc0a2f447ce
git checkout --detach FETCH_HEAD
HF_UI_VERSION=b11551 cmake -S . -B build \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_OSX_ARCHITECTURES=arm64 \
-DCMAKE_OSX_SYSROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX26.sdk" \
-DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON \
-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Apple \
-DLLAMA_BUILD_SERVER=ON \
-DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF -DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=ON \
-DLLAMA_OPENSSL=ON -DOPENSSL_ROOT_DIR="$(brew --prefix openssl@3)"
cmake --build build --parallel 4 --target llama-server
The build uses Metal and a prebuilt browser UI; no Python environment or JavaScript build is required to run the server.
llama.cpp loads GGUF files rather than the BF16 Hugging Face checkpoint directly. Download the reference dory-q5_k_m.gguf export (Q5_K_M) in your browser and place it in llama.cpp-dory/. This is a quantized export; the benchmark results above are for BF16.
The expected file size is 3,673,288,864 bytes. Verify its SHA-256 checksum before launching:
echo "96080be49ffd67b916018b7bdef8c484762576fb52780e10ab98ac4978a660e0 dory-q5_k_m.gguf" | shasum -a 256 -c -
From llama.cpp-dory/, start a local server with two loops and reasoning enabled:
./build/bin/llama-server \
-m ./dory-q5_k_m.gguf \
--alias dory \
--host 127.0.0.1 --port 8080 \
-ngl 999 \
--ctx-size 4096 --parallel 1 \
--batch-size 512 --ubatch-size 512 \
--cache-type-k f16 --cache-type-v f16 \
--flash-attn off \
--no-context-shift \
--jinja --reasoning on \
--temp 1.0 --top-p 0.95 --top-k 0 --min-p 0 --repeat-penalty 1 \
--override-kv "dory.num_loops=int:2,dory.recurrent_kv_cache_mode=str:per_loop"
Once the server is ready, open http://127.0.0.1:8080 on the same Mac for the chat interface. The server is local-only and unauthenticated. Its OpenAI-compatible API is at http://127.0.0.1:8080/v1, with model identifier dory from --alias.
Set dory.num_loops in --override-kv to 1, 2, 3, or 4. As with vLLM, start with two loops, use one loop to minimize compute per token, or consider four loops when accuracy matters more.
The command above uses per-loop KV caching. To keep the KV cache footprint at the one-loop size, replace its --override-kv argument with:
--override-kv "dory.num_loops=int:2,dory.recurrent_kv_cache_mode=str:last_loop"
The examples below use the OpenAI-compatible client. Recommended sampling settings are Temperature 1.0 and Top_P 0.95. Reasoning is always on in vLLM and enabled with --reasoning on in the llama.cpp command above. Choose the client configuration for your server before running the examples.
vLLM registers the model under the identifier passed to --model (nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16) since --served-model-name is not set. If you add a --served-model-name of your own, copy the identifier returned by GET /v1/models into MODEL below.
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
MODEL = "nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16"
MAX_TOKENS = 16000
For llama.cpp, use port 8080 and the dory alias instead. The smaller output limit below leaves room for the prompt in the 4,096-token context; always keep prompt, history, and generated tokens within --ctx-size.
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="EMPTY")
MODEL = "dory"
MAX_TOKENS = 2048
Chat
response = client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": "Write a haiku about GPUs"}],
max_tokens=MAX_TOKENS,
temperature=1.0,
top_p=0.95,
)
print(response.choices[0].message.content)
Streaming
stream = client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": "Explain how a transformer works in two sentences"}],
max_tokens=MAX_TOKENS,
temperature=1.0,
top_p=0.95,
stream=True,
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="", flush=True)
For vLLM, add the following to any serve command above:
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--reasoning-parser nemotron_v3
For llama.cpp, the launch command already includes --jinja for chat-template-based tool calling; the vLLM parser flags above are not needed.
NOTE: For coding agents, add extra_body={"chat_template_kwargs": {"force_nonempty_content": True}} to the API call, as shown below.
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
}]
response = client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": "What's the weather in Santa Clara?"}],
tools=tools,
max_tokens=MAX_TOKENS,
temperature=1.0,
top_p=0.95,
extra_body={"chat_template_kwargs": {"force_nonempty_content": True}},
)
print(response.choices[0].message.tool_calls)
The model was pre-trained with 6T tokens, using NVFP4 for the first ~4.5T tokens and BF16 for the remainder, followed by a context-extension phase of 300B tokens. It supports up to 256K context length.
Stage 1: Pre-Training
Stage 2: Supervised Fine-Tuning
Stage 3: Reinforcement Learning
NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 is a result of the above work.
Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
Data Modality: Text Training Data Size: 6 Trillion Tokens for pre-training; 300 Billion Tokens for context extension Dataset partition: Training [100%], testing [0%], validation [0%] Time period for training data collection: 2013 to December 2025 Time period for testing data collection: 2013 to December 2025 Time period for validation data collection: 2013 to December 2025 Data Collection Method by dataset: Hybrid: Automated, Manually-Collected, Synthetic Labeling Method by dataset: Hybrid: Automated, Manually-Labeled, Synthetic
NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 is pre-trained on a large corpus of high-quality curated and synthetically-generated data. It is trained in the English language and 43 programming languages. Our sources cover a variety of document types such as: webpages, dialogue, articles, and other written materials. The corpus spans domains including legal, math, science, finance, and more. We also include a small portion of question-answering, and alignment style data to improve model accuracy. The model was pre-trained for 6 trillion tokens, followed by 300 billion tokens of context extension.
The post-training corpus for NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 consists of high-quality curated and synthetically-generated data. The primary language used for post-training is English.
These datasets, such as FinePDFs, EssentialWeb, HotpotQA, SQuAD, and HelpSteer3, do not collectively or exhaustively represent all demographic groups (and proportionally therein). For instance, these datasets do not contain explicit mentions of demographic classes such as age, gender, or ethnicity in 64-99% of samples, depending on the source. In the subset where such terms are present, document-based datasets (FinePDFs and EssentialWeb) contain representational skews, such as references to "male" outnumbering those to "female", and mentions of "White" as the most frequent among ethnic identifiers (comprising 43-44% of ethnicity mentions). To mitigate these imbalances, we recommend considering evaluation techniques such as bias audits, fine-tuning with demographically balanced datasets, and mitigation strategies like counterfactual data augmentation to align with the desired model behavior. This evaluation used a 3,000-sample subset per dataset, identified as the optimal threshold for maximizing embedder accuracy.
During post-training, we generate synthetic data by distilling trajectories and solutions from strong teacher models and agent systems, often grounded in real tasks or documents and aggressively filtered for quality. For math, code, and science, we start from curated problem sets and use open source permissive models such as GPT-OSS-120B to produce step-by-step reasoning traces, candidate solutions, best-of-n selection traces, and verified CUDA kernels. For long-context and science, we build synthetic QA and reasoning data by retrieving passages from long documents, generating MCQ/OpenQA questions and answers, and paraphrasing them into multiple prompt/response formats to ensure diversity. Across all pipelines we stack automated verification—compilers, numerical checks, language identification—to ensure our data is high quality.
For all domains, we apply a unified data filtering pipeline to ensure that only high-quality, license-compliant, and verifiable samples are used for post-training. We first discard malformed examples using structural checks (e.g., missing tool definitions when tool calls are present). We then aggressively filter reasoning traces exhibiting pathological repetition, such as repeated n-grams within a sliding window or across the entire trajectory, which we found to be a strong indicator of malformed or low-quality reasoning. Finally, based on internal audits of synthetically generated datasets, we observed that some teacher models occasionally produce reasoning traces and final responses that implicitly align with specific political entities or promote nationalistic narratives. To mitigate this, we apply targeted keyword- and regex-based filters and remove all trajectories matching such behavior.
For more information about the datasets used to train this model, please see the Public Summary of Training Content
For detailed dataset information on the pre-training and reinforcement-learning data, please see the NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 model card.
Data Collection Method by dataset
Data Collection Method by dataset
In addition to this BF16 checkpoint, quantized versions of Dory1.0-4B are available for lower memory use and faster inference:
The table below reports measured inference speedups for Dory1.0-4B on NVIDIA DGX Spark relative to the BF16 baseline. MXFP8 (W8A16) uses 8-bit weights with 16-bit activations, while NVFP4 (W4A16) uses 4-bit weights with 16-bit activations.
| Input sequence length (ISL, tokens) | Output sequence length (OSL, tokens) | Batch size | MXFP8 (W8A16) speedup vs. BF16 | NVFP4 (W4A16) speedup vs. BF16 |
|---|---|---|---|---|
| 2560 | 300 | 1 | 1.69× | 2.17× |
| 128 | 1024 | 1 | 1.84× | 2.44× |
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
We advise against circumvention of any provided safety guardrails contained in the Model without a substantially similar guardrail appropriate for your use case. For more details: Safety and Explainability Subcards.
For more detailed information on ethical considerations for this model, please see the Model Card++ Bias, and Privacy Subcards.
Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.