The smallest working Drama 3. Unofficial int4 quantization of fishaudio/drama-3 — Fish Audio's 35B (3B active) multi-speaker TTS model — with real eval numbers, one-command serving, and a loader patch for SGLang-Omni.
fp8e4nv kernels);
our int4 kernels (moe_wna16 + GPTQ-Marlin) don't[sighs]/[laughs] tags, long-form, Chinese| prompt | clip |
|---|---|
| plain TTS | int4_plain.wav |
| voice clone (reference clip) | int4_clone.wav |
clone + [sighs]/[laughs] tags | int4_tags.wav |
| long-form | int4_long.wav |
| Chinese | int4_zh.wav |
Compared against the strongest community quant, darioooooo0o/drama-3-w4a16-32gb (int8 dense layers, 32 GB), same prompts, same scorer, one A100:
| mean WER ↓ | mean WER excl. zh¹ ↓ | speaker sim ↑ | |
|---|---|---|---|
| this repo (int4 hybrid, 22.1 GB) | 0.197 | 0.082 | 0.846 |
| theirs (int8 dense, 24.5 GB) | 0.259 | 0.154 | 0.846 |
¹ The Chinese prompt scored WER 1.0 for both checkpoints despite nearly identical transcripts —
a jiwer tokenization artifact on unspaced Chinese, not a model difference. Full per-prompt rows in
eval/eval_results.json; clips in eval/.
Notable per-prompt outcomes: their build hallucinated an ending on the long paragraph (WER 0.667 vs our 0.028); ours truncated the tag line's tail on one take (0.333 vs their 0.25). Speaker similarity is a tie.
git clone https://github.com/sgl-project/sglang-omni.git && cd sglang-omni
uv venv -p 3.12 .venv && . .venv/bin/activate
uv pip install --prerelease=allow -e ".[fishaudio]"
uv pip install ninja # Marlin repack JIT-builds on first load
hf download akhaliq/drama-3-int4 --local-dir drama-3-int4
python drama-3-int4/apply_patch.py . # idempotent; registers the drama3-int4 quant method
sgl-omni serve --model-path drama-3-int4 --port 8000 \
--tts_engine.gpu_memory_fraction 0.62 --vocoder.gpu_memory_fraction 0.2
Then the standard Drama 3 API:
/v1/audio/speech with input, optional references for voice cloning, [sighs]/[laughs] tags,
multi-speaker scenes, and previous_takes editing all work as on the original.
Measured serving memory: 57.3 GB GPU at startup, 57.9 GB peak across the five sample generations
(A100-80GB, flags above). On smaller cards, lower tts_engine.gpu_memory_fraction and
vocoder.gpu_memory_fraction proportionally; the weights themselves are 22.1 GB.
| component | original | here | why |
|---|---|---|---|
| text embeddings, lm_head, router | bf16 | bf16 | quality-critical, small |
| attention q/k/v/o (full-attn layers) | bf16 | int4 | big, RTN-tolerant |
| deltanet in/out projections | bf16 | int4 | big, RTN-tolerant |
| shared-expert MLP | bf16 | bf16 | int4 passed all numerics checks but produced zero audio frames at serve time (see shared_expert_verdict.md) |
| routed MoE experts (30,720 matrices) | FP8 (block) | int4 | the bulk: 32.2 GB → 16.6 GB |
| audio decoder + codec | bf16 | bf16 | the audio path stays lossless |
Method: RTN + per-group clip search (group 128, symmetric), no calibration data. GPTQ packing for
SGLang's moe_wna16 kernel (experts) and standard GPTQ v1 fields repacked by GPTQ-Marlin at load
(dense). Reproduce with quantize.py; load with apply_patch.py
— the patch also fixes fused-name matching (gate_up_proj/qkv_proj/in_proj_qkvz), without which
fused modules silently miss their int4 weights and generation returns zero frames (the v3 fix).
Stats in quant_stats.json.
quantize.py.fp8e4nv), unavailable here.Derivative Work of Fish Audio Materials under the Fish Audio Community License v1.0 (LICENSE) — same terms as the original model, including: research/non-commercial use free; commercial use only under USD 500k annual revenue; no hosted TTS/voice-cloning service without a separate license; no cloning real people's voices without consent; do not remove watermarks; do not use outputs to train other speech or foundation models.