TKA KWS — French (Zipformer transducer, ~3.3M, streaming, sherpa-onnx)

Open-vocabulary, streaming keyword spotter for French, compatible with sherpa-onnx. It detects short safety / distress phrases (e.g. arrête, lâche-moi, je suis une policière) in continuous audio, on-device, at ~320 ms latency. It is the French analogue of the published sherpa-onnx-kws-zipformer-gigaspeech-3.3M (English) model, built with the same icefall Zipformer-transducer KWS recipe.

Keywords are runtime-configurable — changing the phrase list only means regenerating keywords.txt; no retraining.

Files

encoder-epoch-20-avg-2-chunk-16-left-64.onnx      # + .int8.onnx
decoder-epoch-20-avg-2-chunk-16-left-64.onnx      # + .int8.onnx
joiner-epoch-20-avg-2-chunk-16-left-64.onnx       # + .int8.onnx
tokens.txt        # token table (required at runtime)
bpe.model         # SentencePiece model (to regenerate keywords.txt)
keywords.txt      # deployed keyword list (boost/threshold baked in)

All of encoder + decoder + joiner are required. For int8 deployment, int8 encoder + joiner with fp32 or int8 decoder both work.

Architecture

Streaming Zipformer transducer, ≈3.3 M parameters: 6×1-layer encoder, encoder/ff dims 128/192, stateless decoder (dim 320), joiner (dim 320), causal with chunk-size 16, left-context-frames 64. Input: 80-dim log-mel fbank, 16 kHz mono. See MODEL_DESCRIPTION.md for the full breakdown.

Usage (sherpa-onnx CLI)

./build/bin/sherpa-onnx-keyword-spotter \
  --encoder=encoder-epoch-20-avg-2-chunk-16-left-64.onnx \
  --decoder=decoder-epoch-20-avg-2-chunk-16-left-64.onnx \
  --joiner=joiner-epoch-20-avg-2-chunk-16-left-64.onnx \
  --tokens=tokens.txt \
  --keywords-file=keywords.txt \
  input.wav          # single-channel, 16-bit PCM

Python (sherpa_onnx.KeywordSpotter) and the C/C++/Java/Swift/Go APIs load the same files. Detections return the keyword's @tag; multi-word phrases come back underscored (je_suis_une_policière) — map _ → space in your app.

Customising keywords

Each line of keywords.txt is a BPE-encoded phrase with optional per-keyword boost (:) and threshold (#) and the original tag (@, no spaces):

▁ar r ête :1.0 #0.2 @arrête

Regenerate from raw phrases (one per line, spaces → _ in the @tag):

sherpa-onnx-cli text2token \
  --tokens tokens.txt --tokens-type bpe --bpe-model bpe.model \
  keywords_raw.txt keywords.txt

Training data

Multilingual LibriSpeech (MLS) French, 300 h, plus Piper-TTS keyword augmentation (each production keyword synthesized 400× across several voices, 60% inside carrier sentences, ×0.85–1.15 speed, MUSAN noise at 0–20 dB SNR). Trained 20 epochs (fp16) on one A10G with the pinned icefall gigaspeech KWS zipformer recipe; exported streaming + int8. Full recipe in MODEL_DESCRIPTION.md.

Evaluation

On the ground-truth clip Final-TKA-Audio.mp3 (536.7 s), swept over boosting × trigger threshold, scoring recall over spoken safety keywords vs. false positives over distractors (same discipline as the project's CLAP report):

Graphs	Best zero-FP recall	False positives	Deployed operating point
fp32	92 % (11/12)	0	boost 1.0, threshold 0.20
int8	92 % (11/12)	0	boost 1.0, threshold 0.20

At the operating point the model detects all 11 deployed French safety keywords; the single uncounted item is a benign control phrase in the eval set (je veux des bananes) that is not a deployed keyword — so recall over the deployed phrases is 11/11 on this clip. int8 matches fp32 (and is marginally more robust across the grid), confirming int8 is safe to ship.

Reference bar: the pretrained English gigaspeech-3.3M model scored 100 % / 0 FP on the same clip.

Full grids, int8 comparison and methodology in TEST_RESULTS.md.

Limitations & responsible use

Not an emergency service — false negatives/positives occur; do not use as a sole safeguard. Trained largely on read audiobook speech + synthetic keywords, so spontaneous/far-field/very-noisy or strongly-accented speech may differ. Numbers come from one clip and a small distractor set (indicative, not a broad benchmark). Verify on your target population and dialects before deployment.

License & attributions

Code/recipe: icefall & sherpa-onnx (Apache-2.0). Training data: MLS (CC BY 4.0, derived from LibriVox) and MUSAN; keyword audio synthesized with Piper voices. Verify the license of each specific Piper voice and confirm attribution requirements before redistribution. license: above is a suggested default — adjust to match your data/voice obligations.

Acknowledgements

Built on next-gen Kaldi (icefall, sherpa-onnx), following the open-vocab KWS approach discussed in icefall issue #2077 and sherpa-onnx issue #920.