K5 Kazakh–Russian STT (streaming, 600M)

Speech recognition for Kazakh, Russian and mixed Kazakh↔Russian speech:

• Model: a 600M Conformer-CTC, fine-tuned from GigaAM-Multilingual large_ctc on 2,364 h of open data, with phone-line augmentation and kk↔ru mixes.
• Two-pass streaming:
  • partials come from a chunk-aware model on a cache-aware engine (a chunk every 320 or 640 ms; committed text never changes);
  • finals at each pause come from the offline model, plus a KenLM 4-gram and Kazakh/Russian inverse text normalization (digits, dates, money).

Quick start

Offline:

from transformers import AutoModel
model = AutoModel.from_pretrained("alibiserikbay/k5-kazakh-russian-stt", trust_remote_code=True)
print(model.transcribe("audio.wav").text)          # <= 30 s; spoken-form text, lowercase

Streaming, two-pass, with LM, ITN and hotwords:

huggingface-cli download alibiserikbay/k5-kazakh-russian-stt --local-dir k5stt
pip install torch torchaudio transformers safetensors hydra-core omegaconf sentencepiece onnxruntime pyctcdecode kenlm soundfile scipy
python k5stt/deploy/transcribe_stream.py call.wav --chunk-ms 320 --lm beam --hotwords "Ерболат,Kaspi"
arecord -f S16_LE -r 16000 -c 1 | python k5stt/deploy/transcribe_stream.py - --lm beam --tentative     # live microphone

For live, segmented use pass --lm beam (beam search without KenLM; hotwords still work): on 100 simulated calls with a 0.4 s end-of-utterance pause the final WER is 10.3 vs 11.1 with KenLM. KenLM still helps on whole utterances (offline numbers below); on long recordings at the default 0.5 s pause the two are equal (9.12 vs 9.17).

The CLI prints JSON lines: {"type": "partial", "text": ...} while you speak, and {"type": "final", "text": ..., "text_itn": ...} at each pause. In Python, use deploy/twopass.py: TwoPass(paths=release_paths(root)).feed(samples).

Results

WER %. Bootstrap 95% CIs (2,000 resamples) where shown. All systems were scored on the same utterances with the same normalizer: lowercase, punctuation stripped, digits kept.

Against open Kazakh/Russian models

• FLEURS kk, paired differences vs K5 + KenLM:
  • shyngys879: +2.44 [+1.91, +2.95];
  • sybyrla: +2.49 [+1.22, +4.30];
  • omniASR CTC-1B: +4.19 [+3.10, +5.23].
• Russian FLEURS: Whisper writes numbers as digits, as those references do, so it leads on all WER. On digit-free utterances K5 is better.
• Decoding of the external models: greedy on CPU; the Whisper models used temperature fallback.

Against the earlier kazakh-russian-mixed-stt (wav2vec2 + KenLM)

WER on the full test sets:

Streaming (two-pass), simulated calls

100 dev "calls", each three utterances with pauses, for 4,308 words in total.

• Final, live segmentation: 9.03 vs 8.84 offline, Δ +0.19 [−0.41, +0.80]. With --eou 0.8 it is 8.89, at +0.3 s.
• Partial lag (320 ms chunks, 4 CPU threads):
  • 0.49 s with a tentative last word, of which 5.7% get corrected;
  • 0.72 s stable-only, never corrected.
  • On GPU ≈0.39 / 0.62 s.
• Partial accuracy is the weak spot. With no lookahead the partials are clearly worse than the finals (FLEURS kk 12.7 at 640 ms, 14.2 at 320 ms; spontaneous speech 30–35). Use the finals as the transcript.

Capacity: ~16 real-time streams per H100 at 640 ms chunks with the per-stream engine. batch_engine.py runs all streams in one forward per tick (2.1–2.6× throughput on CPU).

Extras

• ITN (kk_tn): FLEURS kk 9.28 → 6.36 against digit-written references. Example: "екі мың он тоғызыншы жылы" → "2019 жылы".
• Hotwords (--hotwords):
  • Recall of rare names in the test went from 24% to 80%.
  • Other speech was unchanged: 7.36 → 7.43, with 0 false alarms from 200 distractor entries.

Intended use and limitations

• Intended use: Kazakh, Russian and code-switched kk↔ru transcription, offline or streaming (e.g. call-centre style dictation), on CPU or GPU.
• No real call-centre data was used. Telephony is simulated (band-pass, 8 kHz, G.711, GSM, Opus; no AMR-NB). Expect a gap on real phone calls, and evaluate on your own audio.
• Output is lowercase spoken form. Use kk_tn ITN for digits. It does not add punctuation.
• Partials vs lookahead: partials have no lookahead. Proper lookahead needs Emformer-style training, which is not included.
• Segment length: keep segments ≤ 30 s. The streaming pipeline force-cuts at 20 s.
• Test-set overlap: KSC2 and FLEURS train splits were used in training, as they also were for the GigaAM base. We decontaminated the LM text against FLEURS and FLoRes dev/test by 8-gram overlap.

Training data and licences

Acoustic model:

• ISSAI KSC2 (CC BY 4.0, 30% replay)
• KSD, Al-Farabi KazNU, OpenSLR 140 (CC BY-SA 3.0)
• Google FLEURS kk/ru train (CC BY 4.0; FLoRes text CC BY-SA 4.0)
• SberDevices Golos (SberDevices public licence with attribution and conditions reserved)
• SOVA RuDevices (CC BY 4.0)
• Russian LibriSpeech, OpenSLR 96 (public domain in the US)
• Augmentation: RIRS_NOISES (Apache-2.0), MUSAN noise (CC BY 4.0)

KenLM: CC-100 kk/ru (no licence grant; Common Crawl Terms of Use apply) plus transcripts of the corpora above.

Licence:

• CC BY-SA 4.0 for the model weights and KenLM files, because some training inputs are share-alike (KSD, Golos, FLoRes text).
• modeling_gigaam.py: MIT, © 2024 GigaChat Team. Changed so that the pyannote import is optional.
• deploy/ and kk_tn: Apache-2.0.
• vad/vad.onnx: Silero VAD, MIT.
• Details in NOTICE.md and LICENSES.md.

Required attributions:

• KSC2 © ISSAI, DOI 10.48342/m90y-aj02.
• KSD © Al-Farabi Kazakh National University, CC BY-SA 3.0. Resampled to 16 kHz and text-normalised.
• Golos © SberDevices / Salute Developers, used under its licence (https://github.com/salute-developers/golos). Resampled, normalised, telephony-augmented and mixed. Provided as is, without warranty.
• FLEURS © Google, CC BY 4.0.
• SOVA © Virtual Assistant, LLC, CC BY 4.0.
• MUSAN CC BY 4.0. RIRS_NOISES Apache-2.0.
• GigaAM-Multilingual © 2024 GigaChat Team, MIT.
• Silero VAD © Silero Team, MIT.

Citations

• GigaAM-Multilingual: arXiv 2607.10371
• KSC2: Mussakhojayeva et al., Interspeech 2022
• KazakhTTS2: LREC 2022
• KSD: Kadyrbek, Mansurova et al., BDCC 2023
• FLEURS: Conneau et al., 2022
• FLoRes-200: NLLB Team, 2022
• Golos: Karpov et al., Interspeech 2021
• SOVA dataset
• Russian LibriSpeech (OpenSLR 96)
• MUSAN: Snyder et al., 2015
• RIRS_NOISES: Ko et al., 2017
• CC-100: Conneau et al., 2020; Wenzek et al., 2020
• KenLM: Heafield, 2011

Қысқаша (қазақша)

K5 — қазақ, орыс және аралас қазақ↔орыс сөйлеуге арналған ағынды (streaming) сөйлеуді тану моделі (600M, GigaAM-Multilingual негізінде). Жылдам аралық мәтін (partial) chunk-моделінен, ал әр кідірістегі соңғы мәтін офлайн модельден + KenLM-нен + сандарды қалпына келтіруден (ITN) шығады. FLEURS kk: WER 9.14 (ашық модельдердің ішінде ең жақсысы). Нақты қоңырау орталығының деректері қолданылмаған — өз аудиоңызда тексеріңіз.

Кратко (по-русски)

K5 — потоковое распознавание речи для казахского, русского и смешанной казахско-русской речи (600M, на основе GigaAM-Multilingual). Промежуточные гипотезы даёт чанковая модель, финальный текст на каждой паузе — офлайн-модель + KenLM + обратная нормализация чисел (ITN). FLEURS kk: WER 9.14 — лучше всех открытых моделей, которые мы проверили; на смешанной речи KSC2 — 11.19. Реальные записи колл-центра в обучении не использовались — проверьте на своих данных.