svg

Menu

Case Study · Indic Voice AI & Speech Evaluations

dhvani-eval

An open-source evaluation suite and streaming telephony degradation simulator for Indic & low-resource voice AI (Hindi, Maithili, Bhojpuri, code-mixed Hinglish). Quantifies speech model resilience under real rural telephony constraints.

RepositoryPython 3.14 · Devanagari Normalization · 8kHz Telephony Sim

The Real-World Dilemma

Standard speech recognition (ASR) benchmarks (like LibriSpeech) evaluate pristine, studio-recorded English audio. But when deploying voice assistants in rural India (for welfare schemes like JanSahay or farmer helplines like Gram Vaani), audio arrives over lossy 2G/EDGE cellular networks, 8kHz telephony codecs (G.711 / AMR), noisy village backgrounds, and code-mixed vernacular dialects (Hindi, Maithili, Bhojpuri). Existing benchmark suites completely ignore these channel physics.

Key Architectural Features

  • Devanagari Unicode Normalization: Solves subtle Unicode rendering ambiguities in Hindi script — normalizes nuktas, halants, matra permutations, and Chandrabindu equivalences so phonetic matches aren't penalized by arbitrary font encodings.
  • Telephony Channel Degradation Engine: Emulates 8kHz bandpass downsampling, AMR narrowband codecs, burst packet dropout (simulating mobile handoffs in rural towers), and ambient room/traffic noise injection.
  • Streaming Voice Latency Profiler: Measures Time-to-First-Token (TTFT), End-of-Utterance (EOU) silence cutoffs, and chunk streaming throughput — essential metrics for real-time interactive voice bots.
  • Resilience Degradation Scoring: Computes the exact delta in Word Error Rate (ΔWER) between clean audio and telephony-degraded audio, revealing which models collapse when bandwidth drops.

Target Alignment

Directly targeted at frontier speech and evaluation teams: SuperKalam (AI Applied Engineer - Voice First), Karya (Research Intern - AI Evaluations), Gram Vaani (Voice AI & NLP), and CivicDataLab.