Engineering Tone: How an Open-Weight Text-to-Speech Architecture

Share
Engineering Tone: How an Open-Weight Text-to-Speech Architecture

Was Adapted for Yorùbá

Executive Overview

In the realm of open-weight text-to-speech (TTS) systems, pitch is traditionally treated as linguistic seasoning. In mainstream Western languages like English, a stressed syllable or a downward inflection at the end of a sentence alters emotional intent or pragmatic meaning, but the underlying propositional content—the semantic identity of the words—remains intact. Listeners easily grasp communication despite erratic intonation.

Yorùbá, a tonal Niger-Congo language spoken by tens of millions across West Africa, operates under an entirely different paradigm. In Yorùbá, pitch is part of the spelling; it is a foundational, lexical property. Consider the orthographic sequence o-k-o (augmented with underdots as ọkọ). Depending on the localized tonal contour—whether high, mid, or low—this exact letter sequence transforms into vastly different semantic entries: husband, vehicle, spear, or farm.

When an English-trained neural text-to-speech model approaches such a structure without architectural intervention, it relies on probabilistic guesses that often apply arbitrary rising or falling melodies. The result is structural failure: the model articulates the wrong word entirely, rendering the speech unintelligible or semantically distorted.

To solve this systemic deficiency, an independent developer recently adapted the Kokoro and StyleTTS2 architectures to speak Yorùbá fluently. Utilizing a single Google Colab NVIDIA L4 GPU equipped with 22 GB of VRAM and a meticulously curated dataset of just 2.70 hours of cleaned audio, the project surmounted severe hurdles—ranging from dropped tones and raspy noise to PyTorch 2.6 crashes and infrastructure deployment roadblocks. This technical chronicle explores the precise methodology, engineering workarounds, data curation realities, and remaining architectural frontiers of this groundbreaking Yorùbá TTS implementation.


Detailed Chronology: From G2P Hurdles to Neural Alignment

Building oro-g2p and Resolving Token Slot Constraints

Before touching neural network weights or modifying hidden layers, the project demanded a grapheme-to-phoneme (G2P) converter that understood the precise phonology of Yorùbá. Standard English-focused phonemizers fail instantly. In Yorùbá orthography, digraphs like gb represent a single labial-velar plosive sound (/ɡ͡b/), while the letter p is traditionally pronounced as a voiceless labial-velar plosive (/k͡p/). Furthermore, nasal consonants such as n or m nasalize any subsequent vowels, and syllabic nasals dynamically shift their point of articulation depending on adjacent consonants, even in complex stacked configurations like ǹkan.

To avoid forcing users to install brittle, heavy system dependencies like espeak-ng, the developer engineered oro-g2p, a zero-dependency Python library installable via a straightforward package manager command:

from oro_g2p import g2p

print(g2p("ọkọ̀ ọ̀rọ̀"))
# 'ɔ̄ kɔ̄̀ ɔ̀ rɔ̀'

While converting text into International Phonetic Alphabet (IPA) representations complete with explicit tone marks (High, Mid, Low) proved relatively straightforward, integrating this output into the Kokoro architecture presented an immediate roadblock.

An initial naive implementation emitted vowels and their corresponding tone markers as separate, distinct tokens—for instance, structuring a sequence as ["a", "LOW"]. Within StyleTTS2, this approach failed catastrophically. The internal duration predictor interpreted the tone mark as an independent lexical character, unnaturally stretching trailing tones at phrase boundaries or severing the vital acoustic link between a vowel and the pitch contour it was engineered to sustain.

Rather than resizing Kokoro’s embedding table (n_token = 178), which would have broken direct weight loading from pretrained checkpoints, an audit of BASE_KOKORO_VOCAB was conducted. Slot 0 was confirmed as reserved for padding, beginning-of-sentence (BOS), and end-of-sentence (EOS) tokens, while 114 slots were occupied by Kokoro’s base tokens. This left precisely 63 completely empty, unused slots sitting within the 1 to 177 ID range.

The solution was atomic tokenization. Vowels and their matching tones, syllabic nasals and tones, and complex digraphs were packed directly into these 63 spare slots as single atomic integer IDs (e.g., "kp": 122, "ɡ͡b": 124, "d͡ʒ": 82, "ā": 7, "à": 26, "ǹ": 105). Coupled with a greedy, longest-first tokenizer, and keeping n_token locked at 178, all pretrained tensor shapes aligned perfectly without altering the underlying model graph.

Rectifying Data Pipeline Bottlenecks

Early training runs produced unintelligible output, prompting an initial misplaced frustration with hyperparameters rather than the training data itself. The primary culprit was StyleTTS2’s default dataloader cropping mechanism. By default, the pipeline crops long audio segments down to a maximum length (max_len=120, roughly 1.44 seconds) while continuing to feed the complete corresponding text transcript.

Because many source audio clips averaged between 6 and 11 seconds, the model was essentially forced to listen to 1.4 seconds of audio while attempting to align it against a 10-second sentence. Once leading and trailing silences were rigorously trimmed (top_db=35) and the training set was filtered to retain only clips spanning 1.2 to 5.8 seconds, cross-modal alignment snapped into place.

A secondary complication involved speaker identification configurations. StyleTTS2 pulls a secondary random audio clip from the exact same speaker_id during training to calculate pitch baselines and vocal style vectors. Initially, multiple distinct voices were lumped under speaker_id = 0, causing the internal pitch predictor to receive conflicting references within every single batch. Enforcing strict, discrete speaker segregation instantly eliminated random, erratic pitch leaps.

Inference-time debugging also revealed a critical quirk regarding iSTFTNet vocoders: stripping predicted silence frames before passing feature representations into the decoder destabilizes InstanceNorm1d normalization layers, resulting in audio heavily saturated with raspy static. Waveform silence trimming must occur strictly after the decoder has reconstructed the audio waveform.


Supporting Context & Metrics

Dataset Composition and Training Split

Following extensive data cleaning, filtering, and normalization, the final Gold training corpus comprised 2.70 hours of high-fidelity audio, scaling across 2,168 individual clips. This dataset was aggregated from prominent open-access corpora:

  • AfriSpeech/open-bible-speech-african (distributed under the CC BY-SA 4.0 license)
  • google/WaxalNLP (distributed under CC-BY and CC-BY-SA-4.0 licenses)

Framework Patching for Modern Infrastructure

Executing Stage 2 training routines on Google Colab leveraging modern PyTorch 2.6+ environments necessitated critical upstream interventions. Several framework-level incompatibilities were resolved and pushed upstream via pull requests and issue documentation:

  • Addressed torch.load safety and weight-loading schema updates.
  • Resolved tensor stride and contiguous memory layout crashes during adversarial discriminator updates.
  • Fixed device-placement mismatches occurring during style vector interpolation steps.

Checkpoint Compression and Optimization

The raw checkpoint file (epoch_2nd_00014.pth) scaled to an unwieldy 2,091.5 MB on disk because it encapsulated internal training discriminators, gradient buffers, and full optimizer states. By filtering the state dictionary down strictly to necessary inference modules, the final release checkpoint (kokoro_yoruba_gold.pth) was compressed by approximately 75%, shrinking to a lean 526.0 MB.


Official Implementation Reference

For developers and researchers seeking to integrate or inspect the codebase, the project is structured across modular, dedicated repositories:

from tts_yoruba import YorubaTTS

tts = YorubaTTS(styletts2_dir="./StyleTTS2")
tts.speak(
    "Ọkọ̀ ọ̀rọ̀ àrà ní o! Báwo ni àwọn ará ilé?",
    voice="spk8",
    out_path="greeting.wav",
)

Future Outlook and Version 2 Roadmap

While the current Version 1 release successfully generates intelligible, tonally accurate Yorùbá speech on consumer and cloud hardware, clear limitations remain. Multi-speaker generalization degrades when processing long compound nouns, and stylistic variance across rapid colloquial speech occasionally introduces minor vocal tremors.

Looking toward Version 2, the development roadmap targets several ambitious scaling objectives:

  1. Dataset Expansion: Scale training data past the current 2.70-hour baseline toward 15+ hours of balanced, multi-speaker, studio-grade Yorùbá audio.
  2. Native PLBERT Training: Train a dedicated, native Yorùbá PLBERT representation model to ensure extended compound phrases maintain proper contour tone cohesion over time.
  3. Automated Preprocessing: Integrate robust auto-diacritization pipelines and native English code-switching detection to process raw, unpunctuated text inputs seamlessly before executing the G2P conversion layer.

As the open-source community continues to push the boundaries of speech synthesis for low-resource and tonal languages, methodologies like atomic token mapping and rigorous data alignment provide a viable blueprint for overcoming architectural bias in modern deep learning frameworks.

Did you find this story helpful?

Share it with your friends and colleagues on social media.

Share