Notes on the iteration loop we ran while fine-tuning F5-TTS and StyleTTS2 on a small Northern English corpus. The headline finding is that the listening test isn’t optional polish at the end — it’s the only measurement that catches the failure modes that matter, and each round of listening produces specific phonetic observations that map to specific engineering decisions.

This is a write-up of the methodology, with the concrete examples that forced each decision.

The loop

        ┌────────────────────────┐
        │  render passage        │
        │  (baseline + ft)       │
        └──────────┬─────────────┘
                   ▼
        ┌────────────────────────┐         a feature is "right" if a native
        │  human listens against │         speaker recognises it. Record both
        │  marker list (BATH,    │  ◀───── what's working AND what's broken;
        │  FOOT-STRUT, …)        │         both are signal.
        └──────────┬─────────────┘
                   ▼
        ┌────────────────────────┐         translate audible features
        │  diagnose: why is the  │         to training-side cause:
        │  output the way it is? │  ◀───── · missing accent → under-trained
        └──────────┬─────────────┘         · right accent + glitches → over-trained
                   ▼                       · wrong accent → data or LR direction
        ┌────────────────────────┐         specific knobs:
        │  pick next training    │         · lr ↑/↓ (drift per step)
        │  move                  │  ◀───── · epochs ±N (cumulative drift)
        └──────────┬─────────────┘         · earlier ckpt (rewind)
                   ▼                       · data filter (cleaner signal)
              [iterate]

The verdict-to-action mapping

Listening verdict What it implies physically Engineering response
“No discernible difference from baseline” Cumulative weight drift Σ lr·grad too small. Either lr too low, scheduler decayed it to ~0, or epochs too few. Increase lr or remove decay; add epochs.
“Accent is right but specific words mangled / dropped / truncated” Late-epoch overfitting on training-corpus pace or timing. Crossed from “learning the distribution” to “memorising peculiarities of small corpus”. Step back: pick an earlier checkpoint, or continue at lower lr.
“Accent is wrong direction (e.g. American instead of Northern)” Training data misattributed, or model pulled toward different distribution than expected. Audit data: manifest pointing at right speakers? Diarisation clean? Speaker IDs correct?
“Specific phonetic feature still missing (e.g. monophthongisation absent on ‘sunshine’)” That pattern needs more training-distribution exposure. Some accent features are easier than others. Train more, keeping lr constant. Don’t increase lr to chase one feature — risk catastrophic forgetting.
“Feature drifted past the target (e.g. ‘down’ → ‘doon’)” Over-fit on the broader cluster of related accents. Model has slid past the target sub-region. Step back to earlier checkpoint OR pick checkpoint before the drift.

These categories aren’t theoretical. We hit each of them in real training runs. Examples:

“No discernible difference” → LR scheduler decayed to zero

Run 1 of F5-TTS used the trainer’s default schedule: linear warmup to peak 1e-5, then linear decay across the entire run to ~0. After 5 epochs:

  • Mean loss per epoch: 0.629, 0.677, 0.648, 0.642, 0.670 — flat
  • Listening: indistinguishable from baseline
  • Numerical: waveform correlation with baseline = 0.017 (essentially uncorrelated audio, as expected for diffusion sampling) — looked like the model was doing something, but the perceptual output disagreed

Diagnosis: the schedule shape was wrong. Step-by-step LR values:

  • step 1: lr = 1e-7 (warmup)
  • step 100: lr = 1e-5 (peak, decay starts)
  • step 1000: ≈ 5.5e-6
  • step 2225: lr = 1e-13 — effectively zero

Total weight drift is bounded by Σ lr·grad. With LR linearly decaying to ~0 over 2225 steps, late-epoch gradients are multiplied by near-zero values. Most of run 1’s “5 epochs” was a no-op.

Run 2 fix: 5× higher peak LR (5e-5), constant after warmup, no decay. 10 epochs. Result: per-epoch mean loss decreased (0.701 → 0.683 → 0.661 → 0.646 across the first 4), and listening verdict was audibly Northern — “London” rendered as “Lundun” (FOOT-STRUT vowel collapse, a textbook Northern marker).

“Specific feature missing” → Train more, same LR

After 4 epochs of run 2, FOOT-STRUT had emerged (“Lundun”) but monophthongisation hadn’t (“sunshine” still diphthongised standard). Some phonetic patterns are easier to acquire than others — single-vowel substitutions vs global diphthong→monophthong shifts.

Continuing 6 more epochs at the same constant 5e-5: monophthongisation strengthened (“laughing” landed correctly), but truncation appeared (“sunshine” → “sunshinn”, dropped function words like “her”).

“Accent right but words mangled” → Pick earlier checkpoint

Run 2 epoch 10 had the strongest accent but the most word-truncation. Rendering epochs 6, 7, 8, 9 with the same input passage and listening through revealed epoch 9 as the sweet spot — accent committed, mostly without truncation. Final shipping checkpoint.

“Drifted past the target” → StyleTTS2’s late-epoch failure mode

StyleTTS2 epoch 5 introduced “down” → “doon” (Geordie/Scots realisation). That’s more Northern than the Bolton/Lancashire target. The model had slid past the target sub-region of accent space and was now drifting toward broader Scots/North-East phonetics. Stopped training; epoch 4 became the shipping checkpoint.

Why loss alone can’t replace listening

Three reasons:

  1. Loss flatness is ambiguous. A flat loss curve could mean “converged” or “not learning at all.” Run 1’s flat 0.65 was the latter; only listening (“indistinguishable from baseline”) disambiguated and pointed at the LR scheduler. No purely numerical metric on training loss could distinguish those two cases without an evaluation set.

  2. Some failures look like wins on the loss curve. Late-epoch overfitting drops training loss while degrading output. Lower loss + worse output. Only listening catches it.

  3. The thing being optimised isn’t what you actually want. Flow-matching loss measures velocity-field reconstruction quality on the training distribution. It doesn’t directly measure “is this output Northern English-sounding to a native speaker.” The model can get better at fitting Sara’s training mels while producing audio that sounds different from any actual Sara recording.

This is why every training run produces multiple per-epoch checkpoints and we render the same passage through several of them. The cost (~30s per render × 5–6 epochs = ~3 min) buys you a perceptual gradient across training time that no scalar loss provides.

The phonetic-marker passage as deliberate probe

The test passage is loaded with English-accent markers so a single rendering surfaces multiple aspects of the model’s state. Our standard probe:

“It was a bright morning when the path through the grass led down to the running water. She ran her hand along the back of the chair before sitting down. The young children were laughing in the sunshine, dancing in patterns through the warm afternoon. After tea, the family walked up the hill to look at the view. One of them said, with a small smile: I cannot believe how lovely it is, our little corner of the world.”

What we’re probing Words that probe it “Wrong” sounds like “Northern” sounds like
BATH vowel path, grass, laughing, dancing, after, cannot /pɑːθ/ (RP “parth”) /pæθ/ (rhymes with “trap”)
FOOT-STRUT running, sunshine, hand, hill, up, lovely, our distinct “put”/”putt” collapsed: both /ʊ/, so “London” → “Lundun”
Diphthong→monophthong sunshine (→sunshaan), morning, smile standard /aɪ/, /eɪ/ flat, longer single vowel /aː/
happY-tense lovely, family, every, country tense /iː/ laxer, more /ɪ/-like
R-intrusion / linking chair before, our little none often realised in connected Northern speech

If only some markers come through, that tells us which kinds of changes the model is finding easier vs harder to learn. In run 2 epoch 4 the FOOT-STRUT shift had emerged (“Lundun”) but monophthongisation had not (“sunshine” still diphthongised). That gap motivated continuing training rather than declaring done — specific phonetic gaps mapping to specific training decisions.

Practical recommendations

  1. Save a checkpoint per epoch. They’re cheap to disk and you’ll want the perceptual gradient across training time. Late-epoch isn’t always best.

  2. Curate one phonetic-marker passage that targets the dialect features you care about. Reuse the same passage every render so you build a listening-memory of the model’s progression.

  3. Render with the same reference clip every time. The only variable should be the model weights. If you change the reference clip you’re asking two different questions at once.

  4. Native-speaker listeners are the most reliable test instrument. Their judgement catches features that numerical metrics miss — and importantly, also catches over-fitting failures that look fine numerically.

  5. Both wins and bugs are signal. Don’t just record what’s working; record what’s broken. The combination of “what improved” and “what got worse” defines the engineering response (continue / step back / change data).

  6. Run more checkpoints than you think you need. A/B-ing 6 different epochs of the same run takes 3 minutes of compute. The information gain — perceptual gradient over training time — is worth far more than that.

Provenance

Worked example from a small TTS fine-tuning project: ~3 hours of single-speaker British (Bolton-area) audio + WhisperCPP for transcripts → fine-tuned F5-TTS and StyleTTS2 producing recognisably Northern-English output. Both architectures hit different late-epoch failure modes that only the listening loop caught. The companion piece F5 vs StyleTTS2 architecture trade-off documents what those failure modes implied about the architectures themselves.