ctc-speech-recognition

Utterance-to-Phoneme Speech Recognition (CTC)

A CNN + pyramidal BiLSTM encoder trained with CTC loss to map variable-length speech feature sequences directly to phoneme sequences — no forced alignment required.

Problem

Map a variable-length sequence of speech features to a variable-length sequence of phonemes, without knowing the frame-to-phoneme alignment ahead of time — the core challenge CTC (Connectionist Temporal Classification) loss is designed to solve.

• Input: 28-dimensional MFCC features per frame (LibriSpeech train-clean-100)
• Output: 41 phoneme classes (ARPABET-mapped) + CTC blank token
• Data: 28,539 training utterances, 2,703 validation utterances

Approach

Encoder pipeline:

MFCCs → 3× ResNet-style CNN blocks → 3-layer BiLSTM (256 hidden)
      → 2× pyramidal BiLSTM (2× downsampling each, 4× total) → MLP head (1024→512→41)

• CNN frontend: ResNet-style blocks with skip connections extract local features and stabilize deeper training before the recurrent stack.
• Pyramidal BiLSTM (pBLSTM): halves the time resolution at each of 2 layers (4× total downsampling), which shrinks the CTC alignment search space and makes training far more tractable.
• Decoder head: MLP (1024→512→41) with LayerNorm/BatchNorm, log-softmax output.
• Size: 12.9M parameters.

Training setup:

Component	Choice
Optimizer	AdamW (weight decay)
LR	~8e-4–1e-3; ReduceLROnPlateau initially, later switched to CosineAnnealingLR
Loss	nn.CTCLoss (zero_infinity=True, mean reduction)
Decoding	Beam search (cuda_ctc_decoder), beam width 3
Augmentation	SpecAugment (time + frequency masking)
Epochs	60
Batch size	128–256 (varied across config iterations)

Results

Metric	Score
Best validation Levenshtein distance (beam search)	4.39
Held-out test distance (greedy decoding)	5.27

The gap between validation (beam search) and test (greedy) performance is explained below — it's a real infrastructure finding, not a modeling gap.

Key decisions, debugging, and hindsight

• Biggest lever: the ResNet CNN frontend combined with pyramidal BiLSTM downsampling — skip connections stabilized deeper training, and 4× temporal downsampling made the CTC alignment problem far more tractable.
• Biggest debugging challenge: validation distance got stuck around 130 (near-random output), caused by three compounding bugs found together:
  1. A duplicated self.BLSTMs layer definition in the encoder — the second definition silently overwrote the first, so the intended layer never actually ran.
  2. Over-aggressive SpecAugment (frequency + time masking applied twice each), masking ~85% of the input.
  3. ReduceLROnPlateau decaying the learning rate to near-zero over ~100 epochs, effectively halting learning.
• Second challenge: the cuda_ctc_decoder crashed in the inference environment, forcing a fallback to greedy decoding for the final test run — costing roughly 0.8–1.0 in distance versus what beam search achieved on validation.
• What I'd do differently:
  • Get beam search working reliably at inference time — test it early rather than assuming train/val parity holds — to close that ~1-point gap.
  • Prefer CosineAnnealingLR over ReduceLROnPlateau from the start to avoid the LR-collapse failure mode.
  • Add a sanity check after epoch 1 (val distance should already be well under 50) to catch architecture bugs like the duplicate layer immediately instead of after dozens of epochs.

Tech stack

Python · PyTorch · CTC Loss · BiLSTM / pyramidal BiLSTM · CUDA CTC beam search decoding · SpecAugment

---

Part of a broader project series covering MLP-based speech classification, CNN-based face recognition, and attention-based speech-to-text.