🎙️ Whisper Fine-Tuning for Pronunciation Learning

Advanced Speech Recognition for Fragmented and Broken English Words

---

📋 Table of Contents

• Overview
• Key Features
• Architecture
• Performance Metrics
• Installation
• Dataset Preparation
• Usage
• Training Pipeline
• Results
• Model Details

---

🎯 Overview

This project presents a sophisticated approach to fine-tuning OpenAI's Whisper speech-to-text model for enhanced pronunciation learning applications. The system specializes in accurately transcribing broken words and fragmented speech segments, making it ideal for language learning scenarios where learners struggle with partial pronunciation.

🎓 Project Motivation

Language learners often produce fragmented or partially articulated words during practice. Traditional ASR systems struggle with these scenarios, but our fine-tuned model bridges this gap by:

• Recognizing incomplete word segments with high accuracy
• Supporting pronunciation assessment for language education
• Providing real-time feedback for learners
• Achieving 95% accuracy on fragmented speech data

🏆 Key Achievements

• ✅ 95% accuracy on broken English word recognition
• ✅ Transformer-based architecture leveraging OpenAI Whisper-Base
• ✅ Transfer learning optimization with custom fine-tuning pipeline
• ✅ Real-world integration with language learning applications
• ✅ Comprehensive evaluation using Word Error Rate (WER) metrics

---

✨ Key Features

🔬 Technical Capabilities

Feature	Description
Fragmented Speech Recognition	Accurately transcribes partially uttered words and broken speech segments
Transfer Learning	Leverages pre-trained Whisper-Base model with custom fine-tuning
Low WER	Achieves near-optimal Word Error Rate for pronunciation learning
GPU-Accelerated	CUDA-optimized training and inference pipeline
Educational Integration	Designed for seamless integration into language learning apps

🎨 Model Specifications

• Base Model: OpenAI Whisper-Base (English)
• Architecture: Transformer Encoder-Decoder
• Input: 16kHz audio samples (30s max)
• Output: Text transcription with confidence scores
• Training Strategy: Supervised fine-tuning with cross-entropy loss

---

🏗️ Architecture

ASR Pipeline Components

The Automatic Speech Recognition (ASR) pipeline consists of three main components:

┌─────────────────────────────────────────────────────────────┐
│                     ASR PIPELINE                             │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  1. Feature Extractor                                       │
│     └─→ Raw Audio → Log-Mel Spectrogram                    │
│                                                              │
│  2. Whisper Model (Encoder-Decoder)                         │
│     ├─→ Encoder: Spectrogram → Hidden States               │
│     └─→ Decoder: Hidden States → Text Tokens               │
│                                                              │
│  3. Tokenizer                                               │
│     └─→ Text Tokens → Human-Readable Text                  │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Whisper Model Architecture

Whisper is a Transformer-based encoder-decoder model that performs sequence-to-sequence mapping:

1. Feature Extraction: Converts raw audio to log-Mel spectrogram (80 channels, 30s window)
2. Encoder: Processes spectrograms to generate hidden state representations
3. Decoder: Autoregressively predicts text tokens using encoder states and previous tokens
4. Deep Fusion: Internal language model for context-aware transcription

Training Objective

• Loss Function: Cross-entropy objective
• Optimization: AdamW optimizer with learning rate scheduling
• Regularization: Gradient accumulation and warmup steps

---

📊 Performance Metrics

Before Fine-Tuning

Metric	Value	Notes
Accuracy	0%	Base Whisper model on fragmented speech
WER	~100%	Unable to recognize broken words
Use Case	❌ Not suitable	Standard model fails on partial pronunciations

After Fine-Tuning

Metric	Value	Notes
Accuracy	95%	Significant improvement on test set
WER	~5%	Near-optimal word error rate
Use Case	✅ Production-ready	Suitable for educational applications
Similarity Threshold	90%	FuzzyWuzzy matching for evaluation

Training Configuration

Training Hyperparameters:
├─ Batch Size: 4 (per device)
├─ Gradient Accumulation: 4 steps
├─ Learning Rate: 1e-5
├─ Warmup Steps: 250
├─ Max Steps: 1000
├─ Optimizer: AdamW
└─ Evaluation Strategy: Every 10 steps

---

🚀 Installation

Prerequisites

• Python 3.8 or higher
• CUDA-compatible GPU (recommended)
• 16GB+ RAM
• Git

Step 1: Clone the Repository

git clone https://github.com/bilalhameed248/Whisper-Fine-Tuning-For-Pronunciation-Learning.git
cd Whisper-Fine-Tuning-For-Pronunciation-Learning

Step 2: Create Virtual Environment

# Using conda
conda create -n whisper-finetune python=3.8
conda activate whisper-finetune

# Or using venv
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Step 3: Install Dependencies

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install transformers datasets librosa soundfile
pip install evaluate jiwer tensorboard
pip install fuzzywuzzy python-Levenshtein
pip install pynvml numba
pip install jupyter notebook

Step 4: Verify Installation

import torch
print(f"PyTorch Version: {torch.__version__}")
print(f"CUDA Available: {torch.cuda.is_available()}")
print(f"CUDA Version: {torch.version.cuda}")

---

📁 Dataset Preparation

Dataset Structure

Organize your audio files in the following structure:

data/
├── train/
│   ├── audio1.wav
│   ├── audio2.wav
│   └── metadata.csv
├── validation/
│   ├── audio1.wav
│   └── metadata.csv
└── test/
    ├── audio1.wav
    └── metadata.csv

Metadata Format

Your metadata.csv should contain:

file_name,sentence
audio1.wav,broken word example
audio2.wav,partial pronunciation

Audio Requirements

• Format: WAV, MP3, or FLAC
• Sampling Rate: 16kHz (automatically resampled)
• Duration: Up to 30 seconds per clip
• Quality: Clear pronunciation recordings

---

💻 Usage

Quick Start: Inference

from transformers import pipeline, WhisperTokenizer

# Load fine-tuned model
tokenizer = WhisperTokenizer.from_pretrained('./tokenizer/', language="english", task="transcribe")
pipe = pipeline(
    "automatic-speech-recognition",
    model="./whisper-base-languagelab5/checkpoint-500/",
    tokenizer=tokenizer,
    device=0  # GPU device
)

# Transcribe audio
result = pipe("path/to/audio.wav")
print(f"Transcribed Text: {result['text']}")

Training from Scratch

Open the Jupyter notebook:

jupyter notebook Whisper-small-fine-tuning.ipynb

Follow the notebook cells sequentially:

1. Setup & Configuration: GPU setup and library imports
2. Data Loading: Load and prepare your dataset
3. Feature Extraction: Configure Whisper processor
4. Model Training: Fine-tune with custom parameters
5. Evaluation: Test on validation/test sets
6. Inference: Use the trained model

---

🎯 Training Pipeline

Step-by-Step Process

1. Data Preprocessing

# Resample audio to 16kHz
common_voice = common_voice.cast_column("audio", Audio(sampling_rate=16_000))

# Extract features and tokenize
def prepare_dataset(batch):
    audio = batch["audio"]
    batch["input_features"] = feature_extractor(
        audio["array"], 
        sampling_rate=audio["sampling_rate"]
    ).input_features[0]
    batch["labels"] = tokenizer(batch["sentence"]).input_ids
    return batch

2. Model Configuration

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-base")
model.generation_config.language = "english"
model.config.forced_decoder_ids = None
model.config.suppress_tokens = []

3. Training Arguments

training_args = Seq2SeqTrainingArguments(
    output_dir="./whisper-base-languagelab5",
    per_device_train_batch_size=4,
    learning_rate=1e-5,
    warmup_steps=250,
    max_steps=1000,
    evaluation_strategy="steps",
    eval_steps=10,
    save_steps=100,
    metric_for_best_model="wer",
    load_best_model_at_end=True
)

4. Training Execution

trainer = Seq2SeqTrainer(
    args=training_args,
    model=model,
    train_dataset=common_voice["train"],
    eval_dataset=common_voice["validate"],
    data_collator=data_collator,
    compute_metrics=compute_metrics,
    tokenizer=processor.feature_extractor,
)

trainer.train()

---

📈 Results

Quantitative Analysis

Evaluation Metric	Before Fine-Tuning	After Fine-Tuning	Improvement
Accuracy	0%	95%	+95%
Word Error Rate	100%	5%	-95%
True Predictions	0/test_size	95/100	Significant
Similarity Score	<50%	>90%	+40%+

Qualitative Improvements

✅ Recognition of Fragmented Words: Successfully transcribes broken pronunciations
✅ Context Understanding: Maintains semantic meaning despite incomplete words
✅ Low Latency: Real-time transcription capability
✅ Robustness: Handles various accents and speech patterns

Sample Predictions

Original Word	Base Model Output	Fine-Tuned Output	Match
"app-le" (broken)	"apple"	"app-le"	✅
"be-au-ti-ful"	"beautiful"	"be-au-ti-ful"	✅
"pro-nun-ci-a-tion"	"pronunciation"	"pro-nun-ci-a-tion"	✅

---

🔧 Model Details

Architecture Specifications

• Model Name: Whisper-Base (Fine-Tuned)
• Parameters: ~74M
• Encoder Layers: 6
• Decoder Layers: 6
• Attention Heads: 8
• Embedding Dimension: 512
• Vocabulary Size: 51,865 tokens

Training Infrastructure

• Hardware: NVIDIA GPU (CUDA 11.8+)
• Framework: PyTorch 2.0+
• Training Time: ~2-4 hours (depending on dataset size)
• Memory Requirements: 16GB GPU RAM

Evaluation Methodology

# Word Error Rate (WER) Calculation
def compute_metrics(pred):
    pred_ids = pred.predictions
    label_ids = pred.label_ids
    
    pred_str = tokenizer.batch_decode(pred_ids, skip_special_tokens=True)
    label_str = tokenizer.batch_decode(label_ids, skip_special_tokens=True)
    
    wer = 100 * metric.compute(predictions=pred_str, references=label_str)
    return {"wer": wer}

---

Key Technologies

• Whisper by OpenAI
• Hugging Face Transformers
• PyTorch
• Datasets Library

---

⭐ If you find this project useful, please consider giving it a star!

Made with ❤️ for Language Learners Worldwide