Fine-tuned stt_ar_fastconformer_hybrid_large_pcd model Producing Only <unk> Tokens During Inference

[lekhapm90]

Environment Information

NeMo Version: 2.5.3
Installation Method: pip install nemo_toolkit['all']
Platform: Kaggle
GPU/Accelerator: GPU (single device)
Precision: 16-bit

Model Information

Pretrained Model: nvidia/stt_ar_fastconformer_hybrid_large_pcd_v1.0
Model Type: EncDecHybridRNNTCTCModel
Training Dataset Size: 25 samples
Validation Dataset Size: 10 samples

Problem Description
After fine-tuning the Arabic FastConformer model, inference produces only (unknown) tokens instead of actual transcriptions. The model appears to fine-tune successfully, but the checkpoint fails to generate meaningful output.

Fine-tuning Code

import nemo
import nemo.collections.asr as nemo_asr
import lightning.pytorch as pl
from lightning.pytorch.callbacks import ModelCheckpoint, LearningRateMonitor
from nemo.collections.asr.models import EncDecHybridRNNTCTCModel

checkpoint_callback = ModelCheckpoint(
    dirpath="./checkpoints",
    filename="stt_ar_fastconformer_v1-{epoch:02d}-{val_wer:.4f}",
    save_top_k=3,
    monitor="val_wer",  
    mode="min",          
    save_last=True,
)
lr_monitor = LearningRateMonitor(logging_interval='step')

trainer = pl.Trainer(
    max_epochs=5,
    accelerator="gpu",         
    devices=1,                 
    precision=16,             
    gradient_clip_val=1.0,
    callbacks=[checkpoint_callback, lr_monitor],
    log_every_n_steps=10,
)

pretrained_model_name = "nvidia/stt_ar_fastconformer_hybrid_large_pcd_v1.0"
asr_model = EncDecHybridRNNTCTCModel.from_pretrained(pretrained_model_name, trainer=trainer)

train_manifest = "/kaggle/input/nvidia-nemo-data/train_manifest.json"
val_manifest = "/kaggle/input/nvidia-nemo-data/val_manifest.json"

train_ds_config = {
    "manifest_filepath": train_manifest,
    "batch_size": 2,
    "sample_rate": 16000,
    "shuffle": True,
    "pin_memory": True,
    "max_duration": 6.0, 
}

val_ds_config = {
    "manifest_filepath": val_manifest,
    "batch_size": 2,
    "sample_rate": 16000,
    "shuffle": False,
    "pin_memory": True,
    "max_duration": 6.0, 
    "bucketing_strategy": "none",
}

asr_model.cfg.train_ds.manifest_filepath = train_manifest
asr_model.cfg.validation_ds.manifest_filepath = val_manifest
asr_model.cfg.test_ds.manifest_filepath = val_manifest

tokenizer_dir = "/kaggle/input/arabic-fastconformer-hybrid-model"
asr_model.cfg.tokenizer.dir = tokenizer_dir
asr_model.cfg.tokenizer.type = "bpe"

asr_model.setup_training_data(train_data_config=train_ds_config)
asr_model.setup_validation_data(val_data_config=val_ds_config)

trainer.fit(model=asr_model)

Inference Code

audio_input = "/kaggle/input/test-data/arabic_conv.wav"
asr_model = EncDecHybridRNNTCTCModel.load_from_checkpoint(
    "./checkpoints/stt_ar_fastconformer_v2-epoch=02-val_wer=135.0000.ckpt"
)
asr_model.eval()
transcription = asr_model.transcribe([audio_input])
print(transcription)

Output

The model produces only <unk> tokens:

[Hypothesis(
    score=nan, 
    y_sequence=tensor([0, 0, 0, ..., 0, 0, 0]), 
    text='<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>...'
)]

Observations
Fine-tuning appears to complete without errors
Validation WER is very high (135.0000)

Additional Context

• The manifest files follow the standard NeMo format with audio_filepath, text, and duration fields
• Audio files are in WAV format, 16kHz sample rate
• The tokenizer directory contains both .model and .vocab files from the pretrained model

Sample Train Manifest

{"audio_filepath": "/kaggle/input/nemo-ar-dataset-small/train_0.wav", "duration": 3.06, "text": "فِيهِنَّ خَيْرَاتٌ حِسَانٌ"}
{"audio_filepath": "/kaggle/input/nemo-ar-dataset-small/train_1.wav", "duration": 2.16, "text": "وَإِذَا النُّفوسُ زُوِّجَت"}
{"audio_filepath": "/kaggle/input/nemo-ar-dataset-small/train_2.wav", "duration": 4.176, "text": "مِن نُّطْفَةٍ خَلَقَهُ فَقَدَّرَهُ"}
{"audio_filepath": "/kaggle/input/nemo-ar-dataset-small/train_3.wav", "duration": 2.088, "text": "إستخدم رأسك!"}