TROUBLESHOOTING.md

Troubleshooting Guide

Common Issues and Solutions

NaN Loss During Training

Symptoms:

Step 450 | Loss: nan | LR: 4.81e-05

Causes:

1. Mixed precision incompatibility: BF16 requires modern hardware (NVIDIA Ampere GPUs or newer)
2. Numerical instability: Gradient explosions or underflows with mixed precision
3. Learning rate too high: Can cause gradient explosions
4. Hardware limitations: CPU or older GPUs don't support BF16

Solutions:

Option 1: Disable Mixed Precision (Recommended for testing)

Use the FP32 configuration files:

uv run accelerate launch --config_file accelerate_ds_config_fp32.yaml \
    scripts/run.py --train_config configs/config_quick_test_fp32.json

Or modify your config files:

In configs/config_quick_test.json:

{
  "training_config": {
    "fp16": false,
    "bf16": false,
    ...
  }
}

In accelerate_ds_config.yaml:

mixed_precision: 'no'

Option 2: Use FP16 Instead of BF16

If you have a GPU that supports FP16 but not BF16:

In configs/config_quick_test.json:

{
  "training_config": {
    "fp16": true,
    "bf16": false,
    ...
  }
}

In accelerate_ds_config.yaml:

mixed_precision: fp16

Option 3: Reduce Learning Rate

If using mixed precision, reduce the learning rate:

In configs/config_quick_test.json:

{
  "training_config": {
    "learning_rate": 2e-5,  // Reduced from 5e-5
    ...
  }
}

Option 4: Increase Gradient Clipping

Prevent gradient explosions:

In configs/config_quick_test.json:

{
  "training_config": {
    "max_grad_norm": 0.5,  // Reduced from 1.0
    ...
  }
}

Checking Hardware Support

Check if your GPU supports BF16:

nvidia-smi --query-gpu=compute_cap --format=csv

• Compute capability 8.0+ (Ampere): Supports BF16
• Compute capability 7.0-7.5 (Volta/Turing): Use FP16
• Older GPUs: Use FP32

Check PyTorch BF16 support:

import torch
print(f"BF16 supported: {torch.cuda.is_bf16_supported()}")

DeepSpeed Issues

Error: DeepSpeed not available

# Install DeepSpeed
uv pip install deepspeed

# Or use standard Accelerate without DeepSpeed
uv run accelerate config  # Choose "No" for DeepSpeed

Out of Memory with DeepSpeed

Increase ZeRO stage or enable offloading in accelerate_ds_config.yaml:

deepspeed_config:
  zero_stage: 3
  offload_optimizer_device: cpu
  offload_param_device: cpu

Dataset Issues

Error: Dataset not found

Ensure you have internet connection for first-time download, or download manually:

from datasets import load_dataset
dataset = load_dataset("wikitext", "wikitext-2-raw-v1")

Slow tokenization

Reduce workers or sequence length in config:

{
  "data_config": {
    "max_seq_length": 64,  // Reduced from 128
    "preprocessing_num_workers": 1,  // Reduced from 2
    ...
  }
}

CUDA Out of Memory

Solutions:

1. Reduce batch size:

{
  "training_config": {
    "per_device_train_batch_size": 2,  // Reduced from 4
    "gradient_accumulation_steps": 2,   // Increased to maintain effective batch size
    ...
  }
}

2. Reduce sequence length:

{
  "data_config": {
    "max_seq_length": 64,  // Reduced from 128
    ...
  }
}

3. Enable gradient checkpointing (add to training script or use smaller model)

Docker-Specific Issues

Container not starting

# Check Docker logs
docker logs GLMs-<branch-name>

# Verify GPU access
docker run --rm --gpus all nvidia/cuda:12.1.0-base-ubuntu22.04 nvidia-smi

Permission denied

# Fix permissions
sudo chown -R $USER:$USER .

Best Practices

For Development/Testing

• Use config_quick_test_fp32.json with accelerate_ds_config_fp32.yaml
• FP32 is slower but more stable
• Smaller batch sizes and shorter sequences

For Production Training

• Use FP16 if GPU supports it (compute capability 7.0+)
• Use BF16 only on Ampere or newer (compute capability 8.0+)
• Start with conservative learning rates (2e-5)
• Monitor for NaN losses and reduce LR if they occur

Debugging Steps

1. Start with FP32 to verify the setup works
2. Enable FP16/BF16 only after confirming FP32 works
3. Use smaller batch sizes and learning rates initially
4. Monitor GPU memory usage with nvidia-smi
5. Check logs in ./logs/ directory

Getting Help

If issues persist:

1. Check the full error message in logs
2. Verify hardware compatibility
3. Try the FP32 configuration first
4. Reduce batch size and sequence length
5. Check GitHub issues for similar problems

Quick Commands

# Test with FP32 (stable)
uv run accelerate launch --config_file accelerate_ds_config_fp32.yaml \
    scripts/run.py --train_config configs/config_quick_test_fp32.json

# Test with original config
uv run accelerate launch --config_file accelerate_ds_config.yaml \
    scripts/run.py --train_config configs/config_quick_test.json

# Monitor GPU
watch -n 1 nvidia-smi

# Check logs
tail -f logs/training_*.log