Skip to content

Commit 8dcf170

Browse files
hebbihebbhebbihebb
authored
Merge pull request #12 from hebbihebb/claude/explore-flan-t5-integration-011CUdvVQuiX39GMz9qpR4Xu
Integrate FLAN-T5 Grammar Correction (GPU-accelerated) – Confirmed Working, Formatting Guardrails Needed
2 parents 0f9508c + e9af499 commit 8dcf170

File tree

6 files changed

+457
-0
lines changed

6 files changed

+457
-0
lines changed

‎CUDA_FIX_README.md‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
# GPU Not Detected - Quick Fix Guide
2+
3+
## Problem
4+
5+
You're seeing this in the log:
6+
```
7+
CUDA not available, using CPU (this will be slow)
8+
```
9+
10+
But you have a **GTX 2070 (8GB VRAM)** that should work perfectly!
11+
12+
## Why This Happens
13+
14+
When you run:
15+
```bash
16+
pip install torch
17+
```
18+
19+
...it installs the **CPU-only version** by default. PyTorch requires a special installation command to enable GPU support.
20+
21+
## Impact
22+
23+
**Without GPU (current state)**:
24+
- ❌ 5-30 seconds per sentence
25+
- ❌ 100% CPU usage
26+
- ❌ Very slow for large documents
27+
28+
**With GPU (after fix)**:
29+
- ✅ 0.5-2 seconds per sentence (**10-50x faster!**)
30+
- ✅ GPU accelerated
31+
- ✅ Fast enough for production use
32+
33+
Your GTX 2070 has 8GB VRAM which is perfect for this model (needs ~6-8GB).
34+
35+
## Quick Fix (Automated)
36+
37+
```bash
38+
# 1. Diagnose the issue
39+
python check_cuda.py
40+
41+
# 2. Fix it (installs GPU-enabled PyTorch)
42+
./fix_cuda.sh
43+
44+
# 3. Test it works
45+
python test_t5_integration.py
46+
```
47+
48+
The `fix_cuda.sh` script will:
49+
1. Detect your CUDA version from `nvidia-smi`
50+
2. Uninstall CPU-only PyTorch
51+
3. Install GPU-enabled PyTorch (~2-3 GB download)
52+
4. Verify CUDA is working
53+
54+
## Manual Fix
55+
56+
If you prefer to do it manually:
57+
58+
```bash
59+
# Check your GPU and CUDA version
60+
nvidia-smi
61+
62+
# Uninstall CPU-only PyTorch
63+
pip uninstall torch torchvision torchaudio
64+
65+
# Install with CUDA support (choose based on your CUDA version)
66+
# For CUDA 11.8:
67+
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
68+
69+
# For CUDA 12.1+:
70+
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
71+
72+
# Verify it works
73+
python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
74+
```
75+
76+
## After the Fix
77+
78+
Once CUDA is detected, the log will show:
79+
```
80+
Using CUDA for T5 inference
81+
✓ Model loaded successfully!
82+
```
83+
84+
And the model will run **10-50x faster** on your GPU!
85+
86+
## Files Added
87+
88+
Three new diagnostic/fix tools:
89+
- `check_cuda.py` - Diagnose CUDA detection issues
90+
- `fix_cuda.sh` - Automated fix script
91+
- `CUDA_FIX_README.md` - This guide
92+
93+
## Note About the Download
94+
95+
You mentioned the model is still downloading:
96+
```
97+
model.safetensors: 15%|█▏ | 472M/3.13G
98+
```
99+
100+
This is the model weights (3.13 GB), separate from PyTorch. Let that finish downloading, then run the CUDA fix, and you'll be all set!
101+
102+
## Expected Final Behavior
103+
104+
After both downloads complete and CUDA is fixed:
105+
106+
```bash
107+
$ python test_t5_integration.py
108+
109+
1. Loading T5 model...
110+
Using CUDA for T5 inference # ✅ GPU detected
111+
✓ Model loaded successfully!
112+
113+
2. Testing grammar/spelling corrections:
114+
115+
Test 1:
116+
Original: Thiss sentnce have many speling errrors.
117+
Corrected: This sentence has many spelling errors. # ✅ Fast (0.5-2s)
118+
```
119+
120+
## Questions?
121+
122+
See the full troubleshooting guide in `T5_INTEGRATION_GUIDE.md` (line 247+).

‎T5_INTEGRATION_GUIDE.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -246,6 +246,51 @@ filter = T5GrammarFilter(model_name="vennify/t5-base-grammar-correction")
246246

247247
## Troubleshooting
248248

249+
### Issue: "CUDA not available, using CPU" (but you have a GPU)
250+
251+
**Symptoms**:
252+
- Log shows "CUDA not available, using CPU (this will be slow)"
253+
- You have an NVIDIA GPU (e.g., GTX 2070)
254+
- Very slow inference (5-30 seconds per sentence)
255+
256+
**Cause**: You likely installed the **CPU-only version** of PyTorch
257+
258+
**Solution**:
259+
```bash
260+
# 1. Check if CUDA is detected
261+
python check_cuda.py
262+
263+
# 2. If CUDA is not detected, run the fix script
264+
./fix_cuda.sh
265+
266+
# This will:
267+
# - Uninstall CPU-only PyTorch
268+
# - Install GPU-enabled PyTorch
269+
# - Verify CUDA works
270+
```
271+
272+
**Manual fix**:
273+
```bash
274+
# Check your CUDA version
275+
nvidia-smi
276+
277+
# Uninstall CPU-only PyTorch
278+
pip uninstall torch torchvision torchaudio
279+
280+
# Install with CUDA 11.8 support
281+
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
282+
283+
# OR with CUDA 12.1 support
284+
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
285+
286+
# Verify
287+
python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
288+
```
289+
290+
**Performance difference**:
291+
- CPU: 5-30 seconds per sentence
292+
- GPU (GTX 2070): 0.5-2 seconds per sentence (**10-50x faster!**)
293+
249294
### Issue: "CUDA out of memory"
250295

251296
**Solution**: Reduce max_length or use CPU

‎check_cuda.py‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
#!/usr/bin/env python3
2+
"""
3+
CUDA Diagnostic Script - Check PyTorch and GPU availability
4+
"""
5+
6+
import sys
7+
8+
print("=" * 60)
9+
print("CUDA Availability Diagnostic")
10+
print("=" * 60)
11+
12+
# Check Python version
13+
print(f"\nPython version: {sys.version}")
14+
15+
# Check if torch is installed
16+
try:
17+
import torch
18+
print(f"\n✓ PyTorch installed: version {torch.__version__}")
19+
except ImportError:
20+
print("\n✗ PyTorch not installed!")
21+
print(" Install with: pip install torch torchvision torchaudio")
22+
sys.exit(1)
23+
24+
# Check CUDA availability in PyTorch
25+
print(f"\nCUDA available in PyTorch: {torch.cuda.is_available()}")
26+
27+
if torch.cuda.is_available():
28+
print(f"✓ CUDA detected!")
29+
print(f" CUDA version: {torch.version.cuda}")
30+
print(f" Number of GPUs: {torch.cuda.device_count()}")
31+
32+
for i in range(torch.cuda.device_count()):
33+
print(f"\n GPU {i}: {torch.cuda.get_device_name(i)}")
34+
props = torch.cuda.get_device_properties(i)
35+
print(f" Total memory: {props.total_memory / 1024**3:.2f} GB")
36+
print(f" Compute capability: {props.major}.{props.minor}")
37+
else:
38+
print("✗ CUDA NOT detected by PyTorch")
39+
print("\nPossible reasons:")
40+
print(" 1. CPU-only PyTorch installed (most likely)")
41+
print(" 2. NVIDIA drivers not installed")
42+
print(" 3. CUDA toolkit not installed")
43+
print(" 4. PyTorch CUDA version doesn't match system CUDA")
44+
45+
# Check PyTorch build info
46+
print(f"\nPyTorch build info:")
47+
print(f" Built with CUDA: {torch.version.cuda is not None}")
48+
if torch.version.cuda:
49+
print(f" CUDA version: {torch.version.cuda}")
50+
else:
51+
print(f" ⚠ This is a CPU-only build!")
52+
53+
# Check if NVIDIA GPU exists at system level
54+
print("\n" + "=" * 60)
55+
print("System GPU Check")
56+
print("=" * 60)
57+
58+
try:
59+
import subprocess
60+
result = subprocess.run(['nvidia-smi'], capture_output=True, text=True, timeout=5)
61+
if result.returncode == 0:
62+
print("\n✓ nvidia-smi output:")
63+
print(result.stdout)
64+
else:
65+
print("\n✗ nvidia-smi failed (NVIDIA drivers may not be installed)")
66+
except FileNotFoundError:
67+
print("\n✗ nvidia-smi not found (NVIDIA drivers not installed)")
68+
except Exception as e:
69+
print(f"\n⚠ Error running nvidia-smi: {e}")
70+
71+
# Recommendations
72+
print("\n" + "=" * 60)
73+
print("Recommendations")
74+
print("=" * 60)
75+
76+
if not torch.cuda.is_available():
77+
print("\nTo enable GPU support:")
78+
print("\n1. Check NVIDIA drivers are installed:")
79+
print(" nvidia-smi")
80+
print("\n2. Reinstall PyTorch with CUDA support:")
81+
print(" pip uninstall torch torchvision torchaudio")
82+
print(" pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118")
83+
print("\n (cu118 = CUDA 11.8, use cu121 for CUDA 12.1, etc.)")
84+
print("\n3. Verify CUDA is detected:")
85+
print(" python check_cuda.py")
86+
else:
87+
print("\n✓ Your GPU is ready to use!")
88+
print(" The T5 model will automatically use your GPU for inference.")
89+
print(" Expected speed: 0.5-2 seconds per sentence (vs 5-30s on CPU)")

‎fix_cuda.sh‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
#!/bin/bash
2+
# Fix CUDA support for PyTorch - Install GPU-enabled version
3+
4+
set -e
5+
6+
echo "=========================================="
7+
echo "PyTorch CUDA Fix Script"
8+
echo "=========================================="
9+
10+
# Check if nvidia-smi exists
11+
if ! command -v nvidia-smi &> /dev/null; then
12+
echo ""
13+
echo "✗ nvidia-smi not found!"
14+
echo " NVIDIA drivers are not installed or not in PATH."
15+
echo " Please install NVIDIA drivers first."
16+
exit 1
17+
fi
18+
19+
# Get CUDA version from nvidia-smi
20+
echo ""
21+
echo "1. Checking system CUDA version..."
22+
CUDA_VERSION=$(nvidia-smi | grep "CUDA Version" | awk '{print $9}' | cut -d. -f1,2)
23+
echo " Detected CUDA: $CUDA_VERSION"
24+
25+
# Determine PyTorch CUDA version
26+
if [[ $(echo "$CUDA_VERSION >= 12.1" | bc -l) -eq 1 ]]; then
27+
TORCH_CUDA="cu121"
28+
echo " Will install PyTorch with CUDA 12.1 support"
29+
elif [[ $(echo "$CUDA_VERSION >= 11.8" | bc -l) -eq 1 ]]; then
30+
TORCH_CUDA="cu118"
31+
echo " Will install PyTorch with CUDA 11.8 support"
32+
else
33+
TORCH_CUDA="cu118"
34+
echo " ⚠ CUDA version is older, will try CUDA 11.8 build"
35+
fi
36+
37+
# Show GPU info
38+
echo ""
39+
echo "2. GPU Information:"
40+
nvidia-smi --query-gpu=name,memory.total --format=csv,noheader
41+
42+
# Confirm with user
43+
echo ""
44+
echo "=========================================="
45+
echo "This will:"
46+
echo " 1. Uninstall current PyTorch (CPU-only)"
47+
echo " 2. Install PyTorch with CUDA support ($TORCH_CUDA)"
48+
echo " 3. Download ~2-3 GB of packages"
49+
echo "=========================================="
50+
echo ""
51+
read -p "Continue? (y/n) " -n 1 -r
52+
echo
53+
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
54+
echo "Cancelled."
55+
exit 0
56+
fi
57+
58+
# Activate virtual environment if it exists
59+
if [ -d "venv" ]; then
60+
echo ""
61+
echo "3. Activating virtual environment..."
62+
source venv/bin/activate
63+
echo " ✓ Using: $VIRTUAL_ENV"
64+
elif [ -n "$VIRTUAL_ENV" ]; then
65+
echo ""
66+
echo "3. Using existing virtual environment: $VIRTUAL_ENV"
67+
else
68+
echo ""
69+
echo "⚠ No virtual environment detected!"
70+
echo " Installing to system Python (not recommended)"
71+
read -p "Continue anyway? (y/n) " -n 1 -r
72+
echo
73+
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
74+
echo "Cancelled. Create a venv first: python -m venv venv"
75+
exit 0
76+
fi
77+
fi
78+
79+
# Uninstall existing PyTorch
80+
echo ""
81+
echo "4. Removing CPU-only PyTorch..."
82+
pip uninstall -y torch torchvision torchaudio 2>/dev/null || true
83+
echo " ✓ Uninstalled"
84+
85+
# Install CUDA-enabled PyTorch
86+
echo ""
87+
echo "5. Installing PyTorch with CUDA support..."
88+
echo " This will download ~2-3 GB..."
89+
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/$TORCH_CUDA
90+
91+
echo ""
92+
echo "=========================================="
93+
echo "6. Verifying CUDA support..."
94+
echo "=========================================="
95+
python check_cuda.py
96+
97+
echo ""
98+
echo "=========================================="
99+
echo "✓ Setup Complete!"
100+
echo "=========================================="
101+
echo ""
102+
echo "Your GPU is now ready for T5 inference!"
103+
echo ""
104+
echo "Test it:"
105+
echo " python test_t5_integration.py"
106+
echo ""
107+
echo "Expected performance:"
108+
echo " - CPU: 5-30 seconds per sentence"
109+
echo " - GPU: 0.5-2 seconds per sentence (10-50x faster!)"
110+
echo ""

0 commit comments

Comments
 (0)