|
| 1 | +# GPU Configuration |
| 2 | + |
| 3 | +Configure GPU resources for the aa-tRNA-seq pipeline. |
| 4 | + |
| 5 | +## GPU Requirements |
| 6 | + |
| 7 | +Two rules require GPU access: |
| 8 | + |
| 9 | +| Rule | Purpose | GPU Usage | |
| 10 | +|------|---------|-----------| |
| 11 | +| `rebasecall` | Dorado basecalling | CUDA neural network inference | |
| 12 | +| `classify_charging` | Remora classification | PyTorch model inference | |
| 13 | + |
| 14 | +Both rules benefit significantly from GPU acceleration. CPU-only execution is possible but substantially slower. |
| 15 | + |
| 16 | +## GPU Resource Flow |
| 17 | + |
| 18 | +```mermaid |
| 19 | +flowchart LR |
| 20 | + subgraph GPU Rules |
| 21 | + A[rebasecall<br/>Dorado] --> B[classify_charging<br/>Remora] |
| 22 | + end |
| 23 | +
|
| 24 | + subgraph Resources |
| 25 | + C[POD5 Signal Data] |
| 26 | + D[CUDA GPU] |
| 27 | + end |
| 28 | +
|
| 29 | + C --> A |
| 30 | + D --> A |
| 31 | + D --> B |
| 32 | +``` |
| 33 | + |
| 34 | +## Cluster Configuration |
| 35 | + |
| 36 | +### LSF GPU Settings |
| 37 | + |
| 38 | +In `cluster/lsf/config.yaml`: |
| 39 | + |
| 40 | +```yaml |
| 41 | +# Limit total concurrent GPU jobs |
| 42 | +resources: |
| 43 | + - ngpu=12 |
| 44 | + |
| 45 | +# GPU rule configuration |
| 46 | +set-resources: |
| 47 | + - rebasecall:lsf_queue="gpu" |
| 48 | + - rebasecall:lsf_extra="-gpu num=1:j_exclusive=yes" |
| 49 | + - rebasecall:ngpu=1 |
| 50 | + - rebasecall:mem_mb=24 |
| 51 | + |
| 52 | + - classify_charging:lsf_queue="gpu" |
| 53 | + - classify_charging:lsf_extra="-gpu num=1:j_exclusive=yes" |
| 54 | + - classify_charging:ngpu=1 |
| 55 | + - classify_charging:mem_mb=24 |
| 56 | +``` |
| 57 | +
|
| 58 | +### SLURM GPU Settings |
| 59 | +
|
| 60 | +```yaml |
| 61 | +resources: |
| 62 | + - ngpu=8 |
| 63 | + |
| 64 | +set-resources: |
| 65 | + - rebasecall:partition="gpu" |
| 66 | + - rebasecall:gpu_opts="--gres=gpu:1" |
| 67 | + - rebasecall:ngpu=1 |
| 68 | + - rebasecall:mem_mb=24000 |
| 69 | + |
| 70 | + - classify_charging:partition="gpu" |
| 71 | + - classify_charging:gpu_opts="--gres=gpu:1" |
| 72 | + - classify_charging:ngpu=1 |
| 73 | + - classify_charging:mem_mb=24000 |
| 74 | +``` |
| 75 | +
|
| 76 | +## Configuration Options |
| 77 | +
|
| 78 | +### GPU Concurrency Limit |
| 79 | +
|
| 80 | +Control how many GPU jobs run simultaneously: |
| 81 | +
|
| 82 | +```yaml |
| 83 | +resources: |
| 84 | + - ngpu=8 # Max 8 concurrent GPU jobs |
| 85 | +``` |
| 86 | +
|
| 87 | +Set this to match your available GPUs or queue limits. |
| 88 | +
|
| 89 | +### Exclusive GPU Access |
| 90 | +
|
| 91 | +Request exclusive GPU access to avoid memory conflicts: |
| 92 | +
|
| 93 | +=== "LSF" |
| 94 | +
|
| 95 | + ```yaml |
| 96 | + set-resources: |
| 97 | + - rebasecall:lsf_extra="-gpu num=1:j_exclusive=yes" |
| 98 | + ``` |
| 99 | +
|
| 100 | +=== "SLURM" |
| 101 | +
|
| 102 | + ```yaml |
| 103 | + set-resources: |
| 104 | + - rebasecall:gpu_opts="--gres=gpu:1 --exclusive" |
| 105 | + ``` |
| 106 | +
|
| 107 | +### GPU Type Selection |
| 108 | +
|
| 109 | +If your cluster has multiple GPU types: |
| 110 | +
|
| 111 | +=== "LSF" |
| 112 | +
|
| 113 | + ```yaml |
| 114 | + set-resources: |
| 115 | + - rebasecall:lsf_extra="-gpu num=1:j_exclusive=yes:gtile='!gv100'" |
| 116 | + ``` |
| 117 | +
|
| 118 | +=== "SLURM" |
| 119 | +
|
| 120 | + ```yaml |
| 121 | + set-resources: |
| 122 | + - rebasecall:gpu_opts="--gres=gpu:v100:1" |
| 123 | + ``` |
| 124 | +
|
| 125 | +## Local GPU Execution |
| 126 | +
|
| 127 | +### CUDA_VISIBLE_DEVICES |
| 128 | +
|
| 129 | +The pipeline respects `CUDA_VISIBLE_DEVICES`: |
| 130 | + |
| 131 | +```bash |
| 132 | +# Use specific GPU |
| 133 | +export CUDA_VISIBLE_DEVICES=0 |
| 134 | +pixi run snakemake --cores 4 --configfile=config/config.yml |
| 135 | +
|
| 136 | +# Use multiple GPUs (one per job) |
| 137 | +export CUDA_VISIBLE_DEVICES=0,1 |
| 138 | +pixi run snakemake --cores 4 --resources gpu=2 --configfile=config/config.yml |
| 139 | +``` |
| 140 | + |
| 141 | +### Limit GPU Jobs Locally |
| 142 | + |
| 143 | +```bash |
| 144 | +pixi run snakemake --cores 8 --resources gpu=1 \ |
| 145 | + --configfile=config/config.yml |
| 146 | +``` |
| 147 | + |
| 148 | +## Memory Requirements |
| 149 | + |
| 150 | +GPU rules also require significant system memory: |
| 151 | + |
| 152 | +| Rule | GPU Memory | System Memory | |
| 153 | +|------|------------|---------------| |
| 154 | +| `rebasecall` | ~8-16 GB | 24 GB | |
| 155 | +| `classify_charging` | ~4-8 GB | 24 GB | |
| 156 | + |
| 157 | +## Performance Considerations |
| 158 | + |
| 159 | +### Dorado (rebasecall) |
| 160 | + |
| 161 | +- Processes POD5 signal data through neural network |
| 162 | +- Throughput: ~100-500 reads/second depending on GPU |
| 163 | +- Benefits from newer GPU architectures (Ampere, Ada Lovelace) |
| 164 | + |
| 165 | +### Remora (classify_charging) |
| 166 | + |
| 167 | +- Analyzes signal at CCA 3' end |
| 168 | +- Lower throughput than Dorado |
| 169 | +- Memory usage depends on batch size |
| 170 | + |
| 171 | +## Troubleshooting |
| 172 | + |
| 173 | +### CUDA Out of Memory |
| 174 | + |
| 175 | +**Symptom:** |
| 176 | +``` |
| 177 | +RuntimeError: CUDA out of memory |
| 178 | +``` |
| 179 | +
|
| 180 | +**Solutions:** |
| 181 | +
|
| 182 | +1. Ensure exclusive GPU access: |
| 183 | + ```yaml |
| 184 | + set-resources: |
| 185 | + - rebasecall:lsf_extra="-gpu num=1:j_exclusive=yes" |
| 186 | + ``` |
| 187 | + |
| 188 | +2. Reduce concurrent GPU jobs: |
| 189 | + ```yaml |
| 190 | + resources: |
| 191 | + - ngpu=4 # Reduce from default |
| 192 | + ``` |
| 193 | +
|
| 194 | +3. Check for other GPU processes: |
| 195 | + ```bash |
| 196 | + nvidia-smi |
| 197 | + ``` |
| 198 | + |
| 199 | +### GPU Not Detected |
| 200 | + |
| 201 | +**Symptom:** |
| 202 | +``` |
| 203 | +No CUDA GPUs are available |
| 204 | +``` |
| 205 | + |
| 206 | +**Solutions:** |
| 207 | + |
| 208 | +1. Verify CUDA installation: |
| 209 | + ```bash |
| 210 | + nvidia-smi |
| 211 | + ``` |
| 212 | + |
| 213 | +2. Check CUDA_VISIBLE_DEVICES: |
| 214 | + ```bash |
| 215 | + echo $CUDA_VISIBLE_DEVICES |
| 216 | + ``` |
| 217 | + |
| 218 | +3. Verify job is on GPU node: |
| 219 | + ```bash |
| 220 | + # LSF |
| 221 | + bjobs -l <job_id> | grep -i gpu |
| 222 | + |
| 223 | + # SLURM |
| 224 | + scontrol show job <job_id> | grep -i gres |
| 225 | + ``` |
| 226 | + |
| 227 | +### Wrong GPU Type |
| 228 | + |
| 229 | +**Symptom:** |
| 230 | +Job runs on incompatible GPU. |
| 231 | + |
| 232 | +**Solutions:** |
| 233 | + |
| 234 | +Specify GPU type explicitly in cluster profile: |
| 235 | + |
| 236 | +=== "LSF" |
| 237 | + |
| 238 | + ```yaml |
| 239 | + set-resources: |
| 240 | + - rebasecall:lsf_extra="-gpu num=1:j_exclusive=yes:gmodel=NVIDIAA100" |
| 241 | + ``` |
| 242 | + |
| 243 | +=== "SLURM" |
| 244 | + |
| 245 | + ```yaml |
| 246 | + set-resources: |
| 247 | + - rebasecall:gpu_opts="--gres=gpu:a100:1" |
| 248 | + ``` |
| 249 | + |
| 250 | +### Jobs Waiting for GPU |
| 251 | + |
| 252 | +**Symptom:** |
| 253 | +GPU jobs pending indefinitely. |
| 254 | + |
| 255 | +**Solutions:** |
| 256 | + |
| 257 | +1. Check GPU queue status: |
| 258 | + ```bash |
| 259 | + # LSF |
| 260 | + bqueues -l gpu |
| 261 | + |
| 262 | + # SLURM |
| 263 | + sinfo -p gpu |
| 264 | + ``` |
| 265 | + |
| 266 | +2. Reduce concurrent GPU jobs: |
| 267 | + ```yaml |
| 268 | + resources: |
| 269 | + - ngpu=2 |
| 270 | + ``` |
| 271 | +
|
| 272 | +3. Check fair share limits with your admin. |
| 273 | +
|
| 274 | +## GPU Monitoring |
| 275 | +
|
| 276 | +### NVIDIA SMI |
| 277 | +
|
| 278 | +Monitor GPU usage during execution: |
| 279 | +
|
| 280 | +```bash |
| 281 | +# Watch GPU utilization |
| 282 | +watch -n 1 nvidia-smi |
| 283 | + |
| 284 | +# Log GPU stats |
| 285 | +nvidia-smi --query-gpu=timestamp,name,utilization.gpu,utilization.memory,memory.used --format=csv -l 1 > gpu_log.csv |
| 286 | +``` |
| 287 | + |
| 288 | +### Check Running GPU Jobs |
| 289 | + |
| 290 | +=== "LSF" |
| 291 | + |
| 292 | + ```bash |
| 293 | + bjobs -u $USER -q gpu |
| 294 | + ``` |
| 295 | + |
| 296 | +=== "SLURM" |
| 297 | + |
| 298 | + ```bash |
| 299 | + squeue -u $USER -p gpu |
| 300 | + ``` |
| 301 | + |
| 302 | +## CPU Fallback |
| 303 | + |
| 304 | +If GPUs are unavailable, Dorado can run on CPU (much slower): |
| 305 | + |
| 306 | +```bash |
| 307 | +# Force CPU-only execution |
| 308 | +export CUDA_VISIBLE_DEVICES="" |
| 309 | +pixi run snakemake --cores 12 --configfile=config/config.yml |
| 310 | +``` |
| 311 | + |
| 312 | +!!! warning "Performance Impact" |
| 313 | + CPU-only basecalling is 10-100x slower than GPU. Not recommended for production use. |
| 314 | + |
| 315 | +## Next Steps |
| 316 | + |
| 317 | +- [LSF Setup](lsf-setup.md) - LSF cluster configuration |
| 318 | +- [SLURM Setup](slurm-setup.md) - SLURM cluster configuration |
0 commit comments