Skip to content

Commit 6496235

Browse files
docs: add comprehensive MkDocs documentation website
Add a complete documentation site using MkDocs with Material theme: - Getting Started: installation, quickstart, first analysis guides - User Guide: configuration, sample formats, running pipeline, outputs - Workflow: overview with mermaid diagrams, rules reference (all 21 rules), scripts reference (8 Python scripts), WarpDemuX demultiplexing guide - Cluster Setup: LSF, SLURM, and GPU configuration guides - Troubleshooting: common errors, FAQ, debugging techniques - Development: architecture, adding rules, contributing guidelines Infrastructure: - mkdocs.yml with Material theme, mermaid support, code highlighting - GitHub Actions workflow for auto-deployment to GitHub Pages - Updated README with documentation badge and links Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 9ab215b commit 6496235

24 files changed

+6321
-1
lines changed

.github/workflows/docs.yml

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
name: Deploy Documentation
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
- docs
8+
paths:
9+
- 'docs/**'
10+
- 'mkdocs.yml'
11+
- '.github/workflows/docs.yml'
12+
workflow_dispatch:
13+
14+
permissions:
15+
contents: read
16+
pages: write
17+
id-token: write
18+
19+
concurrency:
20+
group: "pages"
21+
cancel-in-progress: false
22+
23+
jobs:
24+
build:
25+
runs-on: ubuntu-latest
26+
steps:
27+
- name: Checkout
28+
uses: actions/checkout@v4
29+
30+
- name: Setup Python
31+
uses: actions/setup-python@v5
32+
with:
33+
python-version: '3.11'
34+
35+
- name: Install dependencies
36+
run: |
37+
pip install mkdocs mkdocs-material pymdown-extensions
38+
39+
- name: Build documentation
40+
run: mkdocs build --strict
41+
42+
- name: Setup Pages
43+
uses: actions/configure-pages@v4
44+
45+
- name: Upload artifact
46+
uses: actions/upload-pages-artifact@v3
47+
with:
48+
path: 'site'
49+
50+
deploy:
51+
environment:
52+
name: github-pages
53+
url: ${{ steps.deployment.outputs.page_url }}
54+
runs-on: ubuntu-latest
55+
needs: build
56+
steps:
57+
- name: Deploy to GitHub Pages
58+
id: deployment
59+
uses: actions/deploy-pages@v4

README.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,11 @@
22

33
[![CI](https://github.com/rnabioco/aa-tRNA-seq-pipeline/actions/workflows/ci.yml/badge.svg)](https://github.com/rnabioco/aa-tRNA-seq-pipeline/actions/workflows/ci.yml)
44
[![Lint](https://github.com/rnabioco/aa-tRNA-seq-pipeline/actions/workflows/lint.yml/badge.svg)](https://github.com/rnabioco/aa-tRNA-seq-pipeline/actions/workflows/lint.yml)
5+
[![Documentation](https://img.shields.io/badge/docs-GitHub%20Pages-blue)](https://rnabioco.github.io/aa-tRNA-seq-pipeline)
56

67
A Snakemake pipeline to process ONT aa-tRNA-seq data.
78

8-
Downstream analysis to generate figures for the initial preprint can be found at: [https://github.com/rnabioco/aa-tRNA-seq](https://github.com/rnabioco/aa-tRNA-seq)
9+
**[Documentation](https://rnabioco.github.io/aa-tRNA-seq-pipeline)** | **[Downstream Analysis](https://github.com/rnabioco/aa-tRNA-seq)**
910

1011
## Usage
1112

docs/cluster/gpu-configuration.md

Lines changed: 318 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,318 @@
1+
# GPU Configuration
2+
3+
Configure GPU resources for the aa-tRNA-seq pipeline.
4+
5+
## GPU Requirements
6+
7+
Two rules require GPU access:
8+
9+
| Rule | Purpose | GPU Usage |
10+
|------|---------|-----------|
11+
| `rebasecall` | Dorado basecalling | CUDA neural network inference |
12+
| `classify_charging` | Remora classification | PyTorch model inference |
13+
14+
Both rules benefit significantly from GPU acceleration. CPU-only execution is possible but substantially slower.
15+
16+
## GPU Resource Flow
17+
18+
```mermaid
19+
flowchart LR
20+
subgraph GPU Rules
21+
A[rebasecall<br/>Dorado] --> B[classify_charging<br/>Remora]
22+
end
23+
24+
subgraph Resources
25+
C[POD5 Signal Data]
26+
D[CUDA GPU]
27+
end
28+
29+
C --> A
30+
D --> A
31+
D --> B
32+
```
33+
34+
## Cluster Configuration
35+
36+
### LSF GPU Settings
37+
38+
In `cluster/lsf/config.yaml`:
39+
40+
```yaml
41+
# Limit total concurrent GPU jobs
42+
resources:
43+
- ngpu=12
44+
45+
# GPU rule configuration
46+
set-resources:
47+
- rebasecall:lsf_queue="gpu"
48+
- rebasecall:lsf_extra="-gpu num=1:j_exclusive=yes"
49+
- rebasecall:ngpu=1
50+
- rebasecall:mem_mb=24
51+
52+
- classify_charging:lsf_queue="gpu"
53+
- classify_charging:lsf_extra="-gpu num=1:j_exclusive=yes"
54+
- classify_charging:ngpu=1
55+
- classify_charging:mem_mb=24
56+
```
57+
58+
### SLURM GPU Settings
59+
60+
```yaml
61+
resources:
62+
- ngpu=8
63+
64+
set-resources:
65+
- rebasecall:partition="gpu"
66+
- rebasecall:gpu_opts="--gres=gpu:1"
67+
- rebasecall:ngpu=1
68+
- rebasecall:mem_mb=24000
69+
70+
- classify_charging:partition="gpu"
71+
- classify_charging:gpu_opts="--gres=gpu:1"
72+
- classify_charging:ngpu=1
73+
- classify_charging:mem_mb=24000
74+
```
75+
76+
## Configuration Options
77+
78+
### GPU Concurrency Limit
79+
80+
Control how many GPU jobs run simultaneously:
81+
82+
```yaml
83+
resources:
84+
- ngpu=8 # Max 8 concurrent GPU jobs
85+
```
86+
87+
Set this to match your available GPUs or queue limits.
88+
89+
### Exclusive GPU Access
90+
91+
Request exclusive GPU access to avoid memory conflicts:
92+
93+
=== "LSF"
94+
95+
```yaml
96+
set-resources:
97+
- rebasecall:lsf_extra="-gpu num=1:j_exclusive=yes"
98+
```
99+
100+
=== "SLURM"
101+
102+
```yaml
103+
set-resources:
104+
- rebasecall:gpu_opts="--gres=gpu:1 --exclusive"
105+
```
106+
107+
### GPU Type Selection
108+
109+
If your cluster has multiple GPU types:
110+
111+
=== "LSF"
112+
113+
```yaml
114+
set-resources:
115+
- rebasecall:lsf_extra="-gpu num=1:j_exclusive=yes:gtile='!gv100'"
116+
```
117+
118+
=== "SLURM"
119+
120+
```yaml
121+
set-resources:
122+
- rebasecall:gpu_opts="--gres=gpu:v100:1"
123+
```
124+
125+
## Local GPU Execution
126+
127+
### CUDA_VISIBLE_DEVICES
128+
129+
The pipeline respects `CUDA_VISIBLE_DEVICES`:
130+
131+
```bash
132+
# Use specific GPU
133+
export CUDA_VISIBLE_DEVICES=0
134+
pixi run snakemake --cores 4 --configfile=config/config.yml
135+
136+
# Use multiple GPUs (one per job)
137+
export CUDA_VISIBLE_DEVICES=0,1
138+
pixi run snakemake --cores 4 --resources gpu=2 --configfile=config/config.yml
139+
```
140+
141+
### Limit GPU Jobs Locally
142+
143+
```bash
144+
pixi run snakemake --cores 8 --resources gpu=1 \
145+
--configfile=config/config.yml
146+
```
147+
148+
## Memory Requirements
149+
150+
GPU rules also require significant system memory:
151+
152+
| Rule | GPU Memory | System Memory |
153+
|------|------------|---------------|
154+
| `rebasecall` | ~8-16 GB | 24 GB |
155+
| `classify_charging` | ~4-8 GB | 24 GB |
156+
157+
## Performance Considerations
158+
159+
### Dorado (rebasecall)
160+
161+
- Processes POD5 signal data through neural network
162+
- Throughput: ~100-500 reads/second depending on GPU
163+
- Benefits from newer GPU architectures (Ampere, Ada Lovelace)
164+
165+
### Remora (classify_charging)
166+
167+
- Analyzes signal at CCA 3' end
168+
- Lower throughput than Dorado
169+
- Memory usage depends on batch size
170+
171+
## Troubleshooting
172+
173+
### CUDA Out of Memory
174+
175+
**Symptom:**
176+
```
177+
RuntimeError: CUDA out of memory
178+
```
179+
180+
**Solutions:**
181+
182+
1. Ensure exclusive GPU access:
183+
```yaml
184+
set-resources:
185+
- rebasecall:lsf_extra="-gpu num=1:j_exclusive=yes"
186+
```
187+
188+
2. Reduce concurrent GPU jobs:
189+
```yaml
190+
resources:
191+
- ngpu=4 # Reduce from default
192+
```
193+
194+
3. Check for other GPU processes:
195+
```bash
196+
nvidia-smi
197+
```
198+
199+
### GPU Not Detected
200+
201+
**Symptom:**
202+
```
203+
No CUDA GPUs are available
204+
```
205+
206+
**Solutions:**
207+
208+
1. Verify CUDA installation:
209+
```bash
210+
nvidia-smi
211+
```
212+
213+
2. Check CUDA_VISIBLE_DEVICES:
214+
```bash
215+
echo $CUDA_VISIBLE_DEVICES
216+
```
217+
218+
3. Verify job is on GPU node:
219+
```bash
220+
# LSF
221+
bjobs -l <job_id> | grep -i gpu
222+
223+
# SLURM
224+
scontrol show job <job_id> | grep -i gres
225+
```
226+
227+
### Wrong GPU Type
228+
229+
**Symptom:**
230+
Job runs on incompatible GPU.
231+
232+
**Solutions:**
233+
234+
Specify GPU type explicitly in cluster profile:
235+
236+
=== "LSF"
237+
238+
```yaml
239+
set-resources:
240+
- rebasecall:lsf_extra="-gpu num=1:j_exclusive=yes:gmodel=NVIDIAA100"
241+
```
242+
243+
=== "SLURM"
244+
245+
```yaml
246+
set-resources:
247+
- rebasecall:gpu_opts="--gres=gpu:a100:1"
248+
```
249+
250+
### Jobs Waiting for GPU
251+
252+
**Symptom:**
253+
GPU jobs pending indefinitely.
254+
255+
**Solutions:**
256+
257+
1. Check GPU queue status:
258+
```bash
259+
# LSF
260+
bqueues -l gpu
261+
262+
# SLURM
263+
sinfo -p gpu
264+
```
265+
266+
2. Reduce concurrent GPU jobs:
267+
```yaml
268+
resources:
269+
- ngpu=2
270+
```
271+
272+
3. Check fair share limits with your admin.
273+
274+
## GPU Monitoring
275+
276+
### NVIDIA SMI
277+
278+
Monitor GPU usage during execution:
279+
280+
```bash
281+
# Watch GPU utilization
282+
watch -n 1 nvidia-smi
283+
284+
# Log GPU stats
285+
nvidia-smi --query-gpu=timestamp,name,utilization.gpu,utilization.memory,memory.used --format=csv -l 1 > gpu_log.csv
286+
```
287+
288+
### Check Running GPU Jobs
289+
290+
=== "LSF"
291+
292+
```bash
293+
bjobs -u $USER -q gpu
294+
```
295+
296+
=== "SLURM"
297+
298+
```bash
299+
squeue -u $USER -p gpu
300+
```
301+
302+
## CPU Fallback
303+
304+
If GPUs are unavailable, Dorado can run on CPU (much slower):
305+
306+
```bash
307+
# Force CPU-only execution
308+
export CUDA_VISIBLE_DEVICES=""
309+
pixi run snakemake --cores 12 --configfile=config/config.yml
310+
```
311+
312+
!!! warning "Performance Impact"
313+
CPU-only basecalling is 10-100x slower than GPU. Not recommended for production use.
314+
315+
## Next Steps
316+
317+
- [LSF Setup](lsf-setup.md) - LSF cluster configuration
318+
- [SLURM Setup](slurm-setup.md) - SLURM cluster configuration

0 commit comments

Comments
 (0)