Skip to content

Commit c230003

Browse files
edmundmillerclaude
andcommitted
docs: add comprehensive unified package management documentation
Add complete documentation for the new package directive system: - Overview of unified package management architecture - Prerequisites and feature flag enablement - Basic and advanced usage examples with multiple providers - Configuration options and provider-specific settings - Migration guide from legacy conda directive - Best practices and troubleshooting guide - Integration with Wave containers - Support for conda, pixi, and extensible plugin system This documentation provides users with everything needed to adopt the new unified package management system. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> Signed-off-by: Edmund Miller <[email protected]>
1 parent e3b1b47 commit c230003

File tree

1 file changed

+369
-0
lines changed

1 file changed

+369
-0
lines changed

docs/package.md

Lines changed: 369 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,369 @@
1+
(package-page)=
2+
3+
# Package Management
4+
5+
:::{versionadded} 25.04.0-edge
6+
:::
7+
8+
Nextflow provides a unified package management system that allows you to specify dependencies using different package managers through a single, consistent interface. This system supports conda, pixi, and other package managers through a plugin-based architecture.
9+
10+
## Prerequisites
11+
12+
The unified package management system requires:
13+
- The `preview.package` feature flag to be enabled
14+
- The appropriate package manager installed on your system (conda, pixi, etc.)
15+
- The corresponding Nextflow plugin for your chosen package manager
16+
17+
## How it works
18+
19+
Nextflow creates and activates the appropriate environment based on the package specifications and provider you choose. The system abstracts away the differences between package managers, providing a consistent interface regardless of the underlying tool.
20+
21+
## Enabling Package Management
22+
23+
The unified package management system is enabled using the `preview.package` feature flag:
24+
25+
```groovy
26+
// nextflow.config
27+
nextflow.preview.package = true
28+
```
29+
30+
Alternatively, you can enable it with an environment variable:
31+
32+
```bash
33+
export NXF_PREVIEW_PACKAGE=true
34+
```
35+
36+
Or using a command-line option when running Nextflow:
37+
38+
```bash
39+
nextflow run workflow.nf -c <(echo 'nextflow.preview.package = true')
40+
```
41+
42+
## Basic Usage
43+
44+
### Package Directive
45+
46+
Use the `package` directive in your process definitions to specify dependencies:
47+
48+
```nextflow
49+
process example {
50+
package "samtools=1.15 bcftools=1.15", provider: "conda"
51+
52+
script:
53+
"""
54+
samtools --help
55+
bcftools --help
56+
"""
57+
}
58+
```
59+
60+
### Syntax
61+
62+
The basic syntax for the package directive is:
63+
64+
```nextflow
65+
package "<package_specification>", provider: "<provider_name>"
66+
```
67+
68+
- `<package_specification>`: Space-separated list of packages with optional version constraints
69+
- `<provider_name>`: The package manager to use (e.g., "conda", "pixi")
70+
71+
### Multiple Packages
72+
73+
You can specify multiple packages in a single directive:
74+
75+
```nextflow
76+
process analysis {
77+
package "bwa=0.7.17 samtools=1.15 bcftools=1.15", provider: "conda"
78+
79+
script:
80+
"""
81+
bwa mem ref.fa reads.fq | samtools view -bS - | bcftools view
82+
"""
83+
}
84+
```
85+
86+
### Version Constraints
87+
88+
Different package managers support different version constraint syntaxes:
89+
90+
**Conda:**
91+
```nextflow
92+
package "python=3.9 numpy>=1.20 pandas<2.0", provider: "conda"
93+
```
94+
95+
**Pixi:**
96+
```nextflow
97+
package "python=3.9 numpy>=1.20 pandas<2.0", provider: "pixi"
98+
```
99+
100+
## Configuration
101+
102+
### Default Provider
103+
104+
You can set a default provider in your configuration:
105+
106+
```groovy
107+
// nextflow.config
108+
packages {
109+
provider = "conda" // Default provider for all package directives
110+
}
111+
```
112+
113+
### Provider-Specific Settings
114+
115+
Each provider can have its own configuration:
116+
117+
```groovy
118+
// nextflow.config
119+
conda {
120+
enabled = true
121+
cacheDir = "$HOME/.nextflow/conda"
122+
channels = ['conda-forge', 'bioconda']
123+
}
124+
125+
packages {
126+
provider = "conda"
127+
conda {
128+
channels = ['conda-forge', 'bioconda', 'defaults']
129+
useMicromamba = true
130+
}
131+
pixi {
132+
channels = ['conda-forge', 'bioconda']
133+
}
134+
}
135+
```
136+
137+
## Advanced Usage
138+
139+
### Environment Files
140+
141+
You can specify environment files instead of package lists:
142+
143+
```nextflow
144+
process fromFile {
145+
package file("environment.yml"), provider: "conda"
146+
147+
script:
148+
"""
149+
python analysis.py
150+
"""
151+
}
152+
```
153+
154+
### Per-Provider Options
155+
156+
Some providers support additional options:
157+
158+
```nextflow
159+
process withOptions {
160+
package "biopython scikit-learn",
161+
provider: "conda",
162+
channels: ["conda-forge", "bioconda"]
163+
164+
script:
165+
"""
166+
python -c "import Bio; import sklearn"
167+
"""
168+
}
169+
```
170+
171+
## Supported Providers
172+
173+
### Conda
174+
175+
The conda provider supports:
176+
- Package specifications with version constraints
177+
- Custom channels
178+
- Environment files (`.yml`, `.yaml`)
179+
- Micromamba as an alternative backend
180+
181+
```nextflow
182+
process condaExample {
183+
package "bioconda::samtools=1.15 conda-forge::numpy",
184+
provider: "conda"
185+
186+
script:
187+
"""
188+
samtools --version
189+
python -c "import numpy; print(numpy.__version__)"
190+
"""
191+
}
192+
```
193+
194+
### Pixi
195+
196+
The pixi provider supports:
197+
- Package specifications compatible with pixi
198+
- Custom channels
199+
- Project-based environments
200+
201+
```nextflow
202+
process pixiExample {
203+
package "samtools bcftools", provider: "pixi"
204+
205+
script:
206+
"""
207+
samtools --version
208+
bcftools --version
209+
"""
210+
}
211+
```
212+
213+
## Migration from Legacy Directives
214+
215+
### From conda directive
216+
217+
**Before (legacy):**
218+
```nextflow
219+
process oldWay {
220+
conda "samtools=1.15 bcftools=1.15"
221+
222+
script:
223+
"samtools --help"
224+
}
225+
```
226+
227+
**After (unified):**
228+
```nextflow
229+
process newWay {
230+
package "samtools=1.15 bcftools=1.15", provider: "conda"
231+
232+
script:
233+
"samtools --help"
234+
}
235+
```
236+
237+
### Deprecation Warnings
238+
239+
When the unified package management system is enabled, using the legacy `conda` directive will show a deprecation warning:
240+
241+
```
242+
WARN: The 'conda' directive is deprecated when preview.package is enabled.
243+
Use 'package "samtools=1.15", provider: "conda"' instead
244+
```
245+
246+
## Best Practices
247+
248+
### 1. Pin Package Versions
249+
250+
Always specify exact versions for reproducibility:
251+
252+
```nextflow
253+
// Good
254+
package "samtools=1.15 bcftools=1.15", provider: "conda"
255+
256+
// Avoid (may cause reproducibility issues)
257+
package "samtools bcftools", provider: "conda"
258+
```
259+
260+
### 2. Use Appropriate Channels
261+
262+
Specify the most appropriate channels for your packages:
263+
264+
```nextflow
265+
process bioinformatics {
266+
package "bioconda::samtools conda-forge::pandas", provider: "conda"
267+
268+
script:
269+
"""
270+
samtools --version
271+
python -c "import pandas"
272+
"""
273+
}
274+
```
275+
276+
### 3. Group Related Packages
277+
278+
Keep related packages together in the same environment:
279+
280+
```nextflow
281+
process genomicsAnalysis {
282+
package "samtools=1.15 bcftools=1.15 htslib=1.15", provider: "conda"
283+
284+
script:
285+
"""
286+
# All tools are from the same suite and work well together
287+
samtools view input.bam | bcftools view
288+
"""
289+
}
290+
```
291+
292+
### 4. Test Your Environments
293+
294+
Always test your package environments before deploying:
295+
296+
```bash
297+
# Test package resolution
298+
nextflow run test.nf --dry-run -preview
299+
300+
# Test actual execution
301+
nextflow run test.nf -resume
302+
```
303+
304+
## Troubleshooting
305+
306+
### Common Issues
307+
308+
**Package not found:**
309+
- Check package name spelling
310+
- Verify the package exists in specified channels
311+
- Try different channels or provider
312+
313+
**Version conflicts:**
314+
- Relax version constraints if possible
315+
- Check for incompatible package combinations
316+
- Consider using a different provider
317+
318+
**Slow environment creation:**
319+
- Use `useMicromamba = true` for faster conda operations
320+
- Consider pre-built environments
321+
- Use appropriate cache directories
322+
323+
### Environment Inspection
324+
325+
You can inspect created environments using provider-specific commands:
326+
327+
```bash
328+
# For conda environments
329+
conda env list
330+
conda list -n nextflow-env-hash
331+
332+
# For pixi environments
333+
pixi info
334+
```
335+
336+
## Integration with Wave
337+
338+
The package management system integrates seamlessly with Wave containers. When Wave is enabled, environments are automatically containerized:
339+
340+
```groovy
341+
// nextflow.config
342+
wave.enabled = true
343+
nextflow.preview.package = true
344+
```
345+
346+
```nextflow
347+
process containerized {
348+
package "samtools=1.15", provider: "conda"
349+
350+
script:
351+
"""
352+
# This runs in a Wave container with samtools pre-installed
353+
samtools --version
354+
"""
355+
}
356+
```
357+
358+
## Limitations
359+
360+
- The unified package management system is currently in preview
361+
- Plugin availability may vary for different providers
362+
- Some legacy features may not be fully supported yet
363+
- Provider-specific options may be limited
364+
365+
## See Also
366+
367+
- {ref}`conda-page` - Legacy conda directive documentation
368+
- {ref}`config-packages` - Package management configuration options
369+
- {ref}`wave-page` - Wave container integration

0 commit comments

Comments
 (0)