Commit ca02f0a
committed
Simplify to remote include statement:
This dramatically simplifies the implementation by removing unnecessary
complexity (CLI, annotations) and using the clean design requested:
```groovy
include { BOWTIE_ALIGN } from "github:nf-core/modules/modules/bowtie/align@abc123"
```
## What Changed
### Removed (Unnecessary Complexity)
- ❌ CmdModules.groovy - No CLI needed
- ❌ CmdModulesTest.groovy - No CLI tests
- ❌ GrabModule.groovy - No annotation needed
- ❌ GrabModuleTransform.groovy - No AST transform needed
- ❌ Registration in Launcher.groovy
### Added (Core Functionality)
- ✅ Enhanced IncludeDef.groovy - Support remote URLs
- ✅ Remote module resolution via AssetManager
- ✅ Integrity verification via ModuleLockfile
- ✅ Security mode handling (strict/warn/permissive)
- ✅ IncludeDefRemoteTest.groovy - Tests for remote includes
- ✅ modules-remote-includes.md - Comprehensive user guide
### Modified
- ✅ IncludeDef.groovy - Enhanced include statement
- Detects remote URLs (github:, gitlab:, bitbucket:)
- Downloads via AssetManager (same as pipelines)
- Verifies SHA-256 hashes from modules.lock
- Handles security modes
- ✅ modules-command.md - Simplified to quick start guide
## Design Rationale
**Question:** "I really want `include from "github:..."` to work"
**Answer:** Done! This is the cleanest possible design because:
1. **Familiar Pattern** - Same as `nextflow run github:...`
2. **No New Syntax** - Just enhanced existing `include`
3. **Same Infrastructure** - Uses existing AssetManager
4. **Security Built-in** - SHA-256 verification via modules.lock
## How It Works
### First Run:
```
1. Parse remote URL: github:owner/repo/path@revision
2. Download repo via AssetManager (cached in .nextflow/assets/)
3. Calculate SHA-256 hash of module directory
4. Save hash to modules.lock
5. Module ready to use
```
### Subsequent Runs:
```
1. Load expected hash from modules.lock
2. Calculate current hash
3. Compare → Pass/Fail based on security mode
4. Module ready to use (no re-download)
```
## Implementation Details
**IncludeDef.checkValidPath():**
- Before: Threw error on remote URLs
- After: Allows remote URLs, validates format
**IncludeDef.resolveModulePath():**
- Before: Only handled local paths
- After: Detects remote URLs, calls resolveRemoteModule()
**IncludeDef.resolveRemoteModule():**
- New method that handles remote module resolution
- Uses ModuleManager to parse references
- Uses AssetManager to download repositories
- Uses ModuleLockfile for integrity verification
- Handles security modes (strict/warn/permissive)
## Usage Example
```groovy
nextflow.enable.dsl=2
// Remote modules - auto-download, hash, verify
include { FASTQC } from "github:nf-core/modules/modules/nf-core/fastqc@abc123"
include { MULTIQC } from "github:nf-core/modules/modules/nf-core/multiqc@def456"
// Local modules - existing behavior
include { CUSTOM } from "./modules/custom.nf"
workflow {
FASTQC(Channel.fromPath(params.reads))
MULTIQC(FASTQC.out.zip.collect())
}
```
## Security Configuration
```groovy
// nextflow.config
modules {
security = 'strict' // strict|warn|permissive (default: strict)
}
```
## Benefits
**Simplicity:**
- ✅ No CLI commands to learn
- ✅ No annotations to remember
- ✅ Just use familiar `include` statement
**Consistency:**
- ✅ Same pattern as pipeline imports
- ✅ Same infrastructure (AssetManager)
- ✅ Same user experience
**Security:**
- ✅ Cryptographic integrity (SHA-256)
- ✅ Tamper detection
- ✅ Reproducible via modules.lock
**Flexibility:**
- ✅ Three security modes
- ✅ Works offline after first run
- ✅ Supports all Git providers
## Comparison
### Original Issue Request:
```groovy
include { BOWTIE_ALIGN } from "github:nf-core/modules/modules/bowtie/align@somehashabcd"
```
### What We Built:
```groovy
include { BOWTIE_ALIGN } from "github:nf-core/modules/modules/bowtie/align@abc123"
```
**Exactly as requested!** ✨
## Testing
- IncludeDefRemoteTest.groovy - Unit tests
- Remote URL detection
- Path validation
- Security mode handling
- Integration with existing tests
- ModuleLockfileTest.groovy - Integrity verification tests
## Documentation
- modules-remote-includes.md - Comprehensive guide
- Quick start
- Security model
- Best practices
- Migration guide
- FAQ
- modules-command.md - Simplified quick reference
- modules-security-model.md - Security deep dive (existing)
This is the clean, simple design that was requested. No unnecessary
complexity, just enhanced `include` with security.include from "github:...@hash"
1 parent 1a4ef2d commit ca02f0a
File tree
9 files changed
+730
-1121
lines changed- docs
- modules/nextflow/src
- main/groovy/nextflow
- cli
- module
- script
- test/groovy/nextflow
- cli
- script
9 files changed
+730
-1121
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
| 1 | + | |
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
6 | | - | |
7 | | - | |
8 | | - | |
9 | | - | |
10 | | - | |
11 | | - | |
12 | | - | |
13 | | - | |
14 | | - | |
15 | | - | |
| 5 | + | |
16 | 6 | | |
17 | 7 | | |
18 | | - | |
19 | | - | |
20 | | - | |
21 | | - | |
22 | | - | |
23 | | - | |
24 | | - | |
25 | | - | |
| 8 | + | |
26 | 9 | | |
27 | 10 | | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
| 11 | + | |
35 | 12 | | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
40 | | - | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
| 13 | + | |
62 | 14 | | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | 15 | | |
93 | | - | |
94 | | - | |
95 | | - | |
96 | | - | |
97 | | - | |
98 | | - | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
| 16 | + | |
106 | 17 | | |
107 | | - | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | | - | |
112 | | - | |
113 | | - | |
114 | | - | |
115 | | - | |
116 | | - | |
117 | | - | |
118 | | - | |
119 | | - | |
120 | | - | |
121 | | - | |
122 | | - | |
123 | | - | |
124 | | - | |
125 | | - | |
126 | | - | |
127 | | - | |
128 | | - | |
129 | | - | |
130 | | - | |
131 | | - | |
132 | | - | |
133 | | - | |
134 | | - | |
135 | | - | |
136 | | - | |
137 | | - | |
138 | | - | |
139 | | - | |
140 | | - | |
141 | | - | |
142 | | - | |
143 | | - | |
144 | | - | |
145 | | - | |
146 | | - | |
147 | | - | |
148 | | - | |
149 | | - | |
150 | | - | |
151 | | - | |
152 | | - | |
153 | | - | |
154 | | - | |
155 | | - | |
156 | | - | |
157 | | - | |
158 | | - | |
159 | | - | |
160 | | - | |
161 | | - | |
162 | | - | |
163 | | - | |
164 | | - | |
165 | | - | |
166 | | - | |
167 | | - | |
168 | | - | |
169 | | - | |
170 | | - | |
171 | | - | |
172 | | - | |
173 | | - | |
174 | | - | |
175 | | - | |
176 | | - | |
177 | | - | |
178 | | - | |
179 | | - | |
180 | | - | |
181 | | - | |
182 | | - | |
183 | | - | |
184 | | - | |
185 | | - | |
186 | | - | |
187 | | - | |
188 | | - | |
189 | | - | |
190 | | - | |
191 | | - | |
192 | | - | |
193 | | - | |
194 | | - | |
195 | | - | |
196 | | - | |
197 | | - | |
198 | | - | |
199 | | - | |
200 | | - | |
201 | | - | |
202 | | - | |
203 | | - | |
204 | | - | |
205 | | - | |
206 | | - | |
207 | | - | |
208 | | - | |
209 | | - | |
210 | | - | |
211 | | - | |
212 | | - | |
213 | | - | |
214 | | - | |
215 | | - | |
216 | | - | |
217 | | - | |
218 | | - | |
219 | | - | |
220 | | - | |
221 | | - | |
222 | | - | |
223 | | - | |
224 | | - | |
225 | | - | |
226 | | - | |
227 | | - | |
228 | | - | |
229 | | - | |
230 | | - | |
231 | | - | |
232 | | - | |
| 18 | + | |
233 | 19 | | |
234 | 20 | | |
235 | 21 | | |
236 | 22 | | |
237 | | - | |
238 | | - | |
239 | | - | |
240 | | - | |
241 | | - | |
242 | | - | |
243 | | - | |
244 | | - | |
245 | | - | |
246 | | - | |
247 | | - | |
248 | | - | |
249 | | - | |
250 | | - | |
251 | | - | |
252 | | - | |
253 | | - | |
254 | | - | |
255 | | - | |
256 | | - | |
257 | | - | |
258 | | - | |
259 | | - | |
260 | | - | |
261 | | - | |
262 | | - | |
263 | | - | |
264 | | - | |
265 | | - | |
266 | | - | |
267 | | - | |
268 | | - | |
| 23 | + | |
| 24 | + | |
269 | 25 | | |
270 | 26 | | |
271 | | - | |
272 | | - | |
273 | | - | |
274 | | - | |
275 | | - | |
276 | | - | |
| 27 | + | |
277 | 28 | | |
278 | | - | |
279 | | - | |
280 | | - | |
281 | | - | |
282 | 29 | | |
283 | 30 | | |
284 | | - | |
285 | | - | |
286 | | - | |
287 | | - | |
288 | | - | |
289 | | - | |
290 | | - | |
291 | | - | |
292 | | - | |
293 | | - | |
294 | | - | |
295 | | - | |
296 | | - | |
297 | | - | |
298 | | - | |
299 | | - | |
300 | | - | |
301 | | - | |
302 | | - | |
303 | | - | |
304 | | - | |
305 | | - | |
306 | | - | |
307 | | - | |
308 | | - | |
309 | | - | |
310 | | - | |
311 | | - | |
312 | | - | |
313 | | - | |
314 | | - | |
315 | | - | |
316 | | - | |
317 | | - | |
318 | | - | |
319 | | - | |
320 | | - | |
321 | | - | |
322 | | - | |
323 | | - | |
324 | | - | |
325 | | - | |
326 | | - | |
327 | | - | |
328 | | - | |
329 | | - | |
330 | | - | |
| 31 | + | |
331 | 32 | | |
332 | | - | |
333 | | - | |
| 33 | + | |
334 | 34 | | |
335 | 35 | | |
336 | | - | |
337 | | - | |
338 | | - | |
339 | | - | |
340 | | - | |
341 | | - | |
342 | | - | |
343 | | - | |
344 | | - | |
345 | | - | |
346 | | - | |
347 | | - | |
348 | | - | |
349 | | - | |
350 | | - | |
351 | | - | |
352 | | - | |
353 | | - | |
354 | | - | |
355 | | - | |
356 | | - | |
357 | | - | |
358 | | - | |
359 | | - | |
360 | | - | |
361 | | - | |
362 | | - | |
363 | | - | |
364 | | - | |
365 | | - | |
366 | | - | |
367 | | - | |
368 | | - | |
| 36 | + | |
0 commit comments