param priority in multi-config

[suzannejin]

Overview

This issue addresses parameter precedence in the pipeline's multi-configuration system. After extensive discussion, we've reached consensus on a dual-mode approach that separates single-run production use (via config profiles) from multi-run benchmarking use (via paramsheet).

---

✅ Final Decision (2026-02-05)

Two-Mode Architecture

1. Single-Run Mode (Config Profiles) - For production/GxP use

nextflow run nf-core/differentialabundance \
  -profile rnaseq_deseq2 \
  --limma_use_voom=false

• Standard Nextflow precedence: CLI > profile > defaults
• CLI flags override profile parameters
• Covers majority of production users

2. Multi-Run Mode (Paramsheet) - For benchmarking/exploration

nextflow run nf-core/differentialabundance \
  --paramsheet my_configs.yaml

• Paramsheet precedence: paramsheet > CLI > defaults
• Each row defines parameter overrides
• Runs multiple configurations in parallel
• CLI params generally don't override paramsheet (by design)
• No default paramsheet in pipeline (users provide their own)

Key Design Principles

1. Separation of concerns: Profiles handle production paths, paramsheet handles benchmarking
2. No functionality overlap: Profile-related configurations (e.g., deseq2+rnaseq+gsea) stay in config profiles, NOT in paramsheet
3. Minimal maintenance: No default paramsheet to maintain
4. Clear precedence: Each mode has well-defined, documented precedence rules

Timeline

• Target: End of March 2026
• Goal: Move standard configurations back to profile approach for meaningful end-to-end testing

---

🔍 Problem Background

Original Issue

Paramsheet parameters had highest priority, preventing CLI overrides:

Current (problematic):

paramsheet > CLI flags / -params-file > defaults

Initially desired:

CLI flags / -params-file > paramsheet > defaults

However, achieving this in Nextflow required distinguishing CLI params from config defaults, which proved technically challenging.

---

💭 Discussion Summary

Solutions Explored

❌ Option 1: session.cliParams (Initially Preferred)

Approach: Use Nextflow's internal session.cliParams API
Verdict: Rejected - Internal API, incompatible with strict syntax (26.04+)

Key feedback from @bentsherman:

The Session class is internal to the Nextflow codebase, so it isn't meant to be accessed from a Nextflow script. It won't be accessible in the strict syntax, which will be enabled by default in 26.04

❌ Option 2: Schema-Based Detection

Approach: Compare params against nextflow_schema.json defaults to detect user-specified params
Verdict: Rejected - Edge case with bad failure mode

Critical edge case: If user explicitly sets param to schema default to override paramsheet, it won't be detected

• Example: schema default param: "foo", paramsheet sets "bar", user runs --param=foo → still gets "bar"
• Failure mode: Silent - user won't know override was ignored
• Unacceptable for GxP environments

❌ Option 3: Null-Based Approach

Approach: Set all params to null in nextflow.config, handle defaults via paramsheet
Verdict: Rejected - Diverges too far from nf-core conventions, unclear benefit

❌ Option 4: CLI Parsing

Approach: Parse workflow.commandLine with regex
Verdict: Rejected - Fragile, high maintenance burden

Core Design Philosophy (from @bentsherman)

I think there are two core assumptions in Nextflow that you will run up against:

1. Params define what is computed, config defines how it is computed
2. The pipeline script only knows the final resolved params, it does not know whether a given param came from CLI / params file / config

This insight led to reconsidering the entire approach and recognizing that trying to force CLI overrides on paramsheet violated Nextflow's design principles.

---

🎯 Implementation Plan

Phase 1: Revert to Profile + Paramsheet Separation

1. Move standard configurations (deseq2+rnaseq+gsea, etc.) back to config profiles
2. Remove default paramsheet from pipeline
3. Document two-mode usage patterns clearly
4. Paramsheet remains available for multi-run benchmarking, but as an advanced feature

Phase 2: Testing & Validation

1. End-to-end testing with profile-based approach
2. Validate GxP-compatible single-run mode
3. Test multi-run benchmarking scenarios

Phase 3: Documentation

1. Clear separation of single-run vs multi-run modes
2. Document precedence rules for each mode
3. Usage examples for both patterns

---

🔗 Related

• Issue unification of paramsheet and profile #465 - Initial discussion on favoring paramset over profile
• PR Fix parameter priority using Nextflow session variables #623 - Implementation attempt using session.cliParams
• PR [Luna] Fix CLI parameter priority over paramsheet values #624 - Follow-up implementation with filtering
• Module interfaces (nextflow#6737) - Future Nextflow feature that may enable cleaner solutions
• Module params (nextflow#6769) - Future Nextflow feature

---

📝 Long-Term Vision

Once Nextflow natively supports iterating over parameter configurations (via module interfaces, module params, or similar features), we may revisit unifying the two approaches. Until then, the dual-mode architecture provides a pragmatic solution that:

• ✅ Supports production/GxP use cases with standard precedence
• ✅ Enables advanced benchmarking workflows
• ✅ Maintains nf-core standards compliance
• ✅ Avoids technical edge cases and silent failures

[grst]

I can see your point... the parameters in the paramsheet would be method-specific though, so I would find it a bit confusing if they were overriden by global parameters.

Really not sure what's more intuitive behaviour.

[grst]

Current situation:

paramsheet > params > default params

Use case: run a single paramset

I think in this case it's pretty clear that the desired behavior would be params > paramsheet.
The user specifies default parameters, e.g. "-paramsheet rnaseq_deseq2" and then overrides individual parameters to customize.
In fact, it is pretty confusion if one specifies a parameter on the command line that has no effect because it will be overridden by the paramsheet.

Current workaround

Instead of -params-file params.yaml, specify a --paramsheet params.yaml and use that as the only place to specify parameters. The downsides:

• this doesn't work well with seqera platform (can't use the UI defined by the nextflow schema)
• it's different than what's recommended in other workflows
• some parameters (those that are referenced within closures in nextflow config files), e.g. --outdir, can't be set in the paramsheet.

Use case: run multiple methods

Here it becomes more interesting, as potentially, we'd like to have different parameters for different methods. But the more I think about it, the more it seems reasonable to also give precedence to the command-line flags, and simply not specify such flags for parameters you want to set in the paramsheet.

summary

==> in summary: I agree with the behavior you initially proposed.

Do you have an idea how to implement it? I feel it could be tricky to find out if a parameters is the default one or has been set via command line flags or -params-file.

[suzannejin]

Possible implementation

Do you have an idea how to implement it? I feel it could be tricky to find out if a parameters is the default one or has been set via command line flags or -params-file.

Hi @grst , I think we can make use of getDefaultConfigurations() and getParamsheetConfigurations() that are implemented here.

The idea is to

1. get the params scope using getDefaultConfigurations()
2. get the default parameters from the schema. This can be done by running getParamsheetConfigurations() with an almost empty paramsheet - but better ways may exist.
3. then the difference between params scope and the default values would be the ones provided by the user through flag.

probably there are better ways to extract the cmd flags though. open for suggestions @mirpedrol @pinin4fjords @SusiJo

Other issues

this doesn't work well with seqera platform (can't use the UI defined by the nextflow schema)

May you explain a bit more about how this works? To understand how big the problem is?

some parameters (those that are referenced within closures in nextflow config files), e.g. --outdir, can't be set in the paramsheet.

At some point I implemented the folllowing to deal with projectDir closure, but probably something more generic can be implemented to deal with any closure.

    // Substitute ${projectDir} with actual value
    // alternative ways? This can be fragile
    yaml_content = yaml_content.replaceAll(/\$\{projectDir\}/, projectDir.toString())
    yaml_content = yaml_content.replaceAll(/\$projectDir/, projectDir.toString())

[grst]

Hi @suzannejin,

1. get the params scope using getDefaultConfigurations()
2. get the default parameters from the schema. This can be done by running getParamsheetConfigurations() with an almost empty paramsheet - but better ways may exist.
3. then the difference between params scope and the default values would be the ones provided by the user through flag.

I think this would break if you have a default value defined in nextflow.config, which is overridden in a paramsheet and then set back to the default value via params, i.e.

# nextflow.config
some_param = "foo"

# paramsheet.yaml
- name: MyConfig
  some_param: bar

# command-line
nextflow run --paramset MyConfig --some_param=foo

probably there are better ways to extract the cmd flags though.

Maybe by going back to re-parsing the command line? In any case, I think not only command line flags should be considered, but also JSON/YAML that's passed via -param-file.

Another option could be to set all params to null in nextflow.config and exclusively rely on the paramsheet to set defaults. Then we'd know every non-null param to be set by the user.

---

this doesn't work well with seqera platform (can't use the UI defined by the nextflow schema)

May you explain a bit more about how this works? To understand how big the problem is?

With a standard pipeline users have a form like this to set all parameters. If parameters cannot be set via -params-file this form could only be used to specify a path to the paramsheet.yaml which has to be filled and uploaded to a cloud-accessible storage in some other way. It's possible, but quite inconvenient, and undermines one of the main selling points of Seqera platform: that pipelines can be launched by non-technical users via Web UI.

---

some parameters (those that are referenced within closures in nextflow config files), e.g. --outdir, can't be set in the paramsheet.

At some point I implemented the folllowing to deal with projectDir closure, but probably something more generic can be implemented to deal with any closure.

I think there's a misunderstanding here. It's not about supporting closures in the paramsheet.yaml. The problem is that in
modules.config, we have something like

publishDir: { "$params.outdir/results" }

This will access params directly and does not have access to the custom paramsheet logic. Therefore, setting outdir in the paramsheet has no effect.

[pinin4fjords]

Thanks for the thorough discussion. I've been thinking about this and wanted to sketch out a possible approach - keen to hear if others see issues with it or have better ideas.

Possible Approach: Schema-Aware Parameter Layering

The root issue is that params contains both user-specified values and defaults mixed together - Nextflow doesn't expose which is which. However, nextflow_schema.json contains the authoritative defaults, so we could potentially use it to detect user-specified params.

Rough Implementation

/**
 * Extract default values from nextflow_schema.json
 */
def getSchemaDefaults() {
    def schema = new groovy.json.JsonSlurper().parseText(
        new File("${projectDir}/nextflow_schema.json").text
    )
    def defaults = [:]
    
    schema['$defs']?.each { groupName, group ->
        group.properties?.each { paramName, paramProps ->
            if (paramProps.containsKey('default')) {
                defaults[paramName] = paramProps.default
            }
        }
    }
    return defaults
}

/**
 * Identify params that differ from schema defaults (i.e., user-specified)
 */
def getUserSpecifiedParams() {
    def schemaDefaults = getSchemaDefaults()
    def userSpecified = [:]
    
    params.each { key, value ->
        if (!schemaDefaults.containsKey(key) || schemaDefaults[key] != value) {
            userSpecified[key] = value
        }
    }
    return userSpecified
}

Then in getParamsheetConfigurations():

def schemaDefaults = getSchemaDefaults()
def userSpecified = getUserSpecifiedParams()

return paramsheet.collect { row ->
    // Precedence (low to high): schema defaults -> paramsheet -> user-specified
    def fullparamset = schemaDefaults + row + userSpecified
    return fullparamset
}

How This Would Address the Concerns

-params-file support: Values from -params-file end up in params and would diff against schema defaults, so they should be detected as user-specified. No special handling needed.

Seqera Platform: Users could use the schema-driven UI normally. Their values flow into params and would take precedence over paramsheet values.

Closure-referenced params (e.g., outdir in publishDir): These access params directly before our logic runs. I think we'd need to document these as "must be set via command line or -params-file, not paramsheet" and maintain a clear list of affected params. Open to ideas here if anyone has a cleaner solution.

Known Limitation

As @grst identified, if a user explicitly sets a param to its schema default value to override a paramsheet, the schema-diff approach won't detect it:

# schema default: some_param = "foo"
# paramsheet sets: some_param = "bar"  
# user runs: --some_param=foo  -> will get "bar" not "foo"

I suspect this edge case is rare enough that documenting it might be acceptable? If someone hits this, they could modify their paramsheet or run without the paramset feature for that run.

The alternative (setting all params to null in nextflow.config) would be a fairly big breaking change - not sure if it's worth it for this edge case, but interested in others' thoughts.

Resulting Precedence

Priority	Source
Highest	Command line / -params-file (detected via schema diff)
Medium	Paramsheet
Lowest	Schema defaults

---

This might be missing something obvious - let me know if there are holes in this approach or if there's a simpler way to tackle it.

[pinin4fjords]

(yes, that is AI- generated, but the result of a bit of a chat between me and Claude, I think it has a good point)

[grst]

The alternative (setting all params to null in nextflow.config) would be a fairly big breaking change - not sure if it's worth it for this edge case, but interested in others' thoughts.

Not sure if it would be as bad. The same defaults can be defined in the paramsheet, e.g.

- name: default
  study_name: "study"
  # ... (everything usually defined in `nextflow.config`)

- name: rnaseq  
  include: "default"

such that for the user nothing changes. Apart from that I think we are at a point where breaking changes compared to v1.x are a minor concern.

I suspect this edge case is rare enough that documenting it might be acceptable? If someone hits this, they could modify their paramsheet or run without the paramset feature for that run.

Not sure how uncommon it is. E.g. for a boolean --limma_use_voom it is false by default. Then in rnaseq_limma it is set to true. Now I happen to have RNA-seq data that is already pre-transformed. I still want to use limma with rnaseq preset, but disable voom.

In any case I think that it has a undesirable failure mode: it will pass silently and potentially do something completly different than the user expects.