Recommended way to adapt computational resources for rules

[bednarsky]

• asking this questions here, since I think it's a MrBiomics wide issue
• I am working on testing across genomes, technologies, read types rnaseq_pipeline#2
• rule align runs out of time for me
• I think we discussed this already somewhere but couldn't find it: What's the recommended way to change resource specifications for rules?
• In my case have to change multiple things: go to a higher queue since I need more runtime, and change the runtime
• Via the config one can only change the memory
• Changing this via my general slurm profile cannot be the best way, since this would affect everything I ever run, but only for this one rule I want to change the specifications
• In some of my other projects I do it like this, which is flexibly defined by the user in his config, but somewhat complicated

config.yaml

cluster:
  tiny:
    slurm_partition: tinyq
    slurm_extra: "--qos=tinyq"
    mem_mb: 64000
    runtime: 2h

rules.smk

rule scvi_integration_per_major_subset:
    input:
        ...
    output:
        ...
    resources:
        **config['cluster']['tiny']  # this way of specifying it makes each individual param optional

• In this way, one can have multiple settings, e.g., short_job, long_job, short_job_highmem, long_jobs_highmem, and everyone can specify the resources they want to give for each of them
• Works for me, not sure it's generalizable to every usecase

[bednarsky]

One thing I already noted is that dynamic resource allocation is not straightforward with this approach..

[sreichl]

Profiles is the way to go, although maybe rather workflow profile than global profile.
What you have to add and what is supported depends on the underlying and user specific executor-plugin used.
You are (to some degree) at the mercy of the executor-plugin developer.
For the slurm-executor-plugin that you are using, here is the answer to adapt resource specifically for certain rules: https://snakemake.github.io/snakemake-plugin-catalog/plugins/executor/slurm.html#using-configuration-profiles-for-resource-specifications

Let me know if this answers the question and if/how/where we should doc this considering this is a Snakemake issue not MrBiomics (we want to have minimal general Snakemake content here, because it's not maintainable and insufficient compared to the official docs)

[bednarsky]

But changing the profile would affect all rules, right? What if I, as a user, only want to change the resources for the rules that have high resource need, not all of them? That's not something we support without changing source code?

[sreichl]

That's what the special_rule example shows, right?

[bednarsky]

Actually also easy to solve this without profile, and I did it a few times already but just forgot 😁
But probably special rules in profiles would actually be a better solution!

  module spilterlize_integrate_L2:
      snakefile:
          PROJECT_ROOT / 'modules/spilterlize_integrate/workflow/Snakefile'
      config:
          config
  use rule * from spilterlize_integrate_L2 as spilterlize_integrate_L2_*
  # avoid OOM
  use rule plot_heatmap from spilterlize_integrate_L2 as spilterlize_integrate_L2_plot_heatmap with:
      resources:
          **config['cluster']['tiny']
  use rule plot_diagnostics from spilterlize_integrate_L2 as spilterlize_integrate_L2_plot_diagnostics with:
      resources:
          **config['cluster']['tiny']

[sreichl]

Yes, profiles would be much better for the simple reason that this solution is not portable (i.e., cluster specific). A user should never have to edit the workflow to make it work, right?

[bednarsky]

But in the last solution the user doesn't alter the workflow, but only the way it is importet.

Very similar in mindset to a user specific cluster profile, but less powerful, so I would stick to recommend the cluster profiles for sure! Just put it here for completeness

[sreichl]

Maybe I do not understand.

This part

resources:
          **config['cluster']['tiny']

is cluster specific (as well as the corresponding config), right? Hence, this would not work e.g., locally on your MacBook or on Azure as these concepts (e.g., partition, qos, etc.) are not present (caveat: I am guessing what the actual content of **config['cluster']['tiny'] is).

[bednarsky]

But the workflow specific profile you would also have to change if I were to run the code on my MacBook. So here you don't change the workflow specific profile but the config.yaml. But it's a solution I made myself instead of using the one provided by snakemake, so I am sure it's worse!

[burtonjake]

@fwzhao also had the same query, also for the RNA-seq pipeline. See this comment above :)