Problem with determining cpus?

[wdecoster]

Bug report

Expected behavior and actual behavior

A nextflow pipeline I am running crashes with this error, while I was using --max_threads 12. I am very much new to nextflow, so please let me know what I can try to debug.

Error executing process > 'pipeline:process_bams:align_to_transcriptome (1)'

Caused by:
  Process requirement exceeds available CPUs -- req: 6; avail: 4

Command executed:

  minimap2 -ax map-ont         --end-bonus 10         -t 6         transcriptome.fa         reads.fastq         | samtools view -F 2304         |  gawk 'BEGIN{OFS="    ";} /^[^@]/ {print $1,$3}'         > read_query_tr_map.tsv;

Command exit status:
  -

Command output:
  (empty)

Work dir:
  /home/wdecoster/sc-ONT/wf-single-cell/work/e6/721cd43c7a30d7bd23aa7357dcfb45

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

Steps to reproduce the problem

(Provide a test case that reproduce the problem either with a self-contained script or GitHub repository)

Program output

(Copy and paste here output produced by the failing execution. Please highlight it as a code block. Whenever possible upload the .nextflow.log file.)

Environment

• Nextflow version: [?]

  version 22.10.6 build 5843
  created 23-01-2023 23:20 UTC 
  

• Java version: [?]

openjdk version "17.0.3-internal" 2022-04-19
OpenJDK Runtime Environment (build 17.0.3-internal+0-adhoc..src)
OpenJDK 64-Bit Server VM (build 17.0.3-internal+0-adhoc..src, mixed mode, sharing)

• Operating system: [macOS, Linux, etc]

Ubuntu Linux

• Bash version: (use the command $SHELL --version)

GNU bash, version 5.1.16(1)-release (x86_64-pc-linux-gnu)

[pontus]

Is this a desktop machine or something else? lscpu output?

[wdecoster]

This is a server:

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          12
On-line CPU(s) list:             0-11
Thread(s) per core:              2
Core(s) per socket:              6
Socket(s):                       1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           154
Model name:                      12th Gen Intel(R) Core(TM) i7-1265U
Stepping:                        4
CPU MHz:                         2688.013
BogoMIPS:                        5376.02
Virtualization:                  VT-x
Hypervisor vendor:               Microsoft
Virtualization type:             full
L1d cache:                       288 KiB
L1i cache:                       192 KiB
L2 cache:                        7.5 MiB
L3 cache:                        12 MiB
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology tsc_reliable no
                                 nstop_tsc cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single 
                                 ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xs
                                 aves umip waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm serialize flush_l1d arch_capabilities

[pontus]

Is this running as a server and you log in with ssh or is it running with WSL?

I can't describe how to check cgroup-config right now but that's my primary guess. In case you have other recent Java versions it might also be interesting to see if they say the same (but not worth doing extra work for).

[pontus]

(If you don't care about the actual usage of those cores, all nf-core pipelines should support limiting cpus - https://nf-co.re/usage/configuration#max-resources.)

[wdecoster]

I log in with SSH. I don't think cgroups is installed.

$ cgroups
Command 'cgroups' not found, did you mean:
  command 'groups' from deb coreutils (8.32-4.1ubuntu1)

[pditommaso]

Can you please include the .nextflow.log file?

[wdecoster]

Oh definitely, I wasn't aware that file exists :)

For the record, I'm quite new to nextflow, so all of this could be a massive PEBCAK.
nextflow.log

[pditommaso]

The log reports 80 CPUs avail, see

  Version: 22.10.6 build 5843
  Created: 23-01-2023 23:20 UTC 
  System: Linux 5.15.0-56-generic
  Runtime: Groovy 3.0.13 on OpenJDK 64-Bit Server VM 17.0.3-internal+0-adhoc..src
  Encoding: UTF-8 (UTF-8)
  Process: 3854850@megalon [127.0.1.1]
  CPUs: 80 - Mem: 502.6 GB (4.3 GB) - Swap: 8 GB (0)

however later it says the local execution can use max 4 cpus

Feb-24 15:33:15.247 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=4; memory=8 GB; capacity=80; pollInterval=100ms; dumpInterval=5m

I believe, that value has been specified in the pipeline config file. Look for

executor.cpus = 4 

See also https://www.nextflow.io/docs/latest/config.html#scope-executor

[wdecoster]

That seems to be the case indeed: https://github.com/epi2me-labs/wf-single-cell/blob/master/nextflow.config#L84

M apologies if all of this is readily available in the documentation... Since I did not develop this workflow, is there a way for me to fix this?

[pditommaso]

You should ask @cjw85 why they choose that value. In any case you should be able to override it but creating a file nextflow.config in the launching directory adding the value

executor.$local.cpus = 40 // or any other value of your choice

[cjw85]

We set these lowly defaults according to a principle of least surprise, such that a workflow doesn't use all resources on a computer (just as you wouldn't want a simple command line program to use all possible resource).