Module file size cannot be bigger than 1 MB #4298

SwastikLGowda · 2023-09-12T10:57:02Z

SwastikLGowda
Sep 12, 2023

I ran a computation-heavy pipeline which is causing this issue - Module file size cannot be bigger than 1 MB - offending file: /Users/elucidata/Documents/Github_projects/connector_pipelines/pipelines/rnaseq/bin/mmusculus_V107.txt , any help to solve this issue is appreciated, I have attached the logs as well -

Sept-12 13:26:46.056 [main] DEBUG nextflow.cli.Launcher - $> nextflow run -c ./nextflow.config ./ -work-dir 's3://bucket-102/' -resume -with-trace --outDir 's3://bucket-102/' -with-report 's3://bucket-102/report.html' -with-timeline 's3://bucket-102/timeline.html' -params-file ./params.json --ingestion_job_id <ingestion_job_id> --job_id <job_id> --run_id <run_id> --curationtoken <curation_token> --repo testing_ws_rnaseq_conn --token <token> --datalake_s3 's3://bucket-102/' --OPENAI_API_KEY <open_api_key> --dataset_id GSE76656_GPL11154 --check_data_type Yes -profile dev
Sept-12 13:26:46.099 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 23.04.3
Sept-12 13:26:46.110 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/Users/elucidata/.nextflow/plugins; core-plugins: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
Sept-12 13:26:46.116 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Sept-12 13:26:46.116 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Sept-12 13:26:46.117 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Sept-12 13:26:46.124 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Sept-12 13:26:46.458 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /Users/elucidata/Documents/Github_projects/connector_pipelines/pipelines/rnaseq/nextflow.config
Sept-12 13:26:46.459 [main] DEBUG nextflow.config.ConfigBuilder - User config file: /Users/elucidata/Documents/Github_projects/connector_pipelines/pipelines/rnaseq/./nextflow.config
Sept-12 13:26:46.459 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /Users/elucidata/Documents/Github_projects/connector_pipelines/pipelines/rnaseq/nextflow.config
Sept-12 13:26:46.459 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /Users/elucidata/Documents/Github_projects/connector_pipelines/pipelines/rnaseq/nextflow.config
Sept-12 13:26:46.471 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `dev`
Sept-12 13:26:46.506 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `dev`
Sept-12 13:26:46.511 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [prod, test, dev]
Sept-12 13:26:46.525 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declararion
Sept-12 13:26:46.536 [main] INFO  nextflow.cli.CmdRun - Launching `./main.nf` [suspicious_cori] DSL2 - revision: de3c1a797e
Sept-12 13:26:46.537 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins declared=[[email protected]]
Sept-12 13:26:46.537 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[[email protected]]
Sept-12 13:26:46.538 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[[email protected], [email protected]]
Sept-12 13:26:46.538 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-amazon version: 1.16.2
Sept-12 13:26:46.544 [main] INFO  org.pf4j.AbstractPluginManager - Plugin '[email protected]' resolved
Sept-12 13:26:46.545 [main] INFO  org.pf4j.AbstractPluginManager - Start plugin '[email protected]'
Sept-12 13:26:46.570 [main] DEBUG nextflow.plugin.BasePlugin - Plugin started [email protected]
Sept-12 13:26:46.583 [main] DEBUG nextflow.file.FileHelper - > Added 'S3FileSystemProvider' to list of installed providers [s3]
Sept-12 13:26:46.583 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-wave version: 0.8.4
Sept-12 13:26:46.585 [main] INFO  org.pf4j.AbstractPluginManager - Plugin '[email protected]' resolved
Sept-12 13:26:46.585 [main] INFO  org.pf4j.AbstractPluginManager - Start plugin '[email protected]'
Sept-12 13:26:46.591 [main] DEBUG nextflow.plugin.BasePlugin - Plugin started [email protected]
Sept-12 13:26:46.600 [main] DEBUG nextflow.secret.LocalSecretsProvider - Secrets store: /Users/elucidata/.nextflow/secrets/store.json
Sept-12 13:26:46.602 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@330c1f61] - activable => nextflow.secret.LocalSecretsProvider@330c1f61
Sept-12 13:26:46.638 [main] DEBUG nextflow.Session - Session UUID: 17aeea5a-61a6-48b5-8063-70d7766e3235
Sept-12 13:26:46.638 [main] DEBUG nextflow.Session - Run name: suspicious_cori
Sept-12 13:26:46.638 [main] DEBUG nextflow.Session - Executor pool size: 8
Sept-12 13:26:46.644 [main] DEBUG nextflow.file.FileHelper - Creating a file system instance for provider: S3FileSystemProvider
Sept-12 13:26:46.647 [main] DEBUG nextflow.Global - Using AWS credentials defined by environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
Sept-12 13:26:46.650 [main] DEBUG nextflow.file.FileHelper - AWS S3 config details: {secret_key=...., endpoint=https://s3.us-east-1.wasabisys.com, s_3_path_style_access=true, region=us-east-1, max_error_retry=5, access_key=.....}
Sept-12 13:26:46.894 [main] DEBUG com.upplication.s3fs.AmazonS3Client - Setting S3 glacierRetrievalTier=null
Sept-12 13:26:46.939 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=24; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Sept-12 13:26:46.946 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 23.04.3 build 5875
  Created: 11-08-2023 18:37 UTC (12-08-2023 00:07 IDT)
  System: Mac OS X 13.5.2
  Runtime: Groovy 3.0.16 on Java HotSpot(TM) 64-Bit Server VM 20.0.2+9-78
  Encoding: UTF-8 (UTF-8)
  Process: [email protected] [192.168.29.247]
  CPUs: 8 - Mem: 16 GB (70.3 MB) - Swap: 6 GB (1.2 GB)
Sept-12 13:26:48.027 [main] DEBUG nextflow.Session - Work-dir: s3://bucket-102/ [Mac OS X]
Sept-12 13:26:48.053 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[AwsBatchExecutor]
Sept-12 13:26:48.060 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Sept-12 13:26:48.071 [main] DEBUG nextflow.Session - Observer factory: WaveFactory
Sept-12 13:26:48.072 [main] DEBUG io.seqera.wave.plugin.WaveFactory - Detected Fusion enabled -- Enabling bundle project resources -- Disabling upload of remote bin directory
Sept-12 13:26:48.088 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Sept-12 13:26:48.095 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 9; maxThreads: 1000
Sept-12 13:26:48.128 [main] DEBUG nextflow.Session - Session start
Sept-12 13:26:48.133 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /Users/elucidata/Documents/Github_projects/connector_pipelines/pipelines/rnaseq/trace.txt
Sept-12 13:26:48.349 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Sept-12 13:26:48.729 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: k8s
Sept-12 13:26:48.729 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'k8s'
Sept-12 13:26:48.733 [main] DEBUG nextflow.executor.Executor - [warm up] executor > k8s (fusion enabled)
Sept-12 13:26:48.737 [main] DEBUG n.processor.TaskPollingMonitor - Creating task monitor for executor 'k8s' > capacity: 100; pollInterval: 5s; dumpInterval: 5m 
Sept-12 13:26:48.780 [main] DEBUG nextflow.k8s.K8sExecutor - [K8s] config=[namespace:default, context:phoenixnap-poc-rancher-workload-cluster]; API client config=ClientConfig[ server=https://131.153.201.213.sslip.io/k8s/clusters/c-wdfsd, namespace=default, serviceAccount=default, token=kubeconfig-.., sslCert=LS0tLS1CRUd.., clientCert=-, clientKey=-, verifySsl=true, fromFile=false, httpReadTimeout=null, httpConnectTimeout=null, maxErrorRetry=4 ]
Sept-12 13:26:48.808 [main] DEBUG nextflow.processor.TaskProcessor - Creating *combiner* operator for each param(s) at index(es): [0]
Sept-12 13:26:48.830 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: k8s
Sept-12 13:26:48.830 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'k8s'
Sept-12 13:26:48.831 [main] DEBUG nextflow.processor.TaskProcessor - Creating *combiner* operator for each param(s) at index(es): [0, 1]
Sept-12 13:26:48.846 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: k8s
Sept-12 13:26:48.846 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'k8s'
Sept-12 13:26:48.849 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: k8s
Sept-12 13:26:48.849 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'k8s'
Sept-12 13:26:48.854 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: k8s
Sept-12 13:26:48.854 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'k8s'
Sept-12 13:26:48.859 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: k8s
Sept-12 13:26:48.859 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'k8s'
Sept-12 13:26:48.861 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: k8s
Sept-12 13:26:48.862 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'k8s'
Sept-12 13:26:48.863 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: k8s
Sept-12 13:26:48.863 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'k8s'
Sept-12 13:26:48.865 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: k8s
Sept-12 13:26:48.865 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'k8s'
Sept-12 13:26:48.867 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: k8s
Sept-12 13:26:48.867 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'k8s'
Sept-12 13:26:48.875 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: k8s
Sept-12 13:26:48.875 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'k8s'
Sept-12 13:26:48.883 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: k8s
Sept-12 13:26:48.884 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'k8s'
Sept-12 13:26:48.888 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: k8s
Sept-12 13:26:48.888 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'k8s'
Sept-12 13:26:48.891 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: k8s
Sept-12 13:26:48.891 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'k8s'
Sept-12 13:26:48.892 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: k8s
Sept-12 13:26:48.892 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'k8s'
Sept-12 13:26:48.894 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: k8s
Sept-12 13:26:48.894 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'k8s'
Sept-12 13:26:48.896 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: indexes:build_kallisto, metadata:fetch_SRA, generate_datasets, skewer_quality_trimming, clean_fastq_files, curate, processing:fastqc_reports, validate_sra, fastqc_reports, parallel_fastq_dump, processing:clean_fastq_files, build_kallisto, send_analytics, multiqc, processing:multiqc, processing:kallisto_quant, processing:skewer_quality_trimming, collecting:make_gct, processing:parallel_fastq_dump, analytics:send_analytics, make_gct, push:push_to_OA, clean_work_files, fastqc, kallisto_quant, fastq_dump, processing:validate_sra, processing:fastqc, metadata:generate_datasets, push_to_OA, curation:curate, fetch_SRA, filter_urls, processing:fastq_dump
Sept-12 13:26:48.897 [main] DEBUG nextflow.Session - Igniting dataflow network (18)
Sept-12 13:26:48.897 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > curation:curate
Sept-12 13:26:48.897 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > curation:curate
Sept-12 13:26:48.897 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > metadata:generate_datasets
Sept-12 13:26:48.897 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > metadata:generate_datasets
Sept-12 13:26:48.897 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > metadata:fetch_SRA
Sept-12 13:26:48.897 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > indexes:build_kallisto
Sept-12 13:26:48.897 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > processing:validate_sra
Sept-12 13:26:48.897 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > processing:fastq_dump
Sept-12 13:26:48.897 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > processing:fastqc
Sept-12 13:26:48.897 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > processing:parallel_fastq_dump
Sept-12 13:26:48.897 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > processing:skewer_quality_trimming
Sept-12 13:26:48.898 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > processing:fastqc_reports
Sept-12 13:26:48.899 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > processing:clean_fastq_files
Sept-12 13:26:48.899 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > processing:kallisto_quant
Sept-12 13:26:48.899 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > processing:multiqc
Sept-12 13:26:48.899 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > collecting:make_gct
Sept-12 13:26:48.899 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > push:push_to_OA
Sept-12 13:26:48.899 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > analytics:send_analytics
Sept-12 13:26:48.899 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination 
Sept-12 13:26:48.899 [main] DEBUG nextflow.Session - Session await
Sept-12 13:26:48.948 [Actor Thread 1] DEBUG i.s.wave.plugin.config.WaveConfig - Wave strategy not specified - using default: [container, dockerfile, conda]
Sept-12 13:26:48.951 [Actor Thread 1] DEBUG io.seqera.wave.plugin.WaveClient - Wave server endpoint: https://wave.seqera.io
Sept-12 13:26:48.971 [Actor Thread 1] DEBUG io.seqera.wave.plugin.WaveClient - Wave assets bundle bin resources: /Users/elucidata/Documents/Github_projects/connector_pipelines/pipelines/rnaseq/bin
Sept-12 13:26:48.982 [Actor Thread 1] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=metadata:generate_datasets (1); work-dir=null
  error [java.lang.IllegalArgumentException]: Module file size cannot be bigger than 1 MB - offending file: /Users/elucidata/Documents/Github_projects/connector_pipelines/pipelines/rnaseq/bin/mmusculus_V107.txt
Sept-12 13:26:48.988 [Actor Thread 1] ERROR nextflow.processor.TaskProcessor - Error executing process > 'metadata:generate_datasets (1)'

Caused by:
  Module file size cannot be bigger than 1 MB - offending file: /Users/elucidata/Documents/Github_projects/connector_pipelines/pipelines/rnaseq/bin/mmusculus_V107.txt

java.lang.IllegalArgumentException: Module file size cannot be bigger than 1 MB - offending file: /Users/elucidata/Documents/Github_projects/connector_pipelines/pipelines/rnaseq/bin/mmusculus_V107.txt
	at nextflow.script.bundle.ResourcesBundle.withPaths(ResourcesBundle.groovy:81)
	at nextflow.script.bundle.ResourcesBundle.scan(ResourcesBundle.groovy:161)
	at io.seqera.wave.plugin.WaveClient.memoizedMethodPriv$projectResourcesPath(WaveClient.groovy:387)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1258)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
	at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1036)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1019)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:97)
	at io.seqera.wave.plugin.WaveClient$_closure3.doCall(WaveClient.groovy)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
	at groovy.lang.Closure.call(Closure.java:412)
	at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.lambda$call$0(Memoize.java:137)
	at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:137)
	at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:113)
	at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.call(Memoize.java:136)
	at groovy.lang.Closure.call(Closure.java:428)
	at io.seqera.wave.plugin.WaveClient.projectResources(WaveClient.groovy)
	at io.seqera.wave.plugin.WaveClient.resolveAssets0(WaveClient.groovy:361)
	at io.seqera.wave.plugin.WaveClient.memoizedMethodPriv$resolveAssetsTaskRunString(WaveClient.groovy:321)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1258)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
	at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1036)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1019)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:97)
	at io.seqera.wave.plugin.WaveClient$_closure2.doCall(WaveClient.groovy)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
	at groovy.lang.Closure.call(Closure.java:412)
	at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.lambda$call$0(Memoize.java:137)
	at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:137)
	at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:113)
	at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.call(Memoize.java:136)
	at io.seqera.wave.plugin.WaveClient.resolveAssets(WaveClient.groovy)
	at io.seqera.wave.plugin.resolver.WaveContainerResolver.waveContainer(WaveContainerResolver.groovy:107)
	at io.seqera.wave.plugin.resolver.WaveContainerResolver.resolveImage(WaveContainerResolver.groovy:72)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:48)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:166)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:69)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:148)
	at nextflow.processor.TaskRun.containerInfo0(TaskRun.groovy:626)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:362)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:61)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:176)
	at nextflow.processor.TaskRun$_containerInfo_closure7.doCall(TaskRun.groovy:612)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
	at groovy.lang.Closure.call(Closure.java:412)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:50)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:112)
	at jdk.proxy1/jdk.proxy1.$Proxy35.apply(Unknown Source)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:48)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrap.invoke(PojoMetaMethodSite.java:198)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:51)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:148)
	at nextflow.processor.TaskRun.containerInfo(TaskRun.groovy:612)
	at nextflow.processor.TaskRun.getContainer(TaskRun.groovy:634)
	at nextflow.processor.TaskRun.isContainerEnabled(TaskRun.groovy:679)
	at nextflow.processor.TaskRun$isContainerEnabled$5.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:130)
	at nextflow.processor.TaskProcessor.createTaskHashKey(TaskProcessor.groovy:2070)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:48)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:189)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:57)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:185)
	at nextflow.processor.TaskProcessor.invokeTask(TaskProcessor.groovy:617)
	at nextflow.processor.InvokeTaskAdapter.call(InvokeTaskAdapter.groovy:52)
	at groovyx.gpars.dataflow.operator.DataflowOperatorActor.startTask(DataflowOperatorActor.java:120)
	at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor.access$001(ForkingDataflowOperatorActor.java:35)
	at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor$1.run(ForkingDataflowOperatorActor.java:58)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1623)
Sept-12 13:26:49.029 [Actor Thread 1] DEBUG nextflow.Session - Session aborted -- Cause: Module file size cannot be bigger than 1 MB - offending file: /Users/elucidata/Documents/Github_projects/connector_pipelines/pipelines/rnaseq/bin/mmusculus_V107.txt
Sept-12 13:26:49.036 [Actor Thread 1] DEBUG nextflow.Session - The following nodes are still active:
[process] curation:curate
  status=ACTIVE
  port 0: (value) bound ; channel: __$eachinparam<0>
  port 1: (value) bound ; channel: curate
  port 2: (value) bound ; channel: curation_resources
  port 3: (value) bound ; channel: POLLY_TOKEN
  port 4: (value) bound ; channel: OPENAI_API_KEY
  port 5: (cntrl) -     ; channel: $

[process] metadata:generate_datasets
  status=ACTIVE
  port 0: (value) bound ; channel: __$eachinparam<0>
  port 1: (value) bound ; channel: __$eachinparam<1>
  port 2: (cntrl) -     ; channel: $

[process] metadata:fetch_SRA
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: srr
  port 1: (cntrl) -     ; channel: $

[process] indexes:build_kallisto
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: organism
  port 1: (cntrl) -     ; channel: $

[process] processing:validate_sra
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: srr
  port 1: (cntrl) -     ; channel: $

[process] processing:fastq_dump
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] processing:fastqc
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (queue) OPEN  ; channel: RDS
  port 2: (queue) OPEN  ; channel: srr
  port 3: (cntrl) -     ; channel: $

[process] processing:parallel_fastq_dump
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: RDS
  port 1: (queue) OPEN  ; channel: SHORT_READ
  port 2: (queue) OPEN  ; channel: -
  port 3: (cntrl) -     ; channel: $

[process] processing:skewer_quality_trimming
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] processing:fastqc_reports
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] processing:clean_fastq_files
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: file
  port 1: (cntrl) -     ; channel: $

[process] processing:kallisto_quant
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: kallisto
  port 1: (queue) OPEN  ; channel: fastq
  port 2: (cntrl) -     ; channel: $

[process] processing:multiqc
  status=ACTIVE
  port 0: (value) OPEN  ; channel: fastqc_reports
  port 1: (value) OPEN  ; channel: skewer_logs
  port 2: (value) OPEN  ; channel: kallisto_logs
  port 3: (value) bound ; channel: POLLY_TOKEN
  port 4: (value) bound ; channel: DATASET_ID
  port 5: (value) bound ; channel: sync_file
  port 6: (cntrl) -     ; channel: $

[process] collecting:make_gct
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: metadata
  port 1: (value) OPEN  ; channel: counts
  port 2: (queue) OPEN  ; channel: organism
  port 3: (value) bound ; channel: file_path
  port 4: (cntrl) -     ; channel: $

[process] push:push_to_OA
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: dataset_metadata
  port 1: (queue) OPEN  ; channel: sample_metadata
  port 2: (queue) OPEN  ; channel: dataset
  port 3: (value) bound ; channel: repo_id
  port 4: (value) bound ; channel: dataset_id
  port 5: (value) bound ; channel: datalake_s3
  port 6: (value) bound ; channel: ingestion_job_id
  port 7: (queue) OPEN  ; channel: rejected_samples
  port 8: (cntrl) -     ; channel: $

[process] analytics:send_analytics
  status=ACTIVE
  port 0: (value) bound ; channel: repo_id
  port 1: (value) bound ; channel: dataset_id
  port 2: (queue) OPEN  ; channel: metadata
  port 3: (value) bound ; channel: aws_region
  port 4: (value) bound ; channel: ingestion_kinesis_stream_name
  port 5: (queue) OPEN  ; channel: gct_file
  port 6: (cntrl) -     ; channel: $

Sept-12 13:26:49.037 [main] DEBUG nextflow.Session - Session await > all processes finished
Sept-12 13:26:49.037 [main] DEBUG nextflow.Session - Session await > all barriers passed
Sept-12 13:26:49.038 [Actor Thread 6] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=curation:curate; work-dir=null
  error [java.lang.InterruptedException]: java.lang.InterruptedException
Sept-12 13:26:49.039 [main] INFO  nextflow.script.BaseScript - Pipeline completed at: 2023-09-12T13:26:49.038085+05:30
Sept-12 13:26:49.039 [main] INFO  nextflow.script.BaseScript - Execution status: failed
Sept-12 13:26:49.039 [Actor Thread 3] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=metadata:generate_datasets; work-dir=null
  error [java.lang.InterruptedException]: java.lang.InterruptedException
Sept-12 13:26:49.041 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=0ms; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=0; peakCpus=0; peakMemory=0; ]
Sept-12 13:26:49.041 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Sept-12 13:26:49.042 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Sept-12 13:26:49.043 [main] DEBUG nextflow.trace.ReportObserver - Execution report summary data:
  []
Sept-12 13:26:50.030 [FileTransfer-1] DEBUG nextflow.file.FilePorter - Local cache found for foreign file /Users/elucidata/Documents/Github_projects/connector_pipelines/pipelines/rnaseq/bin/curate.py at s3://bucket-102/stage-17aeea5a-61a6-48b5-8063-70d7766e3235/b4/b48ffba27146045fa44e13fed896e6/curate.py
Sept-12 13:26:50.031 [Actor Thread 2] DEBUG io.seqera.wave.plugin.WaveClient - Wave assets bundle bin resources: /Users/elucidata/Documents/Github_projects/connector_pipelines/pipelines/rnaseq/bin
Sept-12 13:26:50.036 [Actor Thread 2] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=curation:curate (1); work-dir=null
  error [java.lang.IllegalArgumentException]: Module file size cannot be bigger than 1 MB - offending file: /Users/elucidata/Documents/Github_projects/connector_pipelines/pipelines/rnaseq/bin/mmusculus_V107.txt
Sept-12 13:26:53.583 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Sept-12 13:26:54.708 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Sept-12 13:26:54.708 [main] INFO  org.pf4j.AbstractPluginManager - Stop plugin '[email protected]'
Sept-12 13:26:54.708 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-wave
Sept-12 13:26:54.708 [main] INFO  org.pf4j.AbstractPluginManager - Stop plugin '[email protected]'
Sept-12 13:26:54.708 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-amazon
Sept-12 13:26:54.733 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

pditommaso · 2023-09-12T12:19:08Z

pditommaso
Sep 12, 2023
Maintainer

The problem is that big files are not allowed in the module directory. It should be hosted somewhere (even in a different GitHub directory) and accessed as task input file

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Module file size cannot be bigger than 1 MB #4298

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Module file size cannot be bigger than 1 MB #4298

Uh oh!

SwastikLGowda Sep 12, 2023

Replies: 1 comment

Uh oh!

pditommaso Sep 12, 2023 Maintainer

SwastikLGowda
Sep 12, 2023

pditommaso
Sep 12, 2023
Maintainer