ERROR on local (real) data file

[CodyCBakerPhD]

Expanding beyond the attempts to use the sample NWB data, I have some real data I tried to run this pipeline on...

I encounter the following error early in the process

 N E X T F L O W   ~  version 25.04.8

Launching `main_multi_backend.nf` [soggy_cajal] DSL2 - revision: 780c3e867c

DATA_PATH: /mnt/e/datalad/OdorSequence/code/aind-ephys-pipeline/pipeline/../../../sourcedata/nwb-raw/sub-M541/ses-2024-08-31/
RESULTS_PATH: /mnt/e/datalad/OdorSequence/code/aind-ephys-pipeline/pipeline/../../../derived/results
PARAMS: [n_jobs:8, executor:local, ecephys_path:/mnt/e/datalad/OdorSequence/code/aind-ephys-pipeline/pipeline/../../../sourcedata/nwb-raw/sub-M541/ses-2024-08-31/, params_file:null]
CONTAINER TAG: si-0.103.0
N JOBS: 8
Using RUNMODE: full
No parameters file provided, using command line arguments.
No sorter specified, defaulting to kilosort4
Using SORTER: kilosort4 with args:
executor >  local (1)
[65/b57e82] job_dispatch (job-dispatch) | 0 of 1
[-        ] preprocessing               -
[-        ] spikesort_kilosort4         -
[-        ] postprocessing              -
[-        ] curation                    -
[-        ] visualization               -
[-        ] results_collector           -
[-        ] quality_control             -
[-        ] quality_control_collector   -
[-        ] nwb_ecephys                 -
[-        ] nwb_units                   -
ERROR ~ Error executing process > 'job_dispatch (job-dispatch)'

Caused by:
  Process `job_dispatch (job-dispatch)` terminated with an error exit status (128)


Command executed:

  #!/usr/bin/env bash
  set -e

  mkdir -p capsule
  mkdir -p capsule/data
  mkdir -p capsule/results
  mkdir -p capsule/scratch

  if [[ local == "slurm" ]]; then
      echo "[job-dispatch] allocated task time: null"
  fi

  TASK_DIR=$(pwd)

  echo "[job-dispatch] cloning git repo..."
  git clone "https://github.com/AllenNeuralDynamics/aind-ephys-job-dispatch.git" capsule-repo
  git -C capsule-repo -c core.fileMode=false checkout 82103472153da7ca25a36f48173e52826825e8cf  --quiet
executor >  local (1)
[65/b57e82] job_dispatch (job-dispatch) | 0 of 1 ✘
[-        ] preprocessing               -
[-        ] spikesort_kilosort4         -
[-        ] postprocessing              -
[-        ] curation                    -
[-        ] visualization               -
[-        ] results_collector           -
[-        ] quality_control             -
[-        ] quality_control_collector   -
[-        ] nwb_ecephys                 -
[-        ] nwb_units                   -
ERROR ~ Error executing process > 'job_dispatch (job-dispatch)'

Caused by:
  Process `job_dispatch (job-dispatch)` terminated with an error exit status (128)


Command executed:

  #!/usr/bin/env bash
  set -e

  mkdir -p capsule
  mkdir -p capsule/data
  mkdir -p capsule/results
  mkdir -p capsule/scratch

  if [[ local == "slurm" ]]; then
      echo "[job-dispatch] allocated task time: null"
  fi

  TASK_DIR=$(pwd)

  echo "[job-dispatch] cloning git repo..."
  git clone "https://github.com/AllenNeuralDynamics/aind-ephys-job-dispatch.git" capsule-repo
  git -C capsule-repo -c core.fileMode=false checkout 82103472153da7ca25a36f48173e52826825e8cf  --quiet
  mv capsule-repo/code capsule/code
  rm -rf capsule-repo

  echo "[job-dispatch] running capsule..."
  cd capsule/code
  chmod +x run
  ./run

  MAX_DURATION_MIN=$(python get_max_recording_duration_min.py)

  cd $TASK_DIR
  echo "$MAX_DURATION_MIN" > max_duration.txt

  echo "[job-dispatch] completed!"

Command exit status:
  128

Command output:
  [job-dispatch] cloning git repo...

Command error:
  WARNING: Your kernel does not support memory swappiness capabilities or the cgroup is not mounted. Memory swappiness discarded.
  [job-dispatch] cloning git repo...
  Cloning into 'capsule-repo'...
  fatal: detected dubious ownership in repository at 'capsule-repo'
  To add an exception for this directory, call:

        git config --global --add safe.directory capsule-repo

Work dir:
  /mnt/e/datalad/OdorSequence/code/aind-ephys-pipeline/pipeline/work/65/b57e82b8dcc3bb527aab6ff0acebb8

Container:
  ghcr.io/allenneuraldynamics/aind-ephys-pipeline-base:si-0.103.0

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

 -- Check '/mnt/e/datalad/OdorSequence/code/aind-ephys-pipeline/pipeline/../../../derived/results/nextflow/nextflow.log' file for details

Normally I would set the git config to declare the repo as safe but this occurs automatically in the pipeline which is in a new work directory on every run. I also tried the local deployment through my own fork of this repo, but I suppose the things it is cloning do not know to adjust to my own forks of things?

Will continue to try to hack away to get this to work

cc @yarikoptic

[alejoe91]

Thanks @CodyCBakerPhD

Maybe this is due to some system-wise stricter rules for git clone...

Normally I would set the git config to declare the repo as safe but this occurs automatically in the pipeline which is in a new work directory on every run. I also tried the local deployment through my own fork of this repo, but I suppose the things it is cloning do not know to adjust to my own forks of things?

What do you mean with this?

Also note that as in #60 you need to tell the pipeline to look for NWB files: --job_dispatch_args "--input nwb"

[CodyCBakerPhD]

What do you mean with this?

I mean following the error's suggestion of flagging the repo as safe (capsule-repo) but that location appears to target a randomly created work/ dir at runtime

I tried all manner of *-like blanket permissions but whatever config the git in the workflow is using, it is not recognizing my efforts

Also note that as in #60 you need to tell the pipeline to look for NWB files: --job_dispatch_args "--input nwb"

I have adjusted this going forward but currently results in the same issue

Perhaps we can discuss tomorrow how/if the workflow can be refactored to perform the required git clone before/outside of the rest of the steps; or if the entire pipeline might be composed into a single container (albeit with one with many layers/steps)

[yarikoptic]

@CodyCBakerPhD could you please share original full invocation so I could try reproducing on typhon?

[CodyCBakerPhD]

DATA_PATH=$DATA_PATH RESULTS_PATH=$RESULTS_PATH nextflow \
    -C $CONFIG_FILE \
    -log $RESULTS_PATH/nextflow/nextflow.log \
    run $PIPELINE_PATH/pipeline/main_multi_backend.nf \
    --work-dir $WORKDIR \
    --job_dispatch_args "--input nwb"

(-resume was removed to force it to run fresh)