modularise models

[KatrionaGoldmann]

This will be useful for troubleshooting and part re-runing analysis, to avoid unnecessary reruns.

[KatrionaGoldmann]

My plan is to output a csv after each inference. Subsequent models will then use the csv from a previous iteration as input.

Each output csv can include additional columns for troubleshooting such as:

• model path
• chunk id
• notes?
• time taken for inference
• device used (CPU, GPU..)

[KatrionaGoldmann]

Rough idea: each step would use the output csv from the previous step as an input. The final endpoint would combine the outputs from all csvs, and should be very similar to the current output, alongside a reference json which describes all models.

[DylanCarbone]

Hi Kat!

Please let me know if you would prefer this to be a separate issue, but it would be great to think about the streamlining of inputs as part of this modularisation process.

Recently, I generated classification results for Jenna from 7 UK deployments. To do this I created a long txt file like this.

# Nettlebed

python 02_generate_keys.py --bucket 'gbr' --deployment_id 'dep000082' --output_file './keys/dep000082_keys.txt'

python 03_pre_chop_files.py --input_file './keys/dep000082_keys.txt' --file_extensions 'jpg' 'jpeg' --chunk_size 100 --output_file './keys/dep000082_workload_chunks.json'

sbatch Nettlebed_analysis.sh

# parsonage 

python 02_generate_keys.py --bucket 'gbr' --deployment_id 'dep000059' --output_file './keys/dep000059_keys.txt'

python 03_pre_chop_files.py --input_file './keys/dep000059_keys.txt' --file_extensions 'jpg' 'jpeg' --chunk_size 100 --output_file './keys/dep000059_workload_chunks.json'

sbatch parsonage_analysis.sh

...

I then created the separate shell files for each deployment (e.g., Nettlebed_analysis.sh). I wonder if this process can be streamlined so in future, we aren't juggling lots of shell files, and we can generate keys and chunks inside the shell script. It would be amazing if we could specify the bucket, deployment, log name, and algorithm paths all in on section, then run a single shell file that would take all these inputs. Please let me know if you would like me to look into this for you, as it would be great to get more involved with the classification workflows. Shall we talk about this more in our call? :)

[DylanCarbone]

Also, we'd need a way of saving the crops/segments. We only have 100 GB of user space on JASMIN, so perhaps we should consider applying for a group workspace

[KatrionaGoldmann]

Hi Kat!

Please let me know if you would prefer this to be a separate issue, but it would be great to think about the streamlining of inputs as part of this modularisation process.

Recently, I generated classification results for Jenna from 7 UK deployments. To do this I created a long txt file like this.

# Nettlebed

python 02_generate_keys.py --bucket 'gbr' --deployment_id 'dep000082' --output_file './keys/dep000082_keys.txt'

python 03_pre_chop_files.py --input_file './keys/dep000082_keys.txt' --file_extensions 'jpg' 'jpeg' --chunk_size 100 --output_file './keys/dep000082_workload_chunks.json'

sbatch Nettlebed_analysis.sh

# parsonage 

python 02_generate_keys.py --bucket 'gbr' --deployment_id 'dep000059' --output_file './keys/dep000059_keys.txt'

python 03_pre_chop_files.py --input_file './keys/dep000059_keys.txt' --file_extensions 'jpg' 'jpeg' --chunk_size 100 --output_file './keys/dep000059_workload_chunks.json'

sbatch parsonage_analysis.sh

...

I then created the separate shell files for each deployment (e.g., Nettlebed_analysis.sh). I wonder if this process can be streamlined so in future, we aren't juggling lots of shell files, and we can generate keys and chunks inside the shell script. It would be amazing if we could specify the bucket, deployment, log name, and algorithm paths all in on section, then run a single shell file that would take all these inputs. Please let me know if you would like me to look into this for you, as it would be great to get more involved with the classification workflows. Shall we talk about this more in our call? :)

I had something similar for the harlequin analysis. As a workaround, I simply ran a loop from within the sh file through the deployment ids, this may work in your case too. But lets discuss, there may be solutions which suit your needs more.

[KatrionaGoldmann]

Following discussion last week, we would like to save if amber inferences in a shared location. The output file structure would look something like:

./data/amber-outputs
├── dep000001
│   ├── 01_localisation
│   │   ├── a.csv
│   │   ├── a.json
│   │   ├── b.csv
│   │   ├── b.json
│   │   ├── c.csv
│   │   └── c.json
│   ├── 02_order_binary
│   │   ├── d.csv
│   │   ├── d.json
│   │   ├── e.csv
│   │   └── e.json
│   └── 03_species
│       ├── f.csv
│       ├── f.json
│       ├── h.csv
│       ├── h.json
│       ├── i.csv
│       └── i.json
├── dep000002
│   └── …
├── all_jobs.csv
└── all_models.json

Job name csv (e.g. {a,b,c}.csv) contains inferences
Job name json (e.g. {a,b,c}.json) contains model information for that specific set of jobs
all_models.json contains information about all the models
all_jobs.csv acts as a log and contains summary information about all the jobs (models used, dates, datasets etc.)

@DylanCarbone, does this look sensible to you?

[DylanCarbone]

Hi Kat,

This looks great! Would it be possible to have a folder between the deployment and the classification stage with the run date and jobname, appended together such as 2025_01_21_job_956337273 or similar? This way we could separate the results in each deployment by model run.