Skip to content

microSALT v.5.0.0#225

Open
karlnyr wants to merge 149 commits intomasterfrom
rc5.0.0
Open

microSALT v.5.0.0#225
karlnyr wants to merge 149 commits intomasterfrom
rc5.0.0

Conversation

@karlnyr
Copy link
Contributor

@karlnyr karlnyr commented Feb 23, 2026

Description

closes #241

This is a major update to microSALT doing the following:

  • Swap to jinja rendering
  • Remove Flask dependency
  • uv as a package manager
  • Transfer project into python 3.12
  • Refreshment of test module
  • Refactored database management

Primary function of PR

  • Hot-fix
  • Patch
  • Minor functionality improvement
  • New type of analysis
  • Backward-breaking functionality improvement
  • This change requires internal documents to be updated
  • This change requires another repository to be updated

Testing

  • bash /home/proj/production/servers/resources/hasta.scilifelab.se/install-microsalt-stage.sh BRANCHNAME
  • us
  • conda activate S_microSALT
  • microSALT analyse --input /path/to/fastq/ SAMPLEINFO_FILE

Test results

These are the results of the tests, and necessary conclusions, that prove the stability of the PR.

Sign-offs

  • Approved to run at Clinical-Genomics by @karlnyr or @Clinical-Genomics/micro

@karlnyr karlnyr requested a review from a team as a code owner February 23, 2026 12:18
karlnyr and others added 25 commits February 24, 2026 09:04
- Add Dockerfiles for skesa, blast, bwa, samtools, picard, trimmomatic, quast
- Wrap all tool invocations in job_creator.py with _singularity_exec()
- Wrap makeblastdb, bwa index, samtools faidx in referencer.py with _singularity_exec()
- Add singularity and containers config sections to configExample.json
- Update test_config.py to expect singularity/containers config keys
- Slim environment.yml to python-only (tools now run via Singularity containers)

Co-Authored-By: Claude <noreply@anthropic.com>
- build_containers_stage.yml: manual workflow_dispatch trigger, builds all 7
  tool images and pushes as '-stage' tagged images to Docker Hub
- build_containers_release.yml: triggers on release creation, builds all 7
  tool images and pushes production images tagged with release version

Both workflows use a matrix strategy across tools (blast, bwa, picard, quast,
samtools, skesa, trimmomatic) and build for linux/amd64.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
- build_containers_stage.yml: manual workflow_dispatch trigger, builds all 7
  tool images and pushes as '-stage' tagged images to Docker Hub
- build_containers_release.yml: triggers on release creation, builds all 7
  tool images and pushes production images tagged with release version

Both workflows use a matrix strategy across tools (blast, bwa, picard, quast,
samtools, skesa, trimmomatic) and build for linux/amd64.

Co-Authored-By: Claude <noreply@anthropic.com>
…NIFEST.in

- Move artwork/ and unique_references/ into microSALT/ package directory
- Replace $CONDA_PREFIX/get_path expec path lookup with importlib.resources
- Declare artwork/* and unique_references/* in pyproject.toml package-data
- Update .gitignore BLAST index pattern to new path
- Remove setup.py (data_files no longer needed) and MANIFEST.in

Co-Authored-By: Claude <noreply@anthropic.com>
@karlnyr
Copy link
Contributor Author

karlnyr commented Mar 12, 2026

testing get-bigsdb-credentials:

# PUBMLST
[10:14] [user@server:~/] [S_microSALT_v5]  (rc5.0.0) $ microsalt utils get-bigsdb-credentials pubmlst
Please log in using your user account at https://pubmlst.org/bigsdb?db=pubmlst_test_seqdef&page=authorizeClient&oauth_token=<SECRET> using a web browser to obtain a verification code.
Please enter verification code: <VERIFICATION_CODE>

Access Token: <ACCESS_TOKEN>
Access Token Secret: <ACCESS?TOKEN_SECRET>
Tokens saved to credentials/pubmlst_credentials.env
# Pasteur
[10:17] [user@server:~/] [S_microSALT_v5]  (rc5.0.0) $ microsalt utils get-bigsdb-credentials --species ecoli pasteur
Please log in using your user account at https://bigsdb.pasteur.fr/cgi-bin/bigsdb/bigsdb.pl?db=pubmlst_ecoli_seqdef&page=authorizeClient&oauth_token=<SECRET> using a web browser to obtain a verification code.
Please enter verification code: <VERIFICATION_CODE>

Access Token: <ACCESS_TOKEN>
Access Token Secret: <ACCESS_TOKEN_SECRET>
Tokens saved to credentials/pasteur_credentials.env

@karlnyr
Copy link
Contributor Author

karlnyr commented Mar 12, 2026

testing ref update:

# running update command
microsalt utils resync update-refs

This generates the lock:

image

Now, we seem to have an issue with the container for making the blastdb:

/.singularity.d/actions/exec: 9: exec: makeblastdb: not found
/.singularity.d/actions/exec: 9: exec: makeblastdb: not found
/.singularity.d/actions/exec: 9: exec: makeblastdb: not found
/.singularity.d/actions/exec: 9: exec: makeblastdb: not found
/.singularity.d/actions/exec: 9: exec: makeblastdb: not found
/.singularity.d/actions/exec: 9: exec: makeblastdb: not found
/.singularity.d/actions/exec: 9: exec: makeblastdb: not found

But disregarding that the database is filled with the ST profiles, and once that was finished the lock is released:

2026-03-12 10:32:05 - main_logger - INFO - Reference update lock released
image

Actions

  • Check the container used, and if it is the blast container, check the presence of makeblastdb.

karlnyr and others added 26 commits March 12, 2026 10:57
- query_rec       -> read_records
- top_index       -> read_top_index
- get_columns     -> read_columns
- exists          -> read_exists
- get_version     -> read_version
- get_report      -> read_report
- list_unresolved -> read_unresolved
- alleles2st      -> read_st
- bestST          -> read_best_st
- bestAlleles     -> read_best_alleles
- get_unique_alleles -> read_unique_alleles

All call sites updated across utils/, cli.py, and tests/.
- setPredictor -> set_predictor
- sync_novel   -> set_novel_st
- rm_novel     -> set_novel_ignored
- set_report   already correct

upd_rec kept as generic updater for profile table operations.
All call sites updated.
- purge_rec -> delete_records

All call sites updated.
Add typed per-model methods for ORM tables:
- _add_orm_record(): shared private helper (replaces ORM path of add_rec)
- add_sample/project/seq_type/resistance/expac/report/collection/version()
- update_sample() — includes CG_ID_sample cascade to child tables
- update_project() / update_version()
- delete_sample() / delete_project() / delete_collection()

Old generic methods (add_rec ORM path, upd_rec, delete_records) still
present; callers migrated in next commit.
…llers of add_rec/upd_rec/delete_records now use the typed per-model\nequivalents (add_sample, add_version, add_report, update_sample,\ndelete_sample, delete_project, delete_collection). The only remaining\nadd_rec call is the ProfileTable (non-ORM) path in set_predictor.
…le only\n\n- Strip ORM path from add_rec; it now only handles non-ORM Table objects (ProfileTable / novel tables)\n- Remove upd_rec entirely (no remaining callers)\n- Remove delete_records entirely (no remaining callers)\n- Update tests: replace remaining upd_rec/delete_records/add_rec-with-string calls with typed per-model methods; drop test_add_rec_unknown_table which tested removed ORM fallback behaviour
…ion helpers\n\n- add_sample/project/seq_type/resistance/expac/report/collection/version(**kwargs)\n are now pure factory functions that construct and return the ORM object;\n callers are responsible for add_to_session() and commit_session()\n- add_to_session(obj): coerces string DateTime fields, skips silently if PK\n already exists, then stages the object\n- commit_session(): thin wrapper around session.commit()\n- Remove _add_orm_record helper\n- Update all callers (scraper, job_creator, conftest, tests) to\n factory(**kwargs) -> add_to_session -> commit_session pattern
- Introduced JobCreationError for better error management in job creation processes.
- Removed unused date_ordered field from Projects model.
- Cleaned up and organized imports across multiple files.
- Updated sampleinfo_projects.json to remove date_ordered entries.
- Refactored methods in Job_Creator to raise JobCreationError when sample information is missing.
- Improved type hints and code readability in various functions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants