Skip to content

Upload artifacts for tags.#172

Open
williamrowell wants to merge 45 commits intodevelop-v3from
feature/upload-artifacts-on-tag
Open

Upload artifacts for tags.#172
williamrowell wants to merge 45 commits intodevelop-v3from
feature/upload-artifacts-on-tag

Conversation

@williamrowell
Copy link
Collaborator

Added new GitHub workflow:

  • miniwdl zip singleton workflow
  • miniwdl zip family workflow
  • upload zip artifacts

This should only run for releases with simple semantic versioning, like 'v[0-9]+.[0-9]+.[0-9]+'.

  • won't match alpha/prerelease/rc
  • won't match *-dockstore releases

@williamrowell williamrowell self-assigned this Nov 27, 2024
@williamrowell williamrowell marked this pull request as ready for review November 27, 2024 00:18
@williamrowell williamrowell added the enhancement New feature or request label Nov 27, 2024
@williamrowell williamrowell force-pushed the develop-v2 branch 4 times, most recently from 015dfc6 to ecdbf3e Compare December 10, 2024 02:28
@williamrowell williamrowell force-pushed the develop-v2 branch 3 times, most recently from de32f4d to 1877926 Compare February 10, 2025 21:57
williamrowell and others added 18 commits February 20, 2025 12:05
- remove the deepvariant_version option and hard-code the default into the subworkflow
- remove the custom_deepvariant_model_tar option
- update the version to 1.8.0
- modify task command calls to 1.8.0 options
- use generic tfrecord filenames

Remove the pharmcat_version option and provide the default directly within the subworkflow.
PEDigree file syntax is generated directly within WDL for each sample independently.  PED files are written within tasks by `write_tsv()` by concatenating sample lines.

`is_duo_kid` and `is_trio_kid` arrays are generated for convenience in preparation for duo-/trio-specific secondary/tertiary analyses.

Move phenotype lookup to tertiary subworkflow.

Adjust tests for changes to write_phrank.

Fix write_phrank test.

If sex cannot be inferred by mosdepth, output is empty string.  Change `pedigree_sex["UNKNOWN"]` to `pedigree_sex[""]`.
williamrowell and others added 23 commits February 25, 2025 11:36
Updated to StarPhase 1.3.1.

- accepts phased SV vcf as input
… where it can take advantage of the phased reads to report haplotype dropouts.
…-dropouts-downstream

Move coverage_dropouts to downstream.
…r than the reported sex.

Inferred sex is compared to the reported sex, and if there is a disagreement, a new `String qc_sex` message is emitted by upstream.  These messages are gathered at top level `Array[String?] qc_messages` and emitted as output.
…red-to-reported-sex

Compare inferred to reported sex for QC
…ate through the BAM once. (#198)

* Refactor tasks that extract stats from BAM files so that we only iterate through the BAM once.

- remove components from pbmm2_align taskthat gather stats about read length and quality
- remove task to merge per-bam stats and plot read length and quality
- remove components from hiphasse task that gather stats about mapping and alignment
- add a new bam_stats task that extracts all tasks in a single pass through the BAM and replaces the outputs from the removed components

* update wdl-ci config file after successful tests

---------

Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
…se memory/cpu attributes for some tasks based on typical available cloud node types.
…P zones that lack n1 nodes. (#207)

* Add the `cpuPlatform` runtime attribute to support the workflow in GCP zones that lack n1 nodes.

- add cpuPlatform to every task
- add cpuPlatform to backend_configuration
- add cpuPlatform input to family and singleton entrypoints
- document input in singleton and family docs
- document input in backend-gcp doc

* update wdl-ci config file after successful tests

* Fix spacing for runtime.zones.  Add cpuPlatform to subworkflow inputs.

* update wdl-ci config file after successful tests

---------

Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Change `msg` syntax.

Change `msg` concept so that task output `msg` is always `Array[String]`.  Any empty messages are pruned by the `Utilities.consolidate_stats` task, which outputs all raised messages to a text file.

* Parallelize alignment by chunking input.

- the split_input_bam task:
  - checks inputs for alignments and kinetics/basemod tags
    - resets alignments and kinetics tags
  - if max_reads_per_chunk is greater than 0, the input is indexed and split into chunks of no more than max_reads_per_chunk records

Note: We are investigating native chunking features in pbmm2, so this implementation may change.
* Update to sawfish 0.12.10
@williamrowell williamrowell force-pushed the feature/upload-artifacts-on-tag branch from 5633e44 to e8b6f1b Compare April 4, 2025 02:06
@williamrowell williamrowell changed the base branch from develop-v2 to develop-v3 April 4, 2025 02:06
williamrowell and others added 3 commits April 8, 2025 17:31
* Update to sawfish and sv_stats.

sawfish:
- Remove sex from inputs.
- Add `--verbose` to all fs operations.
- Switch to long form arguments.
- Clean up compressed/decompressed files at the end of each task.
- Pass supporting_reads_json up to main workflow outputs.

sv_stats:
- Remove InversionBND filtered events.
- Require that all events are FILTER="PASS" and non REF genotypes.
- Parameterize minimum variant size and maximum "scar" size (for sequence swaps)
- Filter "sequence swap" (len(REF) and len(ALT) both >10bp) complex variants from the INS and DEL counts, and put them in a separate SWAP category.  These are variants where both the REF and ALT are longer than 1bp.

* update wdl-ci config file after successful tests

---------

Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
…facts on tags that match semantic versions (only numerical versions).
@williamrowell williamrowell force-pushed the feature/upload-artifacts-on-tag branch from e8b6f1b to a9866b3 Compare April 9, 2025 01:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments