Merge branch 'release/v5.4.0'

Author: ACEnglish

.github/workflows/func_tests.yml

@@ -15,16 +15,21 @@ jobs:
         uses: actions/setup-python@v4
         with:
           python-version: 3.10.4
+      - name: Upgrade pip
+        run: |
+          curl -sS https://bootstrap.pypa.io/get-pip.py -o get-pip.py
+          python3 get-pip.py
+          python3 -m pip install --upgrade pip
       - name: Cache python
         uses: actions/cache@v3
         with:
           path: ${{ env.pythonLocation }}
           key: ${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}
       - name: Install dependencies
         run: |
-          python3 -m pip install --upgrade pip setuptools
+          python3 -m pip install --upgrade setuptools
           python3 -m pip install Cython pylint anybadge coverage
-          python3 -m pip install .
+          python3 -m pip install ./[bwa]
       - name: Running ssshtest
         run: |
           TMPDIR=`pwd` bash repo_utils/truvari_ssshtests.sh

Dockerfile

@@ -20,7 +20,7 @@ RUN wget https://mafft.cbrc.jp/alignment/software/mafft_7.505-1_amd64.deb \
 RUN python3 -m pip install --upgrade pip && \
     python3 -m pip install setproctitle pylint anybadge coverage && \
     python3 -m pip install --upgrade setuptools && \
-    python3 -m pip install ./
+    python3 -m pip install ./[bwa]
 
 WORKDIR /data
 

docs/Home.md

@@ -3,34 +3,28 @@ The wiki holds documentation most relevant for develop. For information on a spe
 Citation:  
 English, A.C., Menon, V.K., Gibbs, R.A. et al. Truvari: refined structural variant comparison preserves allelic diversity. Genome Biol 23, 271 (2022). https://doi.org/10.1186/s13059-022-02840-6
 
-# Before you start
-VCFs aren't always created with a strong adherence to the format's specification. 
-
-Truvari expects input VCFs to be valid so that it will only output valid VCFs. 
-
-We've developed a separate tool that runs multiple validation programs and standard VCF parsing libraries in order to validate a VCF. 
-
-Run [this program](https://github.com/acenglish/usable_vcf) over any VCFs that are giving Truvari trouble. 
-
-Furthermore, Truvari expects 'resolved' SVs (e.g. DEL/INS) and will not interpret BND signals across SVTYPEs (e.g. combining two BND lines to match a DEL call). A brief description of Truvari bench methodology is linked below.
-
-Finally, Truvari does not handle multi-allelic VCF entries and as of v4.0 will throw an error if multi-allelics are encountered. Please use `bcftools norm` to split multi-allelic entries. 
-
 # Index
 
 - [[Updates|Updates]]
 - [[Installation|Installation]]
-- Truvari Commands:
-  - [[anno|anno]]
-  - [[bench|bench]]
-  - [[collapse|collapse]]
-  - [[consistency|consistency]]
-  - [[divide|divide]]
-  - [[ga4gh|ga4gh]]
-  - [[phab|phab]]
-  - [[refine|refine]]
-  - [[segment|segment]]
-  - [[stratify|stratify]]
-  - [[vcf2df|vcf2df]]
 - [[Development|Development]]
-- [[Citations|Citations]]
+
+Truvari Commands:  
+- Benchmarking  
+  - [[bench|bench]] - Performance metrics from comparison of two VCFs  
+  - [[refine|refine]] - Automated bench result refinement with phab  
+- Merging  
+  - [[collapse|collapse]]     Collapse redundant VCF entries  
+  - [[phab|phab]]         Variant harmonization using MSA  
+- Analysis  
+  - [[consistency|consistency]]  Consistency report between multiple VCFs  
+  - [[stratify|stratify]]     Count VCF entries inside BED regions  
+  - [[vcf2df|vcf2df]]       Turn VCF into pandas DataFrame  
+  - [[stratp|Stratp-Test]]       Stratification performance test  
+- Annotation  
+  - [[anno|anno]]         VCF Annotations  
+- Misc  
+  - [[segment|segment]]      Normalization of SVs into disjointed genomic regions  
+  - [[divide|divide]]       Divide a VCF into independent shards  
+  - [[ga4gh|ga4gh]]        Convert Truvari result to GA4GH  
+

docs/Installation.md

@@ -10,6 +10,11 @@ python3 -m pip install truvari==3.2.0
 ```
 See [pypi](https://pypi.org/project/Truvari/#history) for a history of all distributed releases.
 
+When using some annotations (e.g. `truvari anno remap`) The bwapy needs to be available. This can be installed via:
+```
+python3 -m pip install truvari[bwa]
+```
+
 Manual Installation
 ===================
 To build Truvari directly, clone the repository and switch to a specific tag.

docs/Updates.md

@@ -1,5 +1,17 @@
-# Truvari 5.3
-*in progress*
+# Truvari 5.4.0
+*October 7, 2025*
+
+* New `stratp` command to automatically generate benchmark performance evaluation across stratifications
+* `truvari.VariantRecord.allele_freq_annos` now stores the results to speed up reuse in e.g. `collapse`.
+* LazyImporting for faster startup times
+* `collapse` now allows `--sizemax -1` to work with all large SVs easily.
+* New `collapse` argument `--fast-cluster` will dramatically speed up runtime when collapsing large (>100kbp) SVs
+* bwapy, which is a bother to install on macs, is now optional by default (#295)
+* `vcf2df --parquet` will write a parquet file, which is more stable across environments than the default joblib file.
+* Miscellaneous bug fixes (#288, #286, #284, #282, #275)
+
+# Truvari 5.3.0
+*April 21, 2025*
 
 * Fixed FP BNDs being dropped [details](https://github.com/ACEnglish/truvari/discussions/263). 
 * Restore default `--sizemax` - Some callers make SVs that span the entire chromosome, which disrupts truvari's chunking strategy

docs/bench.md

@@ -242,6 +242,8 @@ This VCF makes different results depending on the `--pick` parameter
 | ac        |    TP     |   TP      |    FP     |
 | multi     |    TP     |   TP      |    TP     |
 
+Note that multi-matching should be used with care. By allowing SVs to match multiple times, performance metrics become inflated in a way that’s misleading. Recall can exceed the number of calls made. Precision can be skewed if one baseline event explains many false calls. For example, if a single comparison SV matches to two baseline SVs, the caller only made one prediction, yet it’s getting credit for finding two events, thus inflating recall.
+
 --dup-to-ins
 ============
 

imgs/coverage.svg

@@ -13,7 +13,6 @@ license = { text = "MIT" }
 dynamic = ["version"]
 requires-python = ">=3.8"
 dependencies = [
-    "bwapy>=0.1.4",
     "edlib>=1.3.9",
     "intervaltree>=3.1",
     "joblib>=1.2.0",
@@ -27,6 +26,9 @@ dependencies = [
     "pywfa>=0.5.1",
 ]
 
+[project.optional-dependencies]
+bwa = ["bwapy>=0.1.4"]
+
 [project.scripts]
 truvari = "truvari.__main__:main"
 

pyproject.toml

@@ -1577,13 +1577,13 @@ chr18	74756025	pbsv.INS.1213	C	CCTCCCTCCCTTTCTTTCTTTTT	2	PASS	IMPRECISE;SVTYPE=I
 chr18	74766009	pbsv.DEL.1214	CTGTGTGTGTGTGTGTGTGTG	C	2	PASS	SVTYPE=DEL;SVLEN=-20;SVANN=TANDEM;AC=2	GT:AD:DP:SAC:SUPP	1/1:0,11:11:0,0,3,8:4
 chr18	75067409	chr18-75067410-DEL-86;pbsv.DEL.1215	CGCTGGAAGCTCCACTGCCCTTTACAAGGTTCTATGAGCGCGGGGCTGGAAGCTCCACTGCCCTTTACAAGGTTCTATGAGCGCGGG	C	4	PASS	ID=chr18-75067410-DEL-86;SVTYPE=DEL;SVLEN=-86;TIG_REGION=h1tg006280l:236990-236990,h2tg003451l:128712-128712;QUERY_STRAND=+,-;HOM_REF=0,123;HOM_TIG=0,123;SVANN=TANDEM;AC=3;NumCollapsed=1;NumConsolidated=1;CollapseId=508.0	GT:AD:DP:SAC:DR:DV:GQ:SUPP	1/1:7,1:8:6,1,1,0:.:8:22:7
 chr18	75068024	chr18-75068025-DEL-50	AGAAGATGGCTAAAAGTACGCACAGGGAAGGGGAGCAGGCACTGGTGGATG	A	4	.	ID=chr18-75068025-DEL-50;SVTYPE=DEL;SVLEN=-50;TIG_REGION=h1tg006280l:237519-237519,h2tg003451l:128183-128183;QUERY_STRAND=+,-;HOM_REF=0,15;HOM_TIG=0,15;AC=2;NumCollapsed=1;NumConsolidated=1;CollapseId=509.0	GT:DR:DV:GQ:SUPP	1/1:.:7:19:3
-chr18	75105022	chr18-75105023-DEL-78;pbsv.DEL.1216	CGTGGAAGCTTTGCTGAAATGTCCTGCTTGTGTTTTACTCCGTGGCGAGCACAGCGTGCAGGTGCTCCGTGGAAGCTCA	C	4	PASS	ID=chr18-75105023-DEL-78;SVTYPE=DEL;SVLEN=-78;TIG_REGION=h2tg003451l:91227-91227;QUERY_STRAND=-;HOM_REF=0,10;HOM_TIG=0,10;SVANN=TANDEM;AC=2;NumCollapsed=2;NumConsolidated=1;CollapseId=510.0	GT:AD:DP:SAC:DR:DV:GQ:SUPP	0/1:6,5:11:2,4,1,4:8:3:7:7
+chr18	75105022	chr18-75105023-DEL-78;pbsv.DEL.1216	CGTGGAAGCTTTGCTGAAATGTCCTGCTTGTGTTTTACTCCGTGGCGAGCACAGCGTGCAGGTGCTCCGTGGAAGCTCA	C	4	PASS	ID=chr18-75105023-DEL-78;SVTYPE=DEL;SVLEN=-78;TIG_REGION=h2tg003451l:91227-91227;QUERY_STRAND=-;HOM_REF=0,10;HOM_TIG=0,10;SVANN=TANDEM;AC=2;NumCollapsed=2;NumConsolidated=1;CollapseId=510.0	GT:AD:DP:SAC:DR:DV:GQ:SUPP	0/1:6,5:11:2,4,1,4:9:2:5:5
 chr18	75108304	pbsv.DEL.1217	GCCTCACTCCCGCCAGCCCAGCAC	G	2	PASS	SVTYPE=DEL;SVLEN=-23;SVANN=TANDEM;AC=1	GT:AD:DP:SAC:SUPP	0/1:5,4:9:1,4,2,2:4
 chr18	75132968	pbsv.INS.DUP.1218	A	AGATCCCCGCGGCTCAGTGG	0	PASS	SVTYPE=INS;SVLEN=19;AC=2	GT:AD:DP:SAC:SUPP	1/1:0,9:9:0,0,0,9:4
 chr18	75246403	pbsv.INS.1219	G	GGTGTGTGTGTGTGTGTGTGT	2	PASS	SVTYPE=INS;SVLEN=20;AC=1	GT:AD:DP:SAC:SUPP	0/1:4,5:9:2,2,2,3:4
 chr18	75293906	pbsv.INS.1220	A	AACTGAGACACTGGCTCCTCTGTGGGTGTGGAGAAAGAACTGG	2	PASS	SVTYPE=INS;SVLEN=42;AC=2	GT:AD:DP:SAC:SUPP	1/1:0,9:9:0,0,4,5:4
-chr18	75299569	chr18-75299570-DEL-132;pbsv.DEL.1221	CCAGTGGACGGTGATCCATCGTTAAAGGACATGGTGAGCTTGCACCAGGCACTAGATAGCTGCCCCGCAGTGGACGGTGATCCATCGTTAAAGGACATGGTGAGCTTGCACCAGGCACTAGATAGCTGCCCCG	C	4	PASS	ID=chr18-75299570-DEL-132;SVTYPE=DEL;SVLEN=-132;TIG_REGION=h1tg006280l:469108-469108,h2tg015582l:3860-3860;QUERY_STRAND=+,-;HOM_REF=0,288;HOM_TIG=0,288;SVANN=TANDEM;AC=4;NumCollapsed=1;NumConsolidated=1;CollapseId=511.0	GT:AD:DP:SAC:DR:DV:GQ:SUPP	1/1:0,7:7:0,0,4,3:2:5:6:7
-chr18	75299831	Sniffles2.DEL.36CS11	CCGCAGTGGACGGTGATCCATCGTTAAAGGACATGGTGAGCTTGCACCAGGCACTAGATAGCTGCCCCGCAGTGGACGGTGATCCATCGTTAAAGGACATGGTGAGCTTGCACCAGGCACTAGATAGCTGCC	C	0	PASS	PRECISE;SVTYPE=DEL;SVLEN=-131;SUPPORT=2;COVERAGE=7,7,7,7,7;STRAND=+-;AF=0.286;STDEV_LEN=1.414;STDEV_POS=21.92;AC=1	GT:GQ:DR:DV:SUPP	0/1:6:5:2:2
+chr18	75299569	chr18-75299570-DEL-132;pbsv.DEL.1221	CCAGTGGACGGTGATCCATCGTTAAAGGACATGGTGAGCTTGCACCAGGCACTAGATAGCTGCCCCGCAGTGGACGGTGATCCATCGTTAAAGGACATGGTGAGCTTGCACCAGGCACTAGATAGCTGCCCCG	C	4	PASS	ID=chr18-75299570-DEL-132;SVTYPE=DEL;SVLEN=-132;TIG_REGION=h1tg006280l:469108-469108,h2tg015582l:3860-3860;QUERY_STRAND=+,-;HOM_REF=0,288;HOM_TIG=0,288;SVANN=TANDEM;AC=4;NumCollapsed=1;NumConsolidated=1;CollapseId=511.0	GT:AD:DP:SAC:DR:DV:GQ:SUPP	1/1:0,7:7:0,0,4,3:5:2:6:7
+chr18	75299570	Sniffles2.DEL.36BS11	CAGTGGACGGTGATCCATCGTTAAAGGACATGGTGAGCTTGCACCAGGCACTAGATAGCTGCCCCGCAGTGGACGGTGATCCATCGTTAAAGGACATGGTGAGCTTGCACCAGGCACTAGATAGCTGCCCCGC	C	0	PASS	PRECISE;SVTYPE=DEL;SVLEN=-132;SUPPORT=5;COVERAGE=7,7,7,7,7;STRAND=+-;AF=0.714;STDEV_LEN=0;STDEV_POS=0;AC=1	GT:GQ:DR:DV:SUPP	0/1:6:2:5:2
 chr18	75383945	pbsv.DEL.1222	GGAGAGAGAGAGAGAGAGAGAGAGAGAGAGA	G	2	PASS	SVTYPE=DEL;SVLEN=-30;SVANN=TANDEM;AC=2	GT:AD:DP:SAC:SUPP	1/1:0,9:9:0,0,2,7:4
 chr18	75461601	pbsv.INS.1223	T	TTCCCTCCCTCCCTCCCTCCC	2	PASS	SVTYPE=INS;SVLEN=20;AC=2	GT:AD:DP:SAC:SUPP	1/1:0,6:6:0,0,2,4:4
 chr18	75526339	pbsv.INS.1224	T	TCACTGCCGTAACAAATGGGG	2	PASS	SVTYPE=INS;SVLEN=20;SVANN=TANDEM;AC=1	GT:AD:DP:SAC:SUPP	0/1:1,3:4:0,1,3,0:4

repo_utils/answer_key/collapse/inputintragt_collapsed.vcf

@@ -597,7 +597,7 @@ chr18	75067410	Sniffles2.DEL.367S11	GCTGGAAGCTCCACTGCCCTTTACAAGGTTCTATGAGCGCGGGG
 chr18	75068025	Sniffles2.DEL.368S11	GAAGATGGCTAAAAGTACGCACAGGGAAGGGGAGCAGGCACTGGTGGATGG	G	0	PASS	PRECISE;SVTYPE=DEL;SVLEN=-50;SUPPORT=7;COVERAGE=8,7,7,7,7;STRAND=+-;AF=1;STDEV_LEN=0;STDEV_POS=0;AC=2;PctSeqSimilarity=0.9902;PctSizeSimilarity=1;PctRecOverlap=0.9804;SizeDiff=0;StartDistance=-1;EndDistance=-1;TruScore=99;MatchId=509.0	GT:GQ:DR:DV	./.:.:.:.	1/1:19:0:7	./.:.:.:.
 chr18	75104861	Sniffles2.DEL.369S11	GAAGCTCCGTGGAAGCTTGCTGAAATGTCCTGCTTGTGTTTTACTCCGTGGCGAGCACAGCGTGCAGGCGCTCCGTG	G	0	GT	PRECISE;SVTYPE=DEL;SVLEN=-76;SUPPORT=2;COVERAGE=9,11,11,11,11;STRAND=-;AF=0.182;STDEV_LEN=0.707;STDEV_POS=4.95;AC=0;PctSeqSimilarity=0.9808;PctSizeSimilarity=0.9744;PctRecOverlap=0;SizeDiff=2;StartDistance=161;EndDistance=163;TruScore=65;MatchId=510.0	GT:GQ:DR:DV	./.:.:.:.	0/0:5:9:2	./.:.:.:.
 chr18	75105023	Sniffles2.DEL.36AS11	GTGGAAGCTTTGCTGAAATGTCCTGCTTGTGTTTTACTCCGTGGCGAGCACAGCGTGCAGGTGCTCCGTGGAAGCTCAG	G	0	PASS	PRECISE;SVTYPE=DEL;SVLEN=-78;SUPPORT=3;COVERAGE=10,11,11,11,11;STRAND=+-;AF=0.273;STDEV_LEN=0;STDEV_POS=0;AC=1;PctSeqSimilarity=0.9937;PctSizeSimilarity=1;PctRecOverlap=0.9873;SizeDiff=0;StartDistance=-1;EndDistance=-1;TruScore=99;MatchId=510.0	GT:GQ:DR:DV	./.:.:.:.	0/1:7:8:3	./.:.:.:.
-chr18	75299570	Sniffles2.DEL.36BS11	CAGTGGACGGTGATCCATCGTTAAAGGACATGGTGAGCTTGCACCAGGCACTAGATAGCTGCCCCGCAGTGGACGGTGATCCATCGTTAAAGGACATGGTGAGCTTGCACCAGGCACTAGATAGCTGCCCCGC	C	0	PASS	PRECISE;SVTYPE=DEL;SVLEN=-132;SUPPORT=5;COVERAGE=7,7,7,7,7;STRAND=+-;AF=0.714;STDEV_LEN=0;STDEV_POS=0;AC=1;PctSeqSimilarity=1;PctSizeSimilarity=1;PctRecOverlap=0.9925;SizeDiff=0;StartDistance=-1;EndDistance=-1;TruScore=99;MatchId=511.0	GT:GQ:DR:DV	./.:.:.:.	0/1:6:2:5	./.:.:.:.
+chr18	75299831	Sniffles2.DEL.36CS11	CCGCAGTGGACGGTGATCCATCGTTAAAGGACATGGTGAGCTTGCACCAGGCACTAGATAGCTGCCCCGCAGTGGACGGTGATCCATCGTTAAAGGACATGGTGAGCTTGCACCAGGCACTAGATAGCTGCC	C	0	PASS	PRECISE;SVTYPE=DEL;SVLEN=-131;SUPPORT=2;COVERAGE=7,7,7,7,7;STRAND=+-;AF=0.286;STDEV_LEN=1.414;STDEV_POS=21.92;AC=1;PctSeqSimilarity=0.9962;PctSizeSimilarity=0.9924;PctRecOverlap=0;SizeDiff=1;StartDistance=-262;EndDistance=-261;TruScore=66;MatchId=511.0	GT:GQ:DR:DV	./.:.:.:.	0/1:6:5:2	./.:.:.:.
 chr18	75526515	pbsv.INS.1225	C	CGGAATGGAAGTACATGGCAAGCTCACGGACCATGGGGGGACACAACCAAGTAGAGTCTGGGGAGGTTGGCTGGACGGGGCTGGGAAATGATAGAATTAGCAGAAACACGAT	2	PASS	SVTYPE=INS;SVLEN=111;SVANN=TANDEM;AC=1;PctSeqSimilarity=0.9955;PctSizeSimilarity=0.991;PctRecOverlap=0;SizeDiff=-1;StartDistance=170;EndDistance=170;TruScore=66;MatchId=512.0	GT:AD:DP:SAC	./.:.:.:.	./.:.:.:.	0/1:1,3:4:0,1,3,0
 chr18	75526686	Sniffles2.INS.16CS11	G	GGGAGGTTGGCTGGACGGGGCTGGGAAATGATAGAATTAGCAGAAACACGATGGAATGGAAGTACATGGCAAGCTCACGGACCATGGGGGGACACAACCAAGTAGAGTCT	3	PASS	IMPRECISE;SVTYPE=INS;SVLEN=111;SUPPORT=3;COVERAGE=4,4,4,4,4;STRAND=+;AF=0.75;STDEV_LEN=0.577;STDEV_POS=98.15;SUPPORT_LONG=0;AC=1;PctSeqSimilarity=0.9955;PctSizeSimilarity=0.991;PctRecOverlap=0.991;SizeDiff=-1;StartDistance=-1;EndDistance=-1;TruScore=99;MatchId=512.0	GT:GQ:DR:DV	./.:.:.:.	0/1:1:1:3	./.:.:.:.
 chr18	75712365	pbsv.DEL.1228	CAAAATGGCAGCTGCATGGCTGACTCTCAGATCCAAAATGGCTGCTGCATGGCCGACTCTCTCAGATCC	C	2	PASS	SVTYPE=DEL;SVLEN=-68;SVANN=TANDEM;AC=1;PctSeqSimilarity=0.9855;PctSizeSimilarity=1;PctRecOverlap=0.7971;SizeDiff=0;StartDistance=-14;EndDistance=-14;TruScore=92;MatchId=513.0	GT:AD:DP:SAC	./.:.:.:.	./.:.:.:.	0/1:4,2:6:1,3,2,0