Skip to content

Commit ca7b619

Browse files
authored
fix: prevent integer overflow and truncation in MergedPileup weighted mean (#56)
* fix: prevent integer overflow and truncation in MergedPileup weighted mean The weighted mean calculation `p.total * p.left_pos / total` had two bugs: 1. Integer overflow: `p.total * p.left_pos` is Int*Int which overflows for large genomic positions (up to ~250M) with moderate counts. 2. Per-term truncation: integer division inside map() truncates each term before summing, producing inaccurate results. E.g. three pileups with total=1 at pos=100 with overall total=3: each term becomes 1*100/3=33 (truncated), sum=99 instead of 100. Fix uses Long arithmetic for the product and performs a single division at the end. * Generate docs files
1 parent 81ec5c2 commit ca7b619

File tree

2 files changed

+3
-3
lines changed

2 files changed

+3
-3
lines changed

docs/tools/index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ title: fgsv tools
44

55
# fgsv tools
66

7-
The following tools are available in fgsv version 0.2.1-77ca3f0.
7+
The following tools are available in fgsv version 0.2.1-21111db.
88
## Breakpoint and SV Tools
99

1010
Primary tools for calling and transforming breakpoints and SVs.

src/main/scala/com/fulcrumgenomics/sv/tools/FilterAndMerge.scala

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -105,8 +105,8 @@ object MergedPileup {
105105
val readPairs = pileups.sumBy(_.read_pairs)
106106
val counts = NumericCounter(pileups.map(_.total).iterator)
107107
val total = splitReads + readPairs
108-
val leftMean = pileups.iterator.map(p => p.total * p.left_pos / total).sum
109-
val rightMean = pileups.iterator.map(p => p.total * p.right_pos / total).sum
108+
val leftMean = (pileups.iterator.map(p => p.total.toLong * p.left_pos).sum / total).toInt
109+
val rightMean = (pileups.iterator.map(p => p.total.toLong * p.right_pos).sum / total).toInt
110110
val meanCount = counts.mean()
111111

112112
new MergedPileup(

0 commit comments

Comments
 (0)