Commit 6f80ac3
committed
fix: prevent integer overflow and truncation in MergedPileup weighted mean
The weighted mean calculation `p.total * p.left_pos / total` had two bugs:
1. Integer overflow: `p.total * p.left_pos` is Int*Int which overflows
for large genomic positions (up to ~250M) with moderate counts.
2. Per-term truncation: integer division inside map() truncates each
term before summing, producing inaccurate results. E.g. three
pileups with total=1 at pos=100 with overall total=3: each term
becomes 1*100/3=33 (truncated), sum=99 instead of 100.
Fix uses Long arithmetic for the product and performs a single division
at the end.1 parent 6b12850 commit 6f80ac3
File tree
1 file changed
+2
-2
lines changed- src/main/scala/com/fulcrumgenomics/sv/tools
1 file changed
+2
-2
lines changedLines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
105 | 105 | | |
106 | 106 | | |
107 | 107 | | |
108 | | - | |
109 | | - | |
| 108 | + | |
| 109 | + | |
110 | 110 | | |
111 | 111 | | |
112 | 112 | | |
| |||
0 commit comments