Commit ca7b619
authored
fix: prevent integer overflow and truncation in MergedPileup weighted mean (#56)
* fix: prevent integer overflow and truncation in MergedPileup weighted mean
The weighted mean calculation `p.total * p.left_pos / total` had two bugs:
1. Integer overflow: `p.total * p.left_pos` is Int*Int which overflows
for large genomic positions (up to ~250M) with moderate counts.
2. Per-term truncation: integer division inside map() truncates each
term before summing, producing inaccurate results. E.g. three
pileups with total=1 at pos=100 with overall total=3: each term
becomes 1*100/3=33 (truncated), sum=99 instead of 100.
Fix uses Long arithmetic for the product and performs a single division
at the end.
* Generate docs files1 parent 81ec5c2 commit ca7b619
File tree
2 files changed
+3
-3
lines changed- docs/tools
- src/main/scala/com/fulcrumgenomics/sv/tools
2 files changed
+3
-3
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| |||
Lines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
105 | 105 | | |
106 | 106 | | |
107 | 107 | | |
108 | | - | |
109 | | - | |
| 108 | + | |
| 109 | + | |
110 | 110 | | |
111 | 111 | | |
112 | 112 | | |
| |||
0 commit comments