Commit 737d659
fix: comprehensive resolution of inter-section uniqueness consistency bugs
This addresses the root cause why Section 2 and Section 4 reported different
uniqueness percentages even after the initial fix in commit b1d5305.
Root Cause Analysis:
- Section 2 (Quality) correctly used shared data-quality-utils
- Section 4 (Visualization) gets data from Section 3 (EDA/Streaming)
- Section 3 streaming analyzers had critical bugs in uniqueness calculations
Critical Bugs Fixed:
1. **Text Analyzer**:
- Added valueFrequencies to track unique text values
- Fixed: Was using validValues count instead of actual unique values
- Fixed: Was using totalValues instead of validValues as denominator
2. **Boolean Analyzer**:
- Fixed denominator from totalValues to validValues
- Now correctly calculates percentage of valid values that are unique
3. **DateTime Analyzer**:
- Added dateValueFrequencies to track unique date values
- Fixed: Was using sample size (maxDateSamples=50) instead of actual count
- Now tracks all unique date values properly
Verification Results:
✅ All sections now report identical uniqueness percentages
✅ Manual verification: first_name 66.7% - consistent across sections
✅ All 1,493 tests passing - no regressions
✅ TypeScript compilation clean
This completely resolves GitHub Issue #46.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>1 parent 1b4a4c3 commit 737d659
1 file changed
+13
-5
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
728 | 728 | | |
729 | 729 | | |
730 | 730 | | |
| 731 | + | |
731 | 732 | | |
732 | 733 | | |
733 | 734 | | |
| |||
756 | 757 | | |
757 | 758 | | |
758 | 759 | | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
759 | 763 | | |
760 | 764 | | |
761 | 765 | | |
| |||
822 | 826 | | |
823 | 827 | | |
824 | 828 | | |
825 | | - | |
826 | | - | |
| 829 | + | |
| 830 | + | |
827 | 831 | | |
828 | 832 | | |
829 | 833 | | |
| |||
1069 | 1073 | | |
1070 | 1074 | | |
1071 | 1075 | | |
1072 | | - | |
| 1076 | + | |
1073 | 1077 | | |
1074 | 1078 | | |
1075 | 1079 | | |
| |||
1132 | 1136 | | |
1133 | 1137 | | |
1134 | 1138 | | |
| 1139 | + | |
1135 | 1140 | | |
1136 | 1141 | | |
1137 | 1142 | | |
| |||
1157 | 1162 | | |
1158 | 1163 | | |
1159 | 1164 | | |
| 1165 | + | |
| 1166 | + | |
| 1167 | + | |
1160 | 1168 | | |
1161 | 1169 | | |
1162 | 1170 | | |
| |||
1230 | 1238 | | |
1231 | 1239 | | |
1232 | 1240 | | |
1233 | | - | |
1234 | | - | |
| 1241 | + | |
| 1242 | + | |
1235 | 1243 | | |
1236 | 1244 | | |
1237 | 1245 | | |
| |||
0 commit comments