Formula extraction missing files and failing #42
Description
mentioned in #37 (comment)
Formula Extraction Comparison Report
Old = excavate v0.3.9 with ffi-libarchive-binary, libmspack, seven-zip, ruby-ole
New = excavate v1.0.0 with omnizip 0.3.8 + cabriolet 0.2.2
Total archives tested: 137
Archives with Missing Files (New < Old)
| Archive | Old Files | New Files | Missing |
|---|---|---|---|
| Korean.exe | 2531 | 2218 | 313 |
| Chinese_Simplified.exe | 2548 | 2313 | 235 |
| Office_Preview.pkg | 16227 | 16111 | 116 |
| Arabic.exe | 2527 | 2499 | 28 |
| Chinese_Traditional.exe | 2519 | 2500 | 19 |
Total missing files across regressions: 711
Root causes (known)
- Korean.exe, Chinese_Simplified.exe, Arabic.exe, Chinese_Traditional.exe: These are Microsoft Office Language Pack EXEs containing nested CABs with MSZIP and LZX compression. The cabriolet LZX decompressor has issues with certain LZX-compressed inner CABs.
- Office_Preview.pkg: macOS package format. 116 files missing - needs investigation in omnizip's pkg handler.
Archives that Fail in New (worked in Old)
| Archive | Old Files | New Error |
|---|---|---|
| Adobe_Reader_20.exe | 365 | Omnizip::DecompressionError: Invalid distance: 12 (dict_full: 0) |
| Lucida_Grande.exe | 1016 | TypeError: no implicit conversion of nil into String |
| wine-11.2.tar.xz | 10894 | Omnizip::FormatError: Failed to decode XZ format: Invalid distance: 144415020 (dict_full: 223264) |
Total files lost from failures: 12,275
Root causes (known)
- Adobe_Reader_20.exe: omnizip LZMA2 decoder bug - invalid distance error during decompression
- Lucida_Grande.exe: omnizip 7z stream decoder bug - nil value passed where String expected
- wine-11.2.tar.xz: omnizip XZ/LZMA2 decoder bug - invalid distance during LZMA2 decompression
All 3 failures are in omnizip decoders (LZMA2/XZ/7z), not in cabriolet.
Download Failures
| Archive | Issue |
|---|---|
| montserrat.zip | fontsquirrel.com blocks automated downloads, returns empty response instead of zip |
| open-sans.zip | fontsquirrel.com blocks automated downloads, returns empty response instead of zip |
Summary
| Category | Count |
|---|---|
| Exact match (old == new) | 129 |
| Missing files (new < old) | 5 |
| Old works, new fails | 3 |
| Download still broken (fontsquirrel) | 2 |
| Total tested | 137 |