v0.6.6: aggressive compression + sanitizer + SQuAD validation#34
Merged
Conversation
Phase 1: Quick wins - Fix compression dictionary SQL bug (missing LIMIT in phrases_fts query) 1 character, HIGH impact — dictionary never loaded before - Wire cache feedback loop: feed_cache_metrics result now persists to PerKeyState.paused. compress_messages skipped when paused (KV cache warming). - JSONL logging moved to spawn_blocking to avoid tokio event loop blocking under concurrent load. - Remove 30 lines of dead SSE body compression code (SSE path is true chunked streaming since walltime-fixes). Phase 2: Restore sift adaptive pipeline (+46 tests) Port original sift's content-type-aware compression: - classify.rs: Line struct, skeleton() normalization (UUID/hex/ version/progress/number→placeholders), skeleton_groups(), detect_strategy() (JSON/Diff/Tabular/Prefixed/Normal), compress_tabular() with visual gutter column pruning. - filter.rs: extract_error_blocks(), collapse_prefix_runs(), format_output() per-strategy formatting. - sift_compress_tool_result() uses adaptive pipeline: classify → detect→format→fallthrough. Hardcoded zone-truncation retained as guardrail for >200-line content. - Compression dictionary LIMIT fix in read_summary.rs means COMPRESSION_DICT now returns real FTS5 phrases. compress_assistant_text preserves project symbols.
Cherry-picked adaptive sift pipeline from proxy-compression-v3 branch into a clean break-ceiling-p1 branch and ran content-type smoke tests. Findings (documented in tests/smoke.rs): - cargo REPEAT (same crate): 4.6% of original (95% savings) — works - cargo MIXED (different crates): 100% of original (no savings) — fails - pytest tabular: 99.8% — fails - JSON array: 100% — fails - grep: 97.3% — minimal - ls -la: 100% — fails Root cause: each cargo 'Compiling X v1.0.0' line has unique crate name + version, so skeleton hashes differ per line. The adaptive pipeline can NOT compress cargo output where crate names vary (which is always in real builds). The 'test ... ok' collapse path DOES work, since those lines share skeletons. For real-world cargo output, basic sift (95.5% of original) and adaptive sift (96.3%) are nearly identical. The expected -38% session savings from the original adaptive branch did NOT materialize for cargo content. This means Phase 1 of the ceiling-break plan does not deliver on its promised gain. The remaining levers (FTS5 DF weighting, info-preserving zone) may help but won't break the -65% ceiling for the same reason: cargo output dominates session tokens and we can't structurally compress it without dropping the crate names (which would destroy signal). Recommend: keep break-ceiling-p1 branch but do not pursue further compression mechanism work on cargo output. Pivot to turn-reduction research (the other axis that could move the needle beyond -65% WC).
The basic skeleton preserves words (crate names), so cargo 'Compiling X v1.0.0'
lines have unique skeleton hashes per line and cannot be grouped for compression.
This is why the existing adaptive pipeline compresses cargo output to ~95% of
original (near-zero savings).
The new aggressive_skeleton normalizes ALL alpha words to {w}, so structurally
similar lines with different tokens share the same template:
Normal: 'Compiling serde 1.0.0' → 'compiling serde {ver}'
Aggressive: 'Compiling serde 1.0.0' → 'compiling {w} {ver}'
Aggressive: 'Compiling tokio 1.0.0' → 'compiling {w} {ver}'
The aggressive_skeleton_groups function groups lines by aggressive template and
collapses runs into single-sample + '[N+ more]' markers.
A self-selecting gate (should_use_aggressive) prevents false collapses on file
reads: aggressive mode only activates when ≥80% of non-blank lines share the
same template AND those lines have similar lengths (within 30%). This catches
cargo/pytest/build output (template-filled) while rejecting code files where
similar-looking function signatures are a small fraction of total lines.
Grammar-free: uses the same byte DFA as plain skeleton, just with an extra
word-collapse rule. No tool-specific keywords (no 'Compiling', no 'Building',
no 'FAILED'). The error marker guard (is_error) prevents collapsing lines that
contain errors, regardless of structural similarity.
Compression results on previously-failing content:
cargo MIXED (different crates): 100% → 5.7% of original (94% savings)
cargo REPEAT (same crate): 100% → 4.6% of original (95% savings)
pytest PASSED runs: collapse to [N ok]
error lines: preserved verbatim
file reads: preserved (gate rejects when <80% shared)
61 tests pass: 46 lib + 4 aggressive_compression + 8 aggressive_skeleton + 3 smoke.
Files:
crates/reliary-sift/src/classify.rs: +108 lines (aggressive_skeleton function)
crates/reliary-sift/src/filter.rs: +90/-12 (gate, aggressive_skeleton_groups, formatter wiring)
crates/reliary-sift/tests/aggressive_skeleton.rs: +98 lines (unit tests)
crates/reliary-sift/tests/aggressive_compression.rs: +97 lines (integration tests)
crates/reliary-sift/tests/smoke.rs: updated for new compression behavior
… shipped
This commit completes the break-ceiling plan: all three compression mechanisms
now stack to attack the only remaining uncached cost center (tool results).
Phase 2 — FTS5 Document Frequency Weighting (crates/reliary-search/src/ft_weight.rs):
Uses DF as a grammar-free pseudo-perplexity proxy (LLM Lingua analog without
an LM). Tokens appearing in ≥10 files are 'predictable' boilerplate; tokens in
<5 files are 'surprising' project-specific. Lines with low info-score are
preserved verbatim; high-score lines get compressed harder.
FtWeight::open(path) -> Option<Self> // Open FTS5 index
FtWeight::df(token) -> usize // Document frequency
FtWeight::line_info_score(line) -> f64 // Mean log(DF) per identifier
FtWeight::should_preserve(line) -> bool // score < 1.6 → preserve
FtWeight::compress_weighted(content, compress_fn) -> Option<String>
Grammar-free: no language detection, no keyword lists, pure FTS5 counts via
Porter stemming + phrase_occ join.
Phase 3 — Information-Preserving Zone Truncation
(crates/reliary-sift/src/lib.rs::zone_truncate_info):
Replaces blind first-30 + last-15 zone with score-based selection:
- is_error_line bonus (+10): preserves FAILED, error[, panic, Traceback
- Custom scorer (typically FTS5 DF-based): ranks by info density
- Position bonus (+0.5): first/last 3 lines preferred (primacy/recency)
Pick top-N lines by score, reassemble in original order with
'[…N lines…]' markers for dropped runs. Allows aggressive truncation
(top-15 instead of first-30+last-15 = 45 lines) without signal loss.
Helper is_error_line detects error patterns via substring matches
(FAILED, error[, panic, Traceback, Exception, etc.) — no regex, no AST.
Integration (crates/reliary-agent/src/proxy.rs::sift_compress_tool_result):
Step 1 (zone): if content >200 lines, build_info_scorer() opens FTS5 index
and returns a closure. zone_truncate_info uses the scorer to pick top-15
informative lines (vs legacy blind 30+15). Falls back to blind zone if
no FTS5 index available.
End-to-end verification:
Cargo test output (6100 chars, 205 lines with error in middle):
Original: 6100 chars
Compressed: ~225 chars
Savings: 96% (history_saved=5878)
Test counts:
crates/reliary-search: 14 tests (4 new for FtWeight)
crates/reliary-sift: 67 tests (6 new for zone_info)
Total workspace: 81+ tests passing, 0 failures
Clippy clean with -D warnings
Files:
crates/reliary-search/src/ft_weight.rs: +185 lines (FtWeight struct + tests)
crates/reliary-search/src/lib.rs: +1 line (pub mod ft_weight)
crates/reliary-sift/src/lib.rs: +75 lines (is_error_line, zone_truncate_info)
crates/reliary-sift/tests/zone_info.rs: +110 lines (6 tests)
crates/reliary-agent/src/proxy.rs: +25 lines (build_info_scorer)
…EIGHT gate
Three critical fixes for the proxy:
1. SSE finish_reason preservation (proxy.rs:858-905)
- Holds the chunk containing finish_reason: stop/length and sends it
after all preceding chunks have been forwarded
- Fixes 'Stream ended without finish_reason' error in Pi Agent
when SSE buffer fills and the client closes early
- Test: real DeepSeek V4 Flash now returns stopReason:stop with
input:96, output:38, cacheRead:1792 (was 0,0,0)
2. URL normalization (routes.rs:142-159)
- Handle /v1/<endpoint> pattern (e.g., DeepInfra's /v1/openai)
- Prevents double-pathing to /v1/v1/chat/completions
- Affects all non-Anthropic providers
3. Default SRCR floor at 0.3 + FT_WEIGHT gate
- SRCR (Signal-Preserving Compression Rate): preservation × compression
- Floor blocks destructive compression automatically
- FT_WEIGHT (FTS5 DF weighting) gated behind opt-in env var
- Default 0.3 floor with opt-out via RELIARY_PROXY_SRCR_FLOOR=0
Tests: 16 new (7 sr_floor, 9 ft_weight_gate). All 143 workspace tests pass.
Clippy clean with -D warnings.
Fixes: bench was returning 0 tokens due to 'Stream ended without finish_reason'
caused by the proxy dropping the final SSE chunk when the upstream closes.
When Pi retries after tool errors, it produces assistant messages whose tool_call_ids were already responded to by earlier tool messages. OpenAI/DeepSeek reject this with: 'An assistant message with tool_calls must be followed by tool messages responding to each tool_call_id. (insufficient tool messages following tool_calls message)' This is not an issue with simple one-shot requests, but multi-turn Pi sessions with retry logic hit it. The sanitizer: 1. Detects consecutive assistant-with-tool_calls messages reusing IDs 2. Strips the duplicate tool_calls from the second assistant 3. Removes orphaned tool responses that would otherwise reference non-existent tool_calls 4. Removes empty assistant messages (Pi's retry markers) that aren't the final message Default-on, no-op for well-formed sequences. Opt-out via RELIARY_PROXY_SANITIZER=0. Passthrough mode (RELIARY_PROXY_PASSTHROUGH=1) now actually passes through — zero compression, zero history modification, only the sanitizer (which is necessary for any Pi session to succeed against OpenAI/DeepSeek validation). Includes 4 unit tests covering well-formed, empty assistant, duplicate ID, and final-empty-preservation cases.
bench_squad.py measures whether compression preserves reading comprehension. Pass criterion: F1 retention >= 95% of baseline (matching Headroom's 97% target). Results from 3 runs × 30 samples = 90 calls per condition: baseline: F1=0.770, EM=0.711, Acc=80.00% recommended: F1=0.777, EM=0.733, Acc=78.89% F1=100.9% PASS passthrough: F1=0.791, EM=0.756, Acc=80.00% F1=102.6% PASS Compression preserves reading comprehension. F1 retention is slightly above 100% on both proxy conditions (within 2.7x LLM variance). Also fixed parser bug: previous version concatenated incremental message_update events (each carrying the full partial content), which produced 'FranceFranceFrance' style repetition. Now only the terminal message_end / agent_end event's text is returned. download_data.py fetches 100 SQuAD v2 + 100 BFCL samples from HuggingFace into /tmp/bench_*/ for reproducible benchmarking. README.md documents the benchmark methodology, results, and pass criteria.
Documents the SQuAD v2 benchmark result that proves compression preserves reading comprehension (F1 retention > 100% on both proxy conditions vs baseline).
The 100.9% and 102.6% numbers are within 2.7x LLM variance — not a real improvement signal. Documents this in both the main README Validation section and the benchmarks README Caveats section so users don't mistake noise for a magic accuracy boost. The 95% pass criterion is a regression-detection floor, not an equality. Result rules out compression degrading comprehension, but does not validate improvement.
Prevents future Python benchmarks from polluting git status with bytecode directories. Pure hygiene — no code changes.
…ipts Use REPO_ROOT env var or ~/src/reliary-agent expansion instead of a hardcoded absolute path. Makes the scripts portable across machines and survives repository relocation.
alderpath
force-pushed
the
break-ceiling-p1
branch
from
0f4466e to
8ff1877
Compare
Adds three layers of prevention: - .gitignore: ignore *.sqlite and SQLite journal files project-wide - pre-commit hook: refuse to stage any .sqlite file - CI guardrail: fail if any .sqlite file is tracked in repo These files are runtime databases built by the daemon (.reliary/*.sqlite). They get rebuilt on every project initialization and must never be committed.
alderpath
force-pushed
the
break-ceiling-p1
branch
from
8ff1877 to
fd87d10
Compare
The crate copy was corrupted with sed-substituted variable names (RELIARY_BIN_PATH → empty, DAEMON_HEALTHY → empty) that broke the binary discovery and daemon health check. Restored from the correct pi/gate.js root copy. Fixes gate.js sync CI check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Combines all work from break-ceiling-p1:
to prevent future commits of these files
4 commits, all tests pass, clippy clean.