Bug: consolidation truncates archive input but advances past the full message batch

[dajiaohuang]

Bug Description

Consolidator.archive() truncates the formatted conversation to the consolidation model's input-token budget, but its callers still advance Session.last_consolidated past the entire original message batch.

Any messages or message suffixes removed by _truncate_to_token_budget() are therefore treated as archived even though they were never shown to the consolidation provider and were not written to memory/history.jsonl through the raw fallback path.

This affects both token-triggered consolidation and idle-session compaction.

Steps to Reproduce

1. Create a session containing a message large enough to exceed the consolidation input budget, followed by a message containing a unique marker.
2. Run Consolidator.compact_idle_session() with a small context window.
3. Capture the user content passed to provider.chat_with_retry().
4. Reload the persisted session and inspect last_consolidated and memory/history.jsonl.

Observed on current main:

tail marker present in provider input: false
provider input characters: 16016
persisted last_consolidated: 2
persisted message count: 2
history entries: 1 (summary of the visible prefix only)

The unique marker was never supplied to the consolidator, but the session cursor advanced past it.

The relevant path is:

• Consolidator.archive() formats all messages and truncates the resulting string.
• maybe_consolidate_by_tokens() subsequently sets last_consolidated = end_idx.
• compact_idle_session() subsequently sets last_consolidated = archive_end.

Expected Behavior

Consolidation progress should advance only through source content that was actually archived. If a batch exceeds the provider input budget, it should be processed in bounded sub-batches (including a safe strategy for a single oversized message), or the unprocessed suffix should remain eligible for a later run.

Relevant Logs

No runtime error is emitted. The loss is silent because the provider successfully summarizes the truncated prefix.

nanobot Version

main@af582246f141311d574551b7571a517bcc3df750

Python Version

3.12

Operating System

Windows

Channel / Platform

Other (reproduced directly through Consolidator)

LLM Provider

Other (deterministic fake provider; provider-independent)

Configuration (Optional)

contextWindowTokens: 5000
maxTokens: 4096

Additional Context

A focused fix should preserve the existing input budget while aligning archive progress with the content actually supplied to the provider. Regression coverage should include token-triggered consolidation, idle compaction, and an oversized individual message.

[Solaris-star]

I reproduced this against current main at 86d228b38e63fbc2bb241801f409607cdbf34135 with a deterministic fake provider:

• context_window_tokens=5000, max_tokens=4096;
• first user message: 100,000 x characters;
• second user message: UNIQUE_TAIL_MARKER_5377;
• invoked Consolidator.compact_idle_session() through a real MemoryStore + SessionManager.

Observed output:

marker_in_provider_input: False
provider_input_characters: 16016
persisted_last_consolidated: 2
persisted_message_count: 2
history_entries: 1

So the issue is still present: the suffix is omitted by _truncate_to_token_budget(), while compact_idle_session() advances the cursor across both source messages. The existing regression suite is otherwise green: uv sync --extra dev && uv run pytest -q tests/agent/test_consolidator.py — 49 passed.

I have not opened a PR because the safe cursor contract needs maintainer direction for the oversized-single-message case. The implementation should either return/track the actually materialized source range, or split into bounded message batches while leaving an unprocessed suffix eligible for the next consolidation run. An oversized individual message also needs an explicit raw/archive policy rather than silently advancing past omitted content.

[dajiaohuang]

Thanks for the independent reproduction. The last_consolidated=2 result while the marker was omitted confirms the cursor/materialization mismatch described here. The linked #5379 already takes the bounded, lossless-batch approach and includes oversized-input coverage; all of its current CI checks are passing. Your reproduction is useful confirmation that this is still present on current main rather than only a synthetic edge case.