Skip to content

feat(query): time-based queries with @ts virtual field and index integrity fixes#88

Merged
raaymax merged 14 commits intomasterfrom
feat-time-based-mcp-queries
Mar 26, 2026
Merged

feat(query): time-based queries with @ts virtual field and index integrity fixes#88
raaymax merged 14 commits intomasterfrom
feat-time-based-mcp-queries

Conversation

@raaymax
Copy link
Copy Markdown
Owner

@raaymax raaymax commented Mar 19, 2026

Summary

  • Time-based field filtering: filter values like now-5m, now-1h30m and absolute timestamps are resolved at query time for comparison operators on any field
  • @ts virtual field: filters by index ingestion timestamp (when lazytail captured the line) rather than a log payload field. Works as a bitmap at the scan level — no changes to the Filter trait
  • Capture index offset fix: correct index offsets when appending to existing log files (the root cause of garbled lines and UTF-8 errors in TUI)
  • Corrupt index detection and warnings: validate indexes on open, show popup + persistent status bar warning for broken indexes

Usage

# Time-based field filtering (TUI query mode)
json | timestamp >= "now-5m"
logfmt | ts >= "2024-01-15T10:00:00Z"

# @ts virtual field — filter by ingestion time
@ts >= "now-5m"
@ts >= "now-1h" | @ts < "now-5m" | json | level == "error"
json | @ts >= "now-30m" | level == "error"

# MCP structured query
{"field": "@ts", "op": "gte", "value": "now-5m"}

Changes

Time-based queries

  • time.rs: relative time resolution (now-5m), timestamp parser (RFC 3339, ISO 8601, epoch s/ms), TsBounds struct for @ts bitmap evaluation
  • filter.rs: pre-resolve time values at QueryFilter construction for comparison operators
  • parser.rs: accept @ prefix in field names, route @ts to ts_filters, optional parser prefix (@ts >= "now-5m" works without json |)
  • ast.rs: ts_filters field on FilterQuery, partition_ts_filters() for MCP JSON path
  • search_engine.rs: build @ts bitmap from index timestamps, AND with flags bitmap
  • filter_orchestrator.rs: validate @ts requires indexed file source
  • filter_controller.rs: validate @ts values during live filter preview

Index integrity

  • capture.rs: set current_offset to existing file size when creating fresh index on appended file
  • builder.rs: set_current_offset() method on LineIndexer
  • validate.rs: accept non-zero first offset (for appended captures), reject when byte boundary check fails
  • reader.rs: IndexReader::open now calls validate_index, cross-check content lengths in try_refresh_columnar_offsets
  • file_reader.rs: content-length cross-check at sampled offsets to detect broken indexes

TUI warnings

  • Warning popup on tab open/switch when index is corrupt
  • Persistent status bar warning when viewing a source with corrupt index
  • Stats panel warning in error color

Testing

  • 731 tests pass (13 new), clippy clean
  • New tests for: @ts parsing, TsBounds matching, partition from JSON, standalone @ts queries, broken index detection on refresh, correct appended index acceptance, validate_index offset rejection/acceptance

raaymax added 14 commits March 20, 2026 00:23
Support relative time expressions (now-5m, now-1h30m, now-2d) in
filter values with comparison operators. Timestamp field values are
automatically parsed from ISO 8601, RFC 3339, space-separated
datetime, and epoch seconds/millis formats.
Previously only relative time expressions (e.g., "now-5m") were resolved
to epoch millis for numeric comparison. Absolute timestamps in filter
values (e.g., "2024-01-15T10:00:00Z") fell back to string comparison,
which could produce incorrect results. Now parse_timestamp is used as a
fallback when resolve_relative_time returns None.
- Restrict time resolution to comparison operators only (gt/lt/gte/lte),
  avoiding unnecessary parse_timestamp calls for eq/ne/contains filters
- Remove ambiguous epoch heuristic for non-standard digit lengths;
  only 10-digit (seconds) and 13-digit (millis) are recognized
- Replace format!() allocation in fractional seconds parsing with
  arithmetic normalization to avoid per-line heap allocation
- Shorten MCP search tool description for better LLM comprehension
- Add test for unparseable timestamp field values returning no match
Adds @ts as a virtual field that filters by ingestion timestamp (when
lazytail captured the line) rather than a field in the log payload.
Separated at the AST level and applied as a bitmap at the scan level,
keeping the Filter trait unchanged.

- Parser accepts @ prefix in field names, routes @ts to ts_filters
- TsBounds struct resolves filter values and checks timestamps
- SearchEngine builds @ts bitmap from index, ANDs with flags bitmap
- FilterOrchestrator validates @ts requires indexed file source
- MCP partitions @ts from filters array after JSON deserialization

Usage: json | @ts >= "now-5m" | level == "error"
MCP:   {"field": "@ts", "op": "gte", "value": "now-5m"}
When lazytail -n is run on a file that already has content (append mode),
LineIndexer::create() started current_offset at 0 instead of the file's
current size. This caused all index offsets to point to old content,
producing garbled lines and UTF-8 errors in the TUI.

- Add set_current_offset() to LineIndexer for setting the initial position
- Capture sets current_offset to existing file size on fresh index creation
- validate_index accepts non-zero first offset if preceded by newline
- try_refresh_columnar_offsets cross-checks content lengths at sampled
  offsets to detect broken indexes that pass structural checks by
  coincidence (e.g., old and new lines with similar lengths)
The parser type (json/logfmt) is now optional when the query starts with
@ts filters. A parser can appear later after a pipe separator.

Supported forms:
  @ts >= "now-5m"
  @ts >= "now-1h" | json | level == "error"
  @ts >= "now-1h" | @ts < "now-5m" | logfmt | level == error
Previously @ts filters were silently ignored when no index was available,
causing the query to return all matching lines regardless of time bounds.
Now returns an explicit error.
When an index directory exists but validation fails (e.g., offsets don't
match file content), display a warning in the stats panel instead of
silently falling back to sparse index. The warning tells the user to
delete the .idx/ directory to rebuild.
TsBounds validation (operator + value resolution) now runs in the
filter controller's validate_query, showing errors immediately in the
filter bar as the user types, rather than only on trigger.
IndexReader::open now calls validate_index before loading columns.
Previously it loaded broken indexes without checking, causing the
corrupt index warning to never trigger. Now broken indexes are rejected
at the reader level, surfacing the warning in the TUI stats panel.
When opening or switching to a source with a corrupt index, a centered
popup warns the user and tells them to delete the .idx/ directory.
Dismissed on any key press. Also keeps the warning in the stats panel.
When viewing a source with a corrupt index, the status bar bottom line
shows a bold error message instead of the help text. This ensures the
warning is visible even if the popup was dismissed.
Changed from "delete .idx/ dir to rebuild" to "restart capture to fix"
since deleting the index would lose ingestion timestamps that can't be
reconstructed from the log file.
@raaymax raaymax changed the base branch from master to release-0.10.0 March 26, 2026 17:58
@raaymax raaymax changed the base branch from release-0.10.0 to master March 26, 2026 17:59
@raaymax raaymax changed the title feat(query): add time-based queries for MCP structured search feat(query): time-based queries with @ts virtual field and index integrity fixes Mar 26, 2026
@raaymax raaymax merged commit 8a108e8 into master Mar 26, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant