feat(query): time-based queries with @ts virtual field and index integrity fixes#88
Merged
feat(query): time-based queries with @ts virtual field and index integrity fixes#88
Conversation
Support relative time expressions (now-5m, now-1h30m, now-2d) in filter values with comparison operators. Timestamp field values are automatically parsed from ISO 8601, RFC 3339, space-separated datetime, and epoch seconds/millis formats.
Previously only relative time expressions (e.g., "now-5m") were resolved to epoch millis for numeric comparison. Absolute timestamps in filter values (e.g., "2024-01-15T10:00:00Z") fell back to string comparison, which could produce incorrect results. Now parse_timestamp is used as a fallback when resolve_relative_time returns None.
- Restrict time resolution to comparison operators only (gt/lt/gte/lte), avoiding unnecessary parse_timestamp calls for eq/ne/contains filters - Remove ambiguous epoch heuristic for non-standard digit lengths; only 10-digit (seconds) and 13-digit (millis) are recognized - Replace format!() allocation in fractional seconds parsing with arithmetic normalization to avoid per-line heap allocation - Shorten MCP search tool description for better LLM comprehension - Add test for unparseable timestamp field values returning no match
Adds @ts as a virtual field that filters by ingestion timestamp (when lazytail captured the line) rather than a field in the log payload. Separated at the AST level and applied as a bitmap at the scan level, keeping the Filter trait unchanged. - Parser accepts @ prefix in field names, routes @ts to ts_filters - TsBounds struct resolves filter values and checks timestamps - SearchEngine builds @ts bitmap from index, ANDs with flags bitmap - FilterOrchestrator validates @ts requires indexed file source - MCP partitions @ts from filters array after JSON deserialization Usage: json | @ts >= "now-5m" | level == "error" MCP: {"field": "@ts", "op": "gte", "value": "now-5m"}
When lazytail -n is run on a file that already has content (append mode), LineIndexer::create() started current_offset at 0 instead of the file's current size. This caused all index offsets to point to old content, producing garbled lines and UTF-8 errors in the TUI. - Add set_current_offset() to LineIndexer for setting the initial position - Capture sets current_offset to existing file size on fresh index creation - validate_index accepts non-zero first offset if preceded by newline - try_refresh_columnar_offsets cross-checks content lengths at sampled offsets to detect broken indexes that pass structural checks by coincidence (e.g., old and new lines with similar lengths)
Previously @ts filters were silently ignored when no index was available, causing the query to return all matching lines regardless of time bounds. Now returns an explicit error.
When an index directory exists but validation fails (e.g., offsets don't match file content), display a warning in the stats panel instead of silently falling back to sparse index. The warning tells the user to delete the .idx/ directory to rebuild.
TsBounds validation (operator + value resolution) now runs in the filter controller's validate_query, showing errors immediately in the filter bar as the user types, rather than only on trigger.
IndexReader::open now calls validate_index before loading columns. Previously it loaded broken indexes without checking, causing the corrupt index warning to never trigger. Now broken indexes are rejected at the reader level, surfacing the warning in the TUI stats panel.
When opening or switching to a source with a corrupt index, a centered popup warns the user and tells them to delete the .idx/ directory. Dismissed on any key press. Also keeps the warning in the stats panel.
When viewing a source with a corrupt index, the status bar bottom line shows a bold error message instead of the help text. This ensures the warning is visible even if the popup was dismissed.
Changed from "delete .idx/ dir to rebuild" to "restart capture to fix" since deleting the index would lose ingestion timestamps that can't be reconstructed from the log file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
now-5m,now-1h30mand absolute timestamps are resolved at query time for comparison operators on any field@tsvirtual field: filters by index ingestion timestamp (when lazytail captured the line) rather than a log payload field. Works as a bitmap at the scan level — no changes to the Filter traitUsage
Changes
Time-based queries
time.rs: relative time resolution (now-5m), timestamp parser (RFC 3339, ISO 8601, epoch s/ms),TsBoundsstruct for@tsbitmap evaluationfilter.rs: pre-resolve time values atQueryFilterconstruction for comparison operatorsparser.rs: accept@prefix in field names, route@tstots_filters, optional parser prefix (@ts >= "now-5m"works withoutjson |)ast.rs:ts_filtersfield onFilterQuery,partition_ts_filters()for MCP JSON pathsearch_engine.rs: build@tsbitmap from index timestamps, AND with flags bitmapfilter_orchestrator.rs: validate@tsrequires indexed file sourcefilter_controller.rs: validate@tsvalues during live filter previewIndex integrity
capture.rs: setcurrent_offsetto existing file size when creating fresh index on appended filebuilder.rs:set_current_offset()method onLineIndexervalidate.rs: accept non-zero first offset (for appended captures), reject when byte boundary check failsreader.rs:IndexReader::opennow callsvalidate_index, cross-check content lengths intry_refresh_columnar_offsetsfile_reader.rs: content-length cross-check at sampled offsets to detect broken indexesTUI warnings
Testing
@tsparsing,TsBoundsmatching, partition from JSON, standalone@tsqueries, broken index detection on refresh, correct appended index acceptance, validate_index offset rejection/acceptance