0.4.0

[soulgarden]

Summary by CodeRabbit

Release Notes v0.4.0

• New Features

  • Atomic event handling eliminates file processing race conditions
  • Dead Letter Queue with automatic retry (up to 5 attempts by default)
  • Modern logging for improved observability

• Improvements

  • Enhanced event delivery reliability with automatic backoff during shutdown
  • Better backpressure handling when sending to Elasticsearch

• Requirements

  • Minimum Rust version increased to 1.91+

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Caution

Review failed

The pull request is closed.

Walkthrough

This PR introduces atomic event handling mechanisms across multiple system components. It bumps the version from 0.3.0 to 0.4.0, removes the increment-version Makefile dependency to enable Docker tag pushing, updates documentation, adds Dead Letter Queue retry logic with configurable max retry counts and batch operations, implements a DLQ background retry task with exponential backoff, refactors sender retry behavior using a 50-attempt loop instead of fixed timeouts, extends bounded channels with non-blocking send capability, and refactors file watcher event handling to use atomic collection patterns that eliminate race conditions.

Changes

Cohort / File(s)	Summary
Version & Build Makefile, VERSION	Removed increment-version dependency from docker-build target; version bumped from 0.3.0 to 0.4.0; Docker build now prints version, tags image with two tags, and pushes both $NEW_VERSION and latest.
Documentation README.md	Updated version/test badges to 0.4.0 with 293 tests; replaced "Lock Optimization" with "Atomic Event Handling"; added "Modern Logging" in Architecture section; increased Rust toolchain requirement to 1.91+.
Dead Letter Queue System src/infrastructure/elasticsearch/dead_letter_queue.rs, src/infrastructure/elasticsearch/pool.rs	Added max_retry_count configuration field (default 5) to DeadLetterQueueConfig; implemented take_batch() for FIFO batch retrieval, return_failed() for conditional re-queueing with retry counting, and mark_recovered() for recovery tracking. Pool now stores Settings configuration and introduces start_dlq_retry_task() background task that periodically retries DLQ batches with exponential backoff and timeout handling.
Channel Infrastructure src/transport/channels/bounded.rs, src/transport/bridge/event_bridge.rs	Added try_send() non-blocking method to BoundedSender; extended SendError enum with Full(T) variant for full-channel scenarios. EventBridge now handles SendError::Full with warning log.
Event Delivery & Processing src/sender.rs, src/watcher.rs	Sender refactored to use 50-attempt retry loop (100ms delays) instead of fixed timeout; send_events_to_es() now returns Option<Vec<Event>> for failed events; preserves failed events at batch front on partial failure. Watcher introduces MAX_BATCH_SIZE constant and atomic collection functions handle_create_event() and handle_remove_event() that eliminate race conditions by collecting events under lock before release.

Sequence Diagram

sequenceDiagram
    participant WP as EsWorkerPool
    participant DLQ as DeadLetterQueue
    participant ES as Elasticsearch
    participant Backoff as Backoff Strategy

    Note over WP,Backoff: DLQ Retry Background Task (Periodic)
    WP->>DLQ: take_batch(limit)
    activate DLQ
    DLQ-->>WP: batch of failed events
    deactivate DLQ

    WP->>WP: build ndjson bulk payload
    WP->>Backoff: get current retry interval

    rect rgba(100, 150, 200, 0.2)
        Note over WP,ES: POST with Backoff & Timeout Handling
        WP->>ES: POST bulk request
        alt Success
            ES-->>WP: 200 OK
            WP->>DLQ: mark_recovered(count)
            WP->>Backoff: reset to base interval
        else Timeout or Error
            WP-->>WP: increment backoff interval
            WP->>DLQ: return_failed(events)
            activate DLQ
            DLQ->>DLQ: retry_count++ per event<br/>re-queue if retry_count ≤ max
            deactivate DLQ
        end
    end

Loading

sequenceDiagram
    participant FS as Filesystem
    participant W as Watcher
    participant ST as AppState Lock
    participant CH as Bounded Channel

    Note over W: Atomic Create Event Handling
    FS->>W: file created
    activate W
    W->>ST: acquire lock (single operation)
    W->>W: create FileTracker
    W->>W: collect historical events<br/>(respect MAX_BATCH_SIZE)
    W->>ST: release lock
    deactivate W
    W->>W: insert tracker into file_trackers
    W->>CH: send collected events batch

    Note over W: Atomic Remove Event Handling
    FS->>W: file removed
    activate W
    W->>W: remove tracker from file_trackers
    W->>ST: acquire lock (read remaining)
    W->>W: collect remaining content<br/>(respect MAX_BATCH_SIZE)
    W->>ST: close tracker & cleanup
    W->>ST: release lock
    deactivate W
    W->>CH: send remaining events batch

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Areas requiring extra attention:

• src/infrastructure/elasticsearch/dead_letter_queue.rs: New retry counting logic with conditional re-queueing based on max_retry_count; batch operations and statistics tracking must be validated for correctness under concurrent access.
• src/infrastructure/elasticsearch/pool.rs: Background retry task with exponential backoff and timeout handling; HTTP client integration and error propagation paths need careful review.
• src/sender.rs: Return type change affects control flow; 50-attempt retry loop logic and shutdown sequence modifications require verification that events are not lost during backpressure scenarios.
• src/watcher.rs: New atomic collection patterns and MAX_BATCH_SIZE enforcement; race condition elimination in file creation/removal paths must be validated against the prior lock-and-send pattern.

Possibly related PRs

• - Decoupling: Split into domain/transport/infrastructure/service laye… #8: Modifies Makefile's docker-build target to read VERSION and build/push multi-tagged Docker images with the same runtime tagging strategy.
• Enhance logging configuration - Introduced structured logging with cu… #9: Updates Makefile's docker-build target to use NEW_VERSION at runtime and adjust prerequisite dependencies similarly.
• Major architectural improvements: #6: Contains overlapping code-level changes to dead-letter queue, ES worker pool, bounded channels, sender, and watcher across the same core components.

Poem

🐰 Hops through the queue with mighty grace,
Atomic locks now set the pace,
Batches flow where race-conditions dwelled,
With backoff's dance, resilience spelled!
Lock, collect, release—oh what a sight,
Dead letters dance from wrong to right! ✨

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch 0.4.0

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c7a6de5 and 5a04899.

📒 Files selected for processing (9)

• Makefile (1 hunks)
• README.md (7 hunks)
• VERSION (1 hunks)
• src/infrastructure/elasticsearch/dead_letter_queue.rs (6 hunks)
• src/infrastructure/elasticsearch/pool.rs (3 hunks)
• src/sender.rs (4 hunks)
• src/transport/bridge/event_bridge.rs (1 hunks)
• src/transport/channels/bounded.rs (2 hunks)
• src/watcher.rs (5 hunks)

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}