Skip to content

PROJECT_TIMELINE.md

Latest commit

 

History

History
134 lines (107 loc) · 6 KB

PROJECT_TIMELINE.md

File metadata and controls

134 lines (107 loc) · 6 KB

Project Development Timeline

Phase 1: Infrastructure Setup and Ingestion Pipeline

December 2025

Infrastructure Setup

  • Technologies Chosen:
    • Spring Boot 4.0: Application framework.
    • Kafka: Message streaming platform.
    • Redis: For future indexing.
    • Uber H3: Planned for geospatial indexing.
    • Lombok: To reduce boilerplate code.
  • Docker Compose: Configured to manage services.

Ingestion Pipeline Implementation

  • Objective: Create a REST API to receive driver locations and send them to Kafka.
  • Files Created:
    • DriverLocation.java: Represents driver location data.
    • LocationProducer.java: Sends driver locations to Kafka.
    • DriverController.java: REST API for receiving driver locations.
  • Configuration:
    • Kafka serialization configured in application.properties.
    • Added Jackson dependency to pom.xml to resolve serialization issues.

Testing and Validation

  • REST API tested successfully.
  • Verified that messages were published to Kafka with integrity.
  • Simulated multiple driver locations to ensure robustness.

Outcome:

Phase 1 (ingestion pipeline) completed successfully. The system is ready to proceed to Phase 2 (indexing).

Phase 2: Indexing and Matching Engine

December 2025 (Continued)

Indexer Implementation (Kafka Consumer)

  • Objective: Listen to driver location updates and index them in Redis using H3.
  • Files Created:
    • LocationIndexer.java: Kafka consumer that processes driver locations.
    • GeoService.java: Service for all H3 geospatial operations.
    • DriverLocationRepository.java: Redis operations for driver indexing.
  • Implementation Details:
    • Kafka consumer listens to driver-locations topic.
    • Converts driver lat/lon to H3 hexagon cells (resolution 8).
    • Stores driver IDs in Redis Sets by H3 cell key (h3:{cellId}).
    • Applied 30-second TTL to auto-expire stale driver locations.

Matching Service Implementation (Expanding Ring Algorithm)

  • Objective: Find nearest available driver for rider requests.
  • Files Created:
    • MatchingService.java: Core matching logic with expanding ring algorithm.
    • RideRequest.java: Model for rider match requests.
    • MatchResponse.java: Response model with match results.
    • RiderController.java: REST API for riders to request matches.
  • Algorithm:
    • Ring 0: Check exact H3 cell where rider is located (1 cell).
    • Ring 1: Expand to 6 immediate neighboring cells (7 cells total).
    • Ring 2: Expand to 12 more neighboring cells (19 cells total).
    • Returns first available driver or "no drivers available" message.
  • API Endpoints:
    • POST /api/rides/match - Request driver match with JSON body.
    • GET /api/rides/match?lat=X&lon=Y - Simple GET for testing.

Testing and Validation

  • Driver locations successfully indexed in Redis with H3 cells.
  • Verified expanding ring algorithm:
    • Ring 0: Exact cell matches working.
    • Ring 1: Neighboring cell matches working.
    • Ring 2: Extended search working.
  • Confirmed TTL expiration (30 seconds) auto-removes stale drivers.
  • End-to-end flow validated: Driver location → Kafka → Redis indexing → Rider match.

Outcome:

Phase 2 (indexing and matching) completed successfully. The system can now match riders to nearby drivers efficiently.

Phase 2.5: Production-Grade Improvements

December 2025 (Refinements)

Problem 1: Race Condition (Multiple Riders, Same Driver)

  • Issue: Using Redis SMEMBERS allowed two riders to see the same driver simultaneously.
  • Solution: Replaced getDriversInCell() with claimDriverFromCell() using atomic SPOP.
  • Result: Driver assignment is now mutually exclusive at the database level.
  • Test Proof: 3 drivers → 3 concurrent riders = 3 unique matches (no duplicates).

Problem 2: H3 Resolution Inconsistency

  • Issue: If indexer and matcher use different H3 resolutions, they'll never find matches.
  • Solution:
    • Centralized resolution configuration in application.properties.
    • Added getResolution() method to GeoService.
    • Added resolution logging to verify consistency.
  • Result: All operations confirmed using resolution 8.

Problem 3: Error Handling

  • Issue: Using Optional.empty() required null checks and didn't provide context.
  • Solution:
    • Created NoDriverAvailableException with location context.
    • Updated MatchingService to throw exception instead of returning empty.
    • Updated RiderController with try-catch for graceful responses.
  • Result: Clean error handling with proper JSON responses and detailed logging.

Key Code Improvements:

  • DriverLocationRepository: Added claimDriverFromCell() for atomic Redis SPOP.
  • MatchingService: Changed from Optional<String> to String with exception handling.
  • GeoService: Added resolution getter and documentation on consistency importance.

Journey Point #19 (Blog-Worthy Learning):

"I realized that a 'working' system isn't always a 'correct' system. To prevent race conditions where two riders are assigned the same driver, I swapped my Redis lookup for an atomic SPOP operation. This ensures that driver assignment is mutually exclusive at the database level. I also standardized my H3 resolution across the stack and implemented custom exception handling to manage 'No Match' scenarios gracefully—moving the project from a happy-path demo to a resilient microservice."

Outcome:

Phase 2 is now production-ready with proper concurrency handling, consistent configuration, and graceful error management.

Next Steps

Phase 3: Advanced Features (Future)

  • Real-time driver status tracking (online/busy/offline).
  • Distance-based ranking (not just first-found).
  • Driver rating and preference considerations.
  • Load balancing across multiple app instances.
  • Metrics and monitoring (Prometheus/Grafana).
  • Performance testing at scale.

Last Updated: December 17, 2025
Status: Phase 2 Complete & Production-Ready 🚀