Skip to content

README.md

Latest commit

 

History

History
249 lines (205 loc) · 8.57 KB

README.md

File metadata and controls

249 lines (205 loc) · 8.57 KB

Hear & There 🚶‍♂️

A smart walking tour generator that creates personalized audio-guided tours using AI based on your location and preferences.

Features

🗺️ Tour Generation

  • Smart location-based tour generation - Uses your GPS location or manual coordinates
  • AI-powered route planning - Google Gemini generates personalized walking tours
  • Multiple tour options - Choose from 3 AI-generated tours with different themes
  • Flexible duration - Tours from 15 minutes to 4+ hours
  • Dynamic radius calculation - POI search adapts to tour duration
  • Real-time walking directions - Google Maps Directions API validates and displays routes
  • Historical and cultural context - City and neighborhood summaries from Gemini

🎙️ Audio Guide Generation

  • AI-generated scripts - Gemini creates engaging narratives for each stop
  • Multi-language support - English and Hebrew audio guides
  • Google Cloud TTS - High-quality text-to-speech with Chirp3-HD voices
  • Automatic byte limit handling - Scripts trimmed to fit TTS 5000-byte limit
  • Parallel audio generation - All audio files generated concurrently
  • Cloud storage - Audio files hosted on Google Cloud Storage

🎧 Tour Player

  • Shareable tour links - Each tour gets a unique URL (/tour/:tourId)
  • Interactive map - Google Maps with walking route visualization
  • Audio playback - Play/pause controls for intro and each stop
  • Walking directions - Step-by-step directions between stops
  • Feedback system - Rate tours and provide feedback
  • Mobile-responsive - Optimized for on-the-go use

🔍 Observability & Performance

  • LangSmith tracing - Complete visibility into AI workflows
  • External API tracing - Google Maps, Places, Directions, and TTS calls tracked
  • Error handling - Automatic fallback from gemini-3-pro-preview to gemini-2.5-pro on rate limits
  • Performance monitoring - Track API latency and bottlenecks
  • Two-tier caching - Tour suggestions and POI data cached for 7 days
  • Cache hit optimization - Discrete duration values (30, 60, 90, 120, 180 min) improve cache hits
  • RediSearch indices - Fast geospatial queries with idx:pois and idx:tour_suggestions

Tech Stack

  • Frontend: React + TypeScript + Tailwind CSS + Vite
  • Backend: Node.js + Express + LangGraph
  • AI Models:
    • Google Gemini 2.5 Flash (tour generation)
    • Google Gemini 3 Pro Preview (audioguide scripts, with fallback to 2.5 Pro)
  • APIs:
    • Google Maps Geocoding API
    • Google Places API (Nearby Search)
    • Google Directions API
    • Google Cloud Text-to-Speech API
  • Storage:
    • Redis Stack with RediSearch (sessions, checkpointing, caching, geospatial queries)
    • Google Cloud Storage (audio files)
  • Deployment:
    • Vercel (frontend)
    • Railway (backend)
  • Monitoring: LangSmith for AI observability and tracing

Quick Start

Prerequisites

  • Node.js 18+
  • Redis Stack 7+ (or Redis with RediSearch module)
  • Google Cloud Project with:
    • Maps JavaScript API enabled
    • Geocoding API enabled
    • Places API (New) enabled
    • Directions API enabled
    • Text-to-Speech API enabled
    • Cloud Storage bucket created
  • Google Gemini API key
  • LangSmith account (optional, for tracing)

Setup

# Clone and install
git clone https://github.com/YOUR_USERNAME/hear-and-there.git
cd hear-and-there

# Backend setup
cd backend && npm install
cp .env.example .env  # Add your API keys

# Frontend setup
cd ../frontend && npm install
cp .env.example .env  # Add Google Maps API key

Environment Variables

Backend (.env)

# Redis
REDIS_URL=redis://localhost:6379

# Google APIs
GOOGLE_MAPS_API_KEY=your_maps_api_key
GEMINI_API_KEY=your_gemini_api_key

# Google Cloud (for TTS and Storage)
GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json
GCS_BUCKET_NAME=your-bucket-name

# AI Models
GEMINI_MODEL_TOUR_GENERATION=gemini-2.5-flash
GEMINI_AUDIOGUIDE_MODEL=gemini-3-pro-preview
INTERESTING_MESSAGES_MODEL=gemini-2.5-flash-lite

# TTS Voice
ENGLISH_VOICE=en-GB-Wavenet-B

# LangSmith (optional)
LANGCHAIN_TRACING_V2=true
LANGCHAIN_API_KEY=your_langsmith_api_key
LANGCHAIN_PROJECT=hear-and-there

# Server
PORT=4000

Frontend (.env)

VITE_GOOGLE_MAPS_API_KEY=your_maps_api_key
VITE_ENGLISH_VOICE=en-GB-Wavenet-B

Run Both Services

# Terminal 1: Start Redis
redis-server

# Terminal 2: Start Backend
cd backend && npm start

# Terminal 3: Start Frontend
cd frontend && npm run dev

Open http://localhost:5173

VS Code Debug

Press F5 and select "Launch Full Stack" to run both backend and frontend with debugging enabled

API Endpoints

Tour Generation

  • GET /health - Health check with version info
  • POST /api/session - Create tour generation session
    • Body: { latitude, longitude, durationMinutes, customization?, language? }
    • Returns: { sessionId, status, city, neighborhood, tours }

Audioguide Generation

  • POST /api/session/:sessionId/tour/:tourId/audioguide - Generate audioguide
    • Body: { language? } (defaults to 'english')
    • Returns: { tourId, status, message }

Tour Player

  • GET /api/tour/:tourId - Get shareable tour data
    • Returns: Complete tour with scripts and audio files
  • GET /api/tour/:tourId/feedback - Get tour feedback
  • POST /api/tour/:tourId/feedback - Submit tour feedback
    • Body: { rating, feedback? }

Architecture

Modular LangGraph Design (v1.0.11)

The backend uses a modular node-based architecture for maintainability and scalability:

backend/
├── tourGeneration.js          # Main graph definition (302 lines)
├── nodes/                     # LangGraph nodes (10 files)
│   ├── cache/                 # Caching layer
│   │   ├── checkCacheForTourSuggestions.js
│   │   ├── checkPoiCache.js
│   │   └── saveTourSuggestionsToCache.js
│   ├── poi/                   # POI discovery
│   │   ├── fetchPoisFromGoogleMaps.js
│   │   └── queryPois.js
│   ├── context/               # Area context building
│   │   ├── reverseGeocode.js
│   │   ├── generateAreaSummaries.js
│   │   └── assembleAreaContext.js
│   └── tours/                 # Tour generation & validation
│       ├── generateCandidateTours.js
│       └── validateWalkingTimes.js
└── utils/                     # Shared utilities (4 files)
    ├── tourState.js           # State definition
    ├── poiHelpers.js          # POI search & caching
    ├── geocodingHelpers.js    # Geocoding & summaries
    └── tourHelpers.js         # Tour generation & validation

Tour Generation Flow (LangGraph)

Workflow:

START → check_cache → [cache hit? → END]
                   ↓ [cache miss]
      check_poi_cache → [40+ POIs? → query_pois]
                     ↓ [< 40 POIs]
      fetch_pois_from_google_maps → query_pois
                                  ↓
      reverse_geocode → generate_area_summaries
                     ↓
      assemble_area_context → generate_candidate_tours
                           ↓
      validate_walking_times → save_to_cache → END

Key Features:

  • Two-tier caching - Tour suggestions (7-day TTL) + POI data (7-day TTL)
  • Discrete durations - Normalized to 30, 60, 90, 120, 180 minutes for better cache hits
  • Parallel execution - Reverse geocoding and area summaries run concurrently
  • Smart routing - Skips cache for customized requests
  • RediSearch indices - Fast geospatial queries for POIs and tours

Audioguide Generation Flow (LangGraph)

  1. fan_out_scripts - Generate intro + stop scripts in parallel with Gemini
  2. fan_in_scripts - Collect all generated scripts
  3. fan_out_audio - Synthesize all audio files in parallel with Google TTS
  4. fan_in_audio - Collect all audio URLs

LangSmith Tracing

All external API calls are wrapped with @traceable:

  • reverseGeocode - Geocoding API
  • searchNearbyPois - Places API orchestration
  • searchPlacesNearby - Individual Places API calls
  • getWalkingDirections - Directions API
  • synthesizeAudio - TTS API + GCS upload

Deployment

Frontend (Vercel)

  • Automatic deployment on push to main
  • Environment variables configured in Vercel dashboard
  • SPA routing configured via vercel.json

Backend (Railway)

  • Automatic deployment on push to main
  • Redis add-on provisioned
  • Environment variables configured in Railway dashboard
  • Google Cloud service account credentials added as secret

Version 1.0.12 - Modular architecture with enhanced caching