AI-powered semantic search engine for video content — find any moment, instantly.
ChronoView transforms any video — a lecture, meeting, tutorial, or conference talk — into a fully searchable knowledge base using multimodal AI. Type a natural language query and jump to the exact timestamp where that moment occurs. No scrubbing. No guessing. No re-watching hours of content.
- 500+ hours of video are uploaded every minute globally
- There is no "Ctrl+F" for video content
- Students waste hours scrubbing lecture recordings
- Enterprises lose $37B/year to unsearchable meeting recordings
- Existing tools match keywords — not meaning
ChronoView processes three parallel data streams from every video:
| Stream | Model | Output |
|---|---|---|
| 🎙️ Speech | OpenAI Whisper | Timestamped transcripts |
| 👁️ Visual | Vision Transformer (ViT) | Scene embeddings |
| 📄 On-screen text | Tesseract OCR | Slide & code text |
All three streams are fused using a CLIP-inspired contrastive learning model into a unified semantic embedding — stored in a FAISS vector database for millisecond-speed retrieval.
- 🔍 Semantic Search — natural language query → exact timestamp
- 🧠 Direct Q&A — AI-generated answers extracted from the video
- 📚 Auto-Chapters — AI-generated titled navigation segments
- 🌍 Multilingual Search — query in any language
- 🔗 Shareable Timestamp Links — share exact video moments
- 📊 Engagement Heatmaps — analytics on which segments were searched most
- ✂️ Highlight Reel Export — compile relevant segments into a short clip
Video Input
│
▼
FFmpeg (segment splitting · keyframe extraction · audio strip)
│
├──────────────────────────────────────┐─────────────────────┐
▼ ▼ ▼
Whisper ASR Tesseract OCR ViT Model
(Speech → text) (Slides & code text) (Scene encoding)
│ │ │
└──────────────────────────────────────┘─────────────────────┘
│
▼
CLIP Fusion Model (PyTorch)
Unified semantic embedding space
│
▼
FAISS / ChromaDB
Vector similarity index
│
▼
FastAPI Backend (REST API)
│
▼
React Dashboard + Video Player
| Layer | Technology |
|---|---|
| Video Processing | FFmpeg |
| Speech Recognition | OpenAI Whisper |
| Scene Understanding | Vision Transformer (ViT) |
| OCR | Tesseract |
| Semantic Fusion | CLIP (PyTorch + HuggingFace) |
| Vector Search | FAISS / ChromaDB |
| Backend API | FastAPI |
| Frontend | React / Streamlit |
| Analytics | Plotly |
| Storage | AWS S3 / Google Cloud Storage |
| Containerization | Docker |
Python 3.10+
NVIDIA GPU (RTX 3060 or higher recommended)
CUDA 11.8+
Node.js 18+ (for React frontend)
FFmpeg installed on system# Clone the repository
git clone https://github.com/yourusername/chronoview.git
cd chronoview
# Create virtual environment
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
# Install frontend dependencies
cd frontend
npm install
cd ..# Copy environment template
cp .env.example .env
# Add your keys in .env
OPENAI_WHISPER_MODEL=base
HUGGINGFACE_TOKEN=your_token_here
AWS_ACCESS_KEY=your_key_here
AWS_SECRET_KEY=your_secret_here# Step 1 — Index a video
python pipeline/index_video.py --input your_video.mp4
# Step 2 — Start the backend
uvicorn app.main:app --reload --port 8000
# Step 3 — Start the frontend
cd frontend && npm run devOpen http://localhost:3000 in your browser.
chronoview/
│
├── pipeline/
│ ├── ingest.py # FFmpeg video segmentation
│ ├── transcribe.py # Whisper ASR
│ ├── ocr.py # Tesseract OCR
│ ├── vision.py # ViT scene encoding
│ ├── fuse.py # CLIP fusion model
│ └── index.py # FAISS vector indexing
│
├── app/
│ ├── main.py # FastAPI entry point
│ ├── search.py # Query embedding + retrieval
│ ├── qa.py # Direct Q&A generation
│ └── analytics.py # Heatmap + usage analytics
│
├── frontend/
│ ├── src/
│ │ ├── pages/ # Home, Results, Library, Analytics
│ │ └── components/ # SearchBar, ResultCard, VideoPlayer
│ └── package.json
│
├── models/ # Saved model checkpoints
├── tests/ # Unit and integration tests
├── docker-compose.yml
├── requirements.txt
└── README.md
POST /api/search
Content-Type: application/json
{
"query": "explain gradient descent",
"video_id": "cs229_lecture4",
"top_k": 5
}Response:
{
"query": "explain gradient descent",
"ai_answer": "Gradient descent minimizes loss by...",
"results": [
{
"timestamp": "14:32",
"title": "Gradient descent intuition",
"snippet": "...learning rate alpha controls step size...",
"confidence": 0.97,
"sources": ["audio", "slide"]
}
]
}| # | Paper | Venue |
|---|---|---|
| 1 | Radford et al. — CLIP (2021) | ICML 2021 |
| 2 | Radford et al. — Whisper (2022) | arXiv:2212.04356 |
| 3 | Dosovitskiy et al. — ViT (2020) | ICLR 2021 |
| 4 | Johnson et al. — FAISS (2019) | IEEE Trans. Big Data |
| 5 | Liu et al. — Video Moment Localization (2023) | ACM Computing Surveys |
| Sector | Use Case |
|---|---|
| 🎓 Education | Students search lecture recordings by concept |
| 🏢 Enterprise | Teams retrieve decisions from meeting archives |
| 🔬 Research | Scientists index conference talks and webinars |
| 🧑💻 Developers | Search coding tutorials for exact implementations |
| ♿ Accessibility | Semantic index for hearing-impaired users |
- Multimodal pipeline (Whisper + ViT + OCR)
- CLIP-based semantic fusion
- FAISS vector indexing
- FastAPI search endpoint
- React dashboard
- Cross-video search across entire libraries
- Highlight reel export
- Mobile app
- Enterprise SSO integration
- Fine-tuned domain-specific embedding model
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
# Run tests
pytest tests/
# Format code
black pipeline/ app/MIT License — see LICENSE for details.
Tanmay Built for hackathon — "Skip to the Good Part."
ChronoView does for video what Google did for the web.