Real-time Tokyo transportation data collection and visualization system using ODPT API
Interactive real-time visualization of Tokyo's transportation network
- 🗺️ Interactive Tokyo Station Map with real-time markers
- 📈 Live Charts & Metrics showing passenger flow and performance
- 🚂 Real-time Train Tracking with delay status
- 🚌 Bus Route Monitoring with live positions
- 📱 Mobile Responsive design for all devices
- ⚡ Auto-refresh with 30-second data updates
🔴 LIVE: Tokyo Transportation Monitor
├── Daily Passengers: 2.15M across 5 major stations
├── System Performance: 94.2% punctuality rate
├── Average Delay: 1.0 minutes
└── Active Alerts: 0 service disruptions
- Python 3.8+
- Git
- Ruby 3.1+ (for Jekyll documentation)
# Clone the repository
git clone https://github.com/Tatsuru-Kikuchi/MCP-traffic.git
cd MCP-traffic
# Set up Python environment
python3 -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install Python dependencies
pip install -r requirements.txt
# Set up Jekyll for documentation
bundle install
# Configure the system
cp config/api_config.example.json config/api_config.json
# Edit config/api_config.json with your ODPT API settings# Test the installation
python src/collectors/traffic_collector.py --catalog-only
# Run a single data collection
python scripts/collect_all_data.py
# Start continuous scheduled collection
python scripts/schedule_collection.py
# Serve documentation locally
bundle exec jekyll serveA comprehensive Tokyo traffic data collection and analysis system that leverages the ODPT (Open Data Platform for Transportation) API to gather, process, and monitor real-time transportation data across Tokyo's extensive public transit network.
MCP Traffic (Model Control Protocol for Traffic) is an automated system designed for:
- 🔄 Real-time Data Collection: Continuous gathering of Tokyo transportation data
- ⚡ Data Processing: Automated cleaning, validation, and transformation
- 📈 Analysis & Monitoring: Traffic pattern analysis and system health monitoring
- 🗄️ Data Management: Organized storage with automatic archiving
- 📚 Documentation: Comprehensive guides and API documentation
ODPT API → Data Collectors → Data Processors → Storage → Analysis
↓ ↓ ↓ ↓ ↓
Raw Data → Validation → Transformation → Archive → Insights
MCP-traffic/
├── src/ # Source code
│ ├── collectors/ # Data collection modules
│ └── processors/ # Data processing utilities
├── data/ # Data storage
│ ├── raw/ # Original API responses
│ ├── processed/ # Cleaned data
│ └── archives/ # Historical data
├── scripts/ # Automation scripts
├── config/ # Configuration files
├── monitoring/ # Health checks and alerts
├── logs/ # System logs
├── docs/ # Documentation
└── .github/workflows/ # CI/CD workflows
Edit config/api_config.json:
{
"odpt_api": {
"base_url": "https://ckan.odpt.org/api/3/action/",
"api_key": "your-api-key",
"rate_limit": 100,
"timeout": 30
},
"collection": {
"schedule_interval": 300,
"data_types": ["train", "bus", "station"],
"regions": ["tokyo"]
},
"storage": {
"format": "json",
"compression": true,
"retention_days": 365
}
}The documentation site is configured in _config.yml with:
- GitHub Pages deployment
- SEO optimization
- Responsive design
- Multi-language support
- 🚆 Train Data: Real-time positions, delays, service status
- 🚌 Bus Data: Vehicle locations, route information, schedules
- 🏢 Station Data: Facility information, accessibility features
- 🗺️ Route Data: Network topology, connections, fare information
- 📊 Operational Data: Service disruptions, maintenance schedules
- Scheduled Collection: Automated data gathering at configurable intervals
- On-demand Collection: Manual collection for specific data types
- Error Recovery: Robust retry mechanisms with exponential backoff
- Rate Limiting: Compliance with API usage policies
# Standard Python development setup
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt
python scripts/collect_all_data.py# Build and run with Docker Compose
docker-compose up -d
# Monitor logs
docker-compose logs -f mcp-traffic# Install as systemd service
sudo cp deployment/mcp-traffic.service /etc/systemd/system/
sudo systemctl enable mcp-traffic
sudo systemctl start mcp-traffic- AWS: EC2 + S3 + CloudWatch
- Google Cloud: Compute Engine + Cloud Storage
- Azure: Virtual Machines + Blob Storage
See the Deployment Guide for detailed instructions.
- API Connectivity: Continuous monitoring of ODPT API availability
- Data Quality: Validation and completeness checks
- Storage Usage: Disk space monitoring with alerts
- Performance Metrics: Response times and throughput tracking
# System health overview
python monitoring/health_check.py
# Detailed performance metrics
python monitoring/performance_monitor.py
# Data quality report
python monitoring/data_quality_check.py- Email notifications for system failures
- Slack integration for real-time alerts
- Custom webhook support for external monitoring
- Traffic pattern analysis
- Peak hour identification
- Service reliability metrics
- Historical trend analysis
- Jupyter Notebooks: Interactive data exploration
- Pandas/NumPy: Statistical analysis
- Matplotlib/Plotly: Data visualization
- Apache Spark: Large-scale data processing
- 📊 Live Dashboard - Interactive Tokyo traffic visualization
- 📖 Dashboard User Guide - Complete tutorial and help
- 🌐 Project Website - Main documentation site
- Deployment Guide: Complete deployment instructions
- API Documentation: API reference and examples
- Configuration Guide: System configuration options
- Troubleshooting: Common issues and solutions
- No Personal Data: Only collects aggregate transportation data
- Anonymized Data: All data is inherently anonymous
- Compliance: Adheres to data protection regulations
- API key encryption
- Secure configuration management
- Network security best practices
- Regular security updates
We welcome contributions! Here's how to get started:
# Fork and clone the repository
git clone https://github.com/YOUR-USERNAME/MCP-traffic.git
cd MCP-traffic
# Create development branch
git checkout -b feature/your-feature-name
# Set up development environment
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt
pip install -r requirements-dev.txt
# Run tests
python -m pytest tests/- Code Style: Follow PEP 8 for Python, use meaningful variable names
- Testing: Add tests for new functionality
- Documentation: Update documentation for changes
- Commits: Use clear, descriptive commit messages
- 🐛 Bug Fixes: Report and fix issues
- ✨ Features: New functionality and improvements
- 📚 Documentation: Improve guides and examples
- 🧪 Testing: Enhance test coverage
- 🔧 DevOps: Improve deployment and CI/CD
This project is licensed under the MIT License - see the LICENSE file for details.
- Documentation: Check the docs/ directory
- Issues: Search existing issues
- Discussions: Join GitHub Discussions
- Dashboard Help: Read the User Guide
When reporting issues, please include:
- Operating system and Python version
- Error messages and stack traces
- Steps to reproduce the issue
- Configuration (without sensitive data)
- ODPT (Open Data Platform for Transportation): Primary data source
- Tokyo Metropolitan Government: Open data initiatives
- Transportation Operators: Real-time operational data
- Python: Core programming language
- Jekyll: Documentation site generator
- GitHub Actions: CI/CD and deployment
- Docker: Containerization
- Chart.js: Interactive data visualization
- Leaflet: Interactive mapping
Special thanks to all contributors who have helped improve this project through code, documentation, testing, and feedback.
For more information, visit the project website or check out the live dashboard!