Skip to content

Latest commit

 

History

History
334 lines (255 loc) · 9.77 KB

FISHNET_SUMMARY.md

File metadata and controls

334 lines (255 loc) · 9.77 KB

🎣 FishNet - Illegal Fishing Detection System

Overview

FishNet detects illegal fishing activities by analyzing AIS (Automatic Identification System) silence patterns, correlating dark periods with vessel proximity, location, and time to flag suspicious behavior.

🚀 What Was Built

Backend Analysis System (8 Advanced Scripts)

1. Enhanced Dark Event Detection (enhanced_dark_detection.py)

  • Detects AIS gaps > 10 minutes (configurable threshold)
  • Enriches events with metadata:
    • Region classification (EEZ edges, fishing zones)
    • Start/end locations and timestamps
    • Vessel information and fishing gear type
    • Duration calculations

Output Format:

{
  "mmsi": 231982000,
  "start": "2024-01-01T03:00Z",
  "end": "2024-01-01T05:30Z",
  "region": "Eastern Pacific",
  "location": [15.2, -88.0],
  "duration_hours": 2.5,
  "is_fishing_vessel": true
}

2. Vessel Proximity Index (proximity_index.py)

  • Builds "who was near whom and when" dataset
  • Uses BallTree spatial indexing for efficient querying
  • Time-binned analysis (10-minute windows)
  • Distance-based vessel pairing (< 20 km)
  • Haversine distance calculations for accuracy

3. Dark Event Context Checking (dark_event_context.py)

  • Validates dark events by checking nearby vessel transmissions
  • Calculates coverage reliability:
    • More nearby vessels transmitting = higher confidence
    • Identifies if AIS coverage was available
  • Assigns confidence scores (0-1) to each dark event

4. Multi-Factor Suspicion Scoring (suspicion_scoring.py)

Comprehensive scoring system with weighted factors:

Factor Weight Description
Dark gap length 30% Longer gaps = more suspicious
Coverage reliability 20% Low coverage = higher suspicion
EEZ proximity 20% Near borders = higher risk
Fishing vessel proximity 20% Co-occurrence patterns
Repeat offender 10% Frequency of dark events

Also includes:

  • DBSCAN spatial clustering (identifies hotspots)
  • Hexbin aggregation for heatmap visualization

5. Graph-Based Network Analysis (network_analysis.py)

Builds vessel relationship network:

  • Nodes: Vessels with dark events
  • Edges: Co-occurrence during dark periods

Metrics Calculated:

  • Degree centrality: Vessels with many connections
  • Betweenness centrality: Potential coordinators/bridges
  • Closeness centrality: Network position
  • Community detection: Louvain algorithm to find coordinated fleets
  • Transshipment detection: Identifies potential motherships

6. Advanced Visualization (advanced_visualization.py)

Generates comprehensive visualizations:

  • Suspicion heatmaps (geographical distribution by score/frequency)
  • Network graphs (centrality analysis, community structure)
  • Temporal pattern analysis (hourly distribution, duration patterns)
  • Frontend data package (JSON with top events, hotspots, communities)

7. Dataset Integration (dataset_analysis.py)

Cross-references dark events with:

  • Fishing gear types: longlines, trawlers, purse seines, trollers, pole & line, fixed gear
  • Marine protected areas: WDPA database
  • Fleet composition analysis: Multi-gear vessels, flag states

8. Original Analysis Scripts (from Jupyter notebook)

  • data_preprocessing.py - Data loading and cleaning
  • dark_event_detection.py - Basic dark event detection
  • spatial_proximity_analysis.py - KD-tree proximity search
  • pattern_analysis.py - Pattern flagging
  • visualization.py - Basic plots

🔌 REST API (api.py)

Flask API with CORS-enabled endpoints:

Summary & Stats

  • GET /api/summary - Overall statistics
  • GET /api/network/stats - Network metrics

Dark Events

  • GET /api/suspicious-events?limit=50&min_score=0.7&fishing_only=true
  • GET /api/hotspots?limit=20 - Hexbin aggregated hotspots
  • GET /api/clusters?hotspots_only=true - DBSCAN clusters

Network Analysis

  • GET /api/communities?limit=10 - Suspicious vessel communities
  • GET /api/coordinators?limit=20 - High betweenness centrality vessels
  • GET /api/motherships - Potential transshipment vessels

Details & Search

  • GET /api/vessel/{mmsi} - Individual vessel history
  • GET /api/search?lat=15.2&lon=-88.0&radius_km=50 - Spatial/temporal search

🎨 Frontend

Example Dashboard (frontend/example_dashboard.html)

  • Real-time statistics dashboard
  • Suspicious events list with filtering
  • Community network visualization placeholder
  • Map integration ready (Leaflet/Mapbox/deck.gl)
  • Responsive design with glassmorphism UI

📊 Data Flow

1. AIS Data + Fishing Gear CSVs + Protected Areas
         ↓
2. Enhanced Dark Event Detection
         ↓
3. Vessel Proximity Index (BallTree)
         ↓
4. Context Validation (Coverage Check)
         ↓
5. Multi-Factor Suspicion Scoring
         ↓
6. Spatial Clustering (DBSCAN) + Hexbin
         ↓
7. Network Analysis (Graph Theory)
         ↓
8. Visualization Generation
         ↓
9. Frontend Data Package (JSON)
         ↓
10. REST API ← Frontend Dashboard

🚦 Quick Start

1. Install Dependencies

cd backend
pip install -r requirements.txt

2. Run Analysis Pipeline

cd app

# Fast mode (recommended for testing)
python run_pipeline.py

# Full mode (includes proximity index - slower)
python run_pipeline.py --full

3. Start API Server

python api.py
# Server runs on http://localhost:5000

4. Open Frontend

open ../frontend/example_dashboard.html
# or serve with: python -m http.server 8000

📁 Output Files Generated

JSON Data

  • enhanced_dark_events.json - Enriched dark events
  • contextualized_dark_events.json - With coverage context
  • scored_dark_events.json - Suspicion scores
  • dark_zone_clusters.json - DBSCAN clusters
  • dark_zone_hexbins.json - Hexbin heatmap data
  • vessel_communities.json - Network communities
  • centrality_scores.json - Network centrality
  • potential_motherships.json - Transshipment candidates
  • frontend_data_package.json - Main frontend data source
  • dataset_analysis.json - Gear type cross-reference

Visualizations

  • suspicion_heatmap.png - Geographic heatmap
  • network_viz.png - Network analysis plots
  • temporal_analysis.png - Temporal patterns

Network Graph

  • vessel_network.gexf - Graph file (Gephi/Cytoscape compatible)

🔬 Key Algorithms & Techniques

  1. Spatial Indexing: BallTree (scikit-learn) for O(log n) proximity queries
  2. Clustering: DBSCAN for density-based hotspot detection
  3. Network Analysis: NetworkX for graph metrics and community detection
  4. Community Detection: Louvain algorithm for fleet identification
  5. Scoring: Weighted multi-factor model with normalization
  6. Aggregation: Hexagonal binning for heatmap visualization

📈 Performance

  • Small datasets (< 100k records): ~2-5 minutes (full pipeline)
  • Medium datasets (100k-500k): ~10-20 minutes
  • Large datasets (> 500k): Use --fast mode (skips proximity index)

Optimization tips:

  • Proximity index is O(n²) - most expensive step
  • Use sampling for initial testing
  • Run on server with adequate RAM for large datasets

🎯 Use Cases for Frontend

1. Compliance Monitoring

Display high-suspicion vessels for regulatory review

2. Pattern Visualization

Show dark zone hotspots on interactive map (deck.gl, kepler.gl)

3. Network Investigation

Visualize vessel relationships and coordinated fleets

4. Alerts & Notifications

Real-time monitoring of new suspicious events

5. Reporting

Generate compliance reports with evidence packages

🔮 Future Enhancements

Already Implemented ✅

  • Multi-factor scoring
  • Network analysis
  • Spatial clustering
  • REST API
  • Frontend dashboard

Potential Additions 🚧

  1. Real-time streaming: Process AIS data in real-time
  2. Machine learning: Train models on historical patterns
  3. Satellite imagery: Validate dark events with imagery
  4. Weather overlay: Correlate with weather data
  5. EEZ boundaries: Use actual EEZ polygon data
  6. Temporal modeling: Kalman filter for position prediction
  7. Alert system: Email/SMS notifications
  8. Multi-source fusion: Combine with Global Fishing Watch data

📚 Documentation

  • Backend README: /backend/README.md - Complete API documentation
  • Code comments: Each script has detailed docstrings
  • This summary: High-level overview

🏆 What Makes This Special

  1. Comprehensive Pipeline: End-to-end analysis from raw AIS to frontend-ready insights
  2. Advanced Methods: Graph theory, spatial clustering, multi-factor scoring
  3. Production Ready: REST API, modular design, error handling
  4. Well Documented: README, docstrings, example dashboard
  5. Flexible: Fast/full modes, configurable parameters
  6. Dataset Integration: Fishing gear types, protected areas

🤝 Team & Credits

HackHarvard 2024 - Truth & Trust Track (Security)

Project: FishNet - Detecting illegal fishing via AIS silence patterns

🎓 Technical Stack

Backend:

  • Python 3.9+
  • pandas, numpy (data processing)
  • scikit-learn (BallTree, DBSCAN)
  • NetworkX (graph analysis)
  • scipy (spatial algorithms)
  • Flask + Flask-CORS (API)
  • matplotlib, seaborn (visualization)

Frontend:

  • HTML5, CSS3, JavaScript (ES6+)
  • Fetch API for REST calls
  • Ready for: Leaflet, Mapbox, deck.gl integration

📞 Getting Help

  1. Check README: /backend/README.md
  2. API Health: curl http://localhost:5000/api/health
  3. Run Tests: Individual scripts can run standalone
  4. Debug Mode: API runs with debug=True

Status: ✅ Complete and ready for deployment Branch: maureen-fishnet-analysis Last Updated: 2025-10-04