tennis-vision

End-to-end tennis broadcast analysis. Takes a 720p or 1080p match clip, produces an annotated overlay (player skeletons, ball trace, court overlay, mini-court bounce heatmap, rolling rally event panel, OCR-grounded scoreboard, per-player stats) and structured JSON sidecars covering every shot, rally, and scoring state.

Two-phase pipeline. A patched fork of yastrebksv/TennisProject handles ball, court, and person detection on GPU (Colab T4/L4) and writes a single upstream.npz artifact. Phase 2 runs locally on CPU: pose estimation, shot detection (rule-based or F3ED), rally segmentation, scoreboard OCR, score-delta reconciliation, and the overlay render. Iterating on Phase 2 logic never re-runs the GPU detection pass.

The score-delta reconciler is the project's main contribution: it pairs F3ED's shot output with OCR-derived score states to recover events F3ED has no class for (ace, double_fault, first_serve_fault). See docs/blog/F3ED-OCR-finding.md for the audit and finding.

What it outputs

Artifact	Contents
overlay.mp4	h264-encoded broadcast video with skeletons, ball trail, court overlay, bounce heatmap on minimap, rolling rally panel (F3ED labels with direction + outcome), scoreboard echo, per-player W/UE/FE/ACE stats
pose.parquet	Dense per-player, per-frame, per-keypoint pose table (zstd-compressed, long format)
shots.json	Event list per shot: frame, player, shot label (rule-based or F3ED raw_elements), contact in px + meters, ball speeds in/out (km/h), bounce linkage, OCR-corrected outcome (outcome_corrected)
rallies.json	Rally segments with server, winner, method, last-bounce coordinates, outcome kind
scoreboard.json	Sampled OCR readings (~1 Hz): names, games, points, derived server
game.json	Reconciled rallies with score-delta-grounded outcomes + scoring state timeline

Pipeline

Phase 1 (Colab GPU, ~15 min on L4 for a 20-min 720p clip)
  video.mp4 ──► TrackNetV2 (ball)        ──┐
              ── Court keypoint net ──────┤
              ── YOLOv8x (person)  ───────┼──► upstream.npz
              ── CatBoost (bounce) ───────┘     { ball_track, homography_matrices,
                                                  kps_court, persons_top/bottom,
                                                  bounces, scenes, fps, sha256 }

Phase 2 (local CPU, ~10 min on M-series for the same clip)
  upstream.npz ──► pose             (RTMPose-m)
                ── events           (F3ED or rule-based shot detector)
                ── rallies          (gap-segmenter + bounce-projection winner rule)
                ── scoreboard       (EasyOCR + tennis-grammar decoder)
                ── reconcile        (score-delta classifier; emits outcome_corrected)
                ── render           (h264 via ffmpeg subprocess)

Quick start

uv sync --extra dev

# 1. Upload scripts/colab_run.ipynb to Colab, Runtime → GPU, Run all.
#    Replace input_video path in cell 14 with your match clip on Drive.
#    Downloads upstream.npz when complete.

# 2. Place upstream.npz at outputs/<slug>/upstream.npz, then run Phase 2:
.venv/bin/python scripts/batch_process.py configs/example_manifest.yaml --only <slug> --force

# 3. Individual stages (when iterating on a single piece):
uv run tennis-vision pose       <upstream.npz> <out.parquet>           --video <local.mp4>
uv run tennis-vision events     <upstream.npz> <pose.parquet> <shots.json> --detector f3ed --f3set-src <path>
uv run tennis-vision scoreboard <video.mp4> <out.json>                  --layout split_open_1080p
uv run tennis-vision render     <upstream.npz> <pose.parquet> <shots.json> <overlay.mp4> \
    --video <local.mp4> --style full --scoreboard <scoreboard.json>

scripts/batch_process.py configs/example_manifest.yaml orchestrates all six Phase-2 stages and skips ones whose outputs already exist (override with --force).

Architecture

• Pose — RTMPose-m (COCO-17, 256×192) via rtmlib ONNX runtime. Top-down, on upstream's player bboxes. Smoothed with 1€ filter + short-gap linear interpolation (8-frame max gap).
• Shot detection — Two interchangeable detectors selected per match in the manifest:
  • F3ED (NeurIPS 2024): 29-class multi-label per shot covering player half, court side, category (serve/return/stroke), wing (fh/bh), technique (gs/slice/volley/drop/lob), direction (T/B/W/CC/DL/DM/II/IO), and outcome (in/winner/forced-err/unforced-err). Best F1 of the open baselines on TenniSet V006 (0.54).
  • Rule-based: pose+ball heuristic with sigmoid(wrist_speed_z) × exp(-ball_wrist_dist / 120px) × clip(ball_dir_change / 90°). F1=0.824 on a hand-labeled fixture; doesn't surface direction or outcome.
• Bounce filtering — CatBoost over-fires bounces on broadcast tennis. Two filters trim 21-27% of false bounces: temporal dedup (~400 ms minimum separation, fps-aware) and court-locality (drops bounces whose ball pixel projects outside the court polygon + 200 px buffer). See src/tennis_vision/events/player_tracker.py:mask_ball_track_to_broadcast.
• Scoreboard OCR — EasyOCR cropping a per-layout ROI (split_open_1080p, split_open_720p, bloomfield_720p), then a tennis-grammar decoder that rejects illegal score transitions and majority-votes within a sample window.
• Score-delta reconciler — For each single-shot serve rally, looks at the OCR scoreboard before and after the rally. If the server's points went up: ace. Receiver's: double_fault. Neither: first_serve_fault. Writes outcome_corrected onto the rally's last shot. ~30 lines, microseconds per rally. (src/tennis_vision/scoreboard/reconcile.py)
• Render — h264 via ffmpeg subprocess (5-6× smaller files than the legacy mp4v fourcc, smooth scrub everywhere). Composed back-to-front: court overlay → ball trail → skeletons → hit flash → minimap heatmap → rolling rally panel (top-left) → scoreboard echo (under rally) → per-player stats (bottom-right). Rolling panel is causal — only events at or before the current frame are shown.

Config

Per-match config lives in configs/example_manifest.yaml. Each entry sets video path, layout, player names, shot detector (rule or f3ed), and initial server. Defaults at the top of the file apply to all matches unless overridden.

Testing

uv run pytest -m "not integration"    # ~200 unit tests (~3 s, no video/weights needed)
uv run pytest -m integration          # integration tests (need video files + artifacts)
uv run ruff check src/ tests/

Credits

• Upstream Phase-1 pipeline: yastrebksv/TennisProject, pinned to b7552e9. Patches (720p/1080p coordinate scaling, static-camera stabilization, player-track smoothing, YOLOv8m → YOLOv8x, upstream.npz artifact dump) live at third_party/yastrebksv_TennisProject_patches/ — apply over a fresh clone of upstream at the pinned SHA. Upstream has no LICENSE, so we don't redistribute the source.
• Shot detector: F3Set / F3ED (Liu et al., NeurIPS 2024).
• Pose model: RTMPose via rtmlib.
• Scoreboard OCR groundwork: TenniSet eval framework (Faulkner & Dick, DICTA 2017).
• Project seed: TennisExpert (Liu et al., 2026).
• Built with Claude Code.

License

MIT.