Open benchmark for burned-in video subtitle OCR — watermark-adjacent text tanks CER 2-2.6x across every engine we tested

[YeefyisOK]

We built a benchmark for a problem general OCR benchmarks don't cover: burned-in video subtitles that sit next to watermarks/credits. 600 samples, 6 languages, ~19% with a synthetic overlapping watermark to test detection robustness specifically.

Tested raw (no post-filtering) detection+recognition across EasyOCR (with MPS enabled on Apple Silicon), PaddleOCR PP-OCRv6 (tiny/small/medium), PP-OCRv5 (ONNX build), and Tesseract. A couple of findings that might be relevant to you:

1. Watermark-adjacent text tanks CER by 2-2.6x across every engine we tested, EasyOCR included — 0.56 → 1.26 CER when a second text region overlaps the subtitle. Seems like a shared blind spot across detection-based OCR generally, not specific to any one model.
2. On Apple Silicon, easyocr.Reader(gpu=True) gets a real speedup from PyTorch's MPS backend once enabled (591ms → 142ms/image in our testing) — worth knowing if you're benchmarking on Mac and only testing the CPU default.

Repo + eval script + all prediction files (including our EasyOCR run) are open: https://github.com/GeekLinkDev/geeklink-ocr-benchmark

Disclosure: I'm the developer of GeekLink, a subtitle extraction tool that uses OCR (currently PaddleOCR-based models, not EasyOCR) — sharing this because the watermark-robustness problem seems like it'd be useful for any OCR project to have real numbers on.

[YeefyisOK]

Quick correction: found a real bug after posting this — 437 samples (38%) had ground-truth text wider than the video frame, so the rendered subtitle got clipped off-screen. Fixed with proper line wrapping and extended the set from 600 to 1,140 samples in the same pass. Repo and README both updated: https://github.com/GeekLinkDev/geeklink-ocr-benchmark