视频打分流程 On Ray 长尾问题

[INBreezefall]

Before Asking 在提问之前

• I have read the README carefully. 我已经仔细阅读了 README 上的操作指引。

• I have pulled the latest code of main branch to run again and the problem still existed. 我已经拉取了主分支上最新的代码，重新运行之后，问题仍不能解决。

Search before asking 先搜索，再提问

• I have searched the Data-Juicer issues and found no similar questions. 我已经在 issue列表 中搜索但是没有发现类似的问题。

Question

DataJuicer 1.2.0 基于 Ray 1.40.0 构建的 RayData Pipeline(OCR->AES->OptFlow) 会有稳定复现的 长尾视频处理, 我尝试过将输入 JOSNL 的条目按照切片大小排序, 但仍能明显观测到某一张卡最后单独持续运行了很久. 请问这种场景下如何做性能优化?

Additional 额外信息

No response

[yxdyc]

Hi, thank you for reporting this. The "long-tail" effect in video processing is a common challenge, where a few workers remain active long after others finish. Since video processing (OCR, AES, OptFlow) is highly sensitive to resolution, frame rate, and content complexity, sorting by file size often isn't enough to balance the actual computational load.

We have recently introduced several core engine optimizations that specifically target these kinds of large-scale, heterogeneous workloads. We recommend upgrading to the latest version of Data-Juicer to take advantage of new enhancements, e.g.,

1. Improved Resource Allocation (PR #825)
   It introduces:

• Precise resource validation: Prevents GPU idling and ensures actors are utilized at their optimal capacity.
• Robust concurrency logic: Better handling of floating-point precision during resource checks, which helps in more balanced task distribution across your GPUs.

2. Partitioning and Fault Tolerance (PR #748 - Merging Soon)
   We are introducing a new ray_partitioned executor, which helps with long-tail issues by:

• Job partitioning: Breaking down the massive JSONL into smaller, manageable partitions. This prevents a single "heavy" video from blocking the entire global pipeline's progress.
• Enhanced observability: A new event logging system allows you to track the progress of each operator and partition in real-time, making it easier to identify exactly which videos are causing the bottleneck.
• Checkpointing: If a long-tail process leads to a timeout or failure, you can resume from the last successful partition/operator rather than restarting the whole process.

3. Immediate Tuning Tips
   While upgrading, you might also try:

• Reducing batch_size: Smaller batches allow Ray's scheduler to redistribute tasks more dynamically.
• Explicit Repartitioning: Adding a .repartition() step before the most expensive op (like OptFlow) can help re-balance the workload across the cluster.
• Pre-normalization: Down-sampling or resizing videos to a uniform resolution/FPS early in the pipeline to reduce computational variance.

Collaboration & Feedback
We are constantly refining the Data-Juicer engine to handle these "long-tail" edge cases more gracefully.

• Share your case: If possible, please share your cfg yaml and a sample of the problematic data. We would love to use it for further benchmarking and optimization.
• Contribute: We welcome any suggestions or custom optimization strategies you might have. If you develop a fix or a better scheduling approach, we’d be excited to discuss with your Pull Request!