Video Pipeline/README Improvement (#1096)

* better readme

Signed-off-by: Ao Tang <aot@nvidia.com>

* better args

Signed-off-by: Ao Tang <aot@nvidia.com>

* remove video_read_example.py

Signed-off-by: Ao Tang <aot@nvidia.com>

* Add instruction for IV2

Signed-off-by: Ao Tang <aot@nvidia.com>

---------

Signed-off-by: Ao Tang <aot@nvidia.com>

Author: suiyoubi

tutorials/video/getting-started/README.md

@@ -1,6 +1,161 @@
 # Getting Started with Video Curation
 
-The Python scripts in this directory contain examples for how to run video curation workflows with NeMo Curator. In particular:
+The Python scripts in this directory contain examples for how to run video curation workflows with NeMo Curator.
 
-- `video_read_example.py` contains a simple pipeline which reads video data
-- `video_split_clip_example.py` contains a more complex pipeline which reads video data, applies a splitting algorithm, transcodes the clips, applies a motion filter, scores and filters by aesthetic content, generates embeddings, generates captions, generates previews, and saves the results
+## Scripts Overview
+
+- **`video_split_clip_example.py`**: Complete pipeline that reads videos, splits into clips, transcodes, filters, generates embeddings/captions, and saves results
+
+## Quick Start
+
+### Prerequisites
+
+1. **Set up directories**:
+   ```bash
+   export VIDEO_DIR="/path/to/your/videos" # Video to be processed
+   export OUTPUT_DIR="/path/to/output" 
+   export MODEL_DIR="./models"  # Will download models if not exist
+   ```
+
+2. **Minimal working example**:
+   ```bash
+   python video_split_clip_example.py \
+     --video-dir "$VIDEO_DIR" \
+     --output-clip-path "$OUTPUT_DIR" \
+     --splitting-algorithm fixed_stride \
+     --fixed-stride-split-duration 10.0
+   ```
+The example above demonstrates how to run a minimal video curation pipeline using NeMo Curator. It processes all videos in the specified `VIDEO_DIR`, splits each video into fixed-length clips (10 seconds each, as set by `--fixed-stride-split-duration 10.0`), and saves the resulting clips to `OUTPUT_DIR`. This is a basic workflow to get started with automated video splitting and curation, and can be extended with additional options for embedding, captioning, filtering, and transcoding as shown in later sections.
+
+### Common Use Cases
+
+**Basic video splitting with embeddings**:
+```bash
+python video_split_clip_example.py \
+  --video-dir "$VIDEO_DIR" \
+  --output-clip-path "$OUTPUT_DIR" \
+  --splitting-algorithm fixed_stride \
+  --fixed-stride-split-duration 10.0 \
+  --embedding-algorithm cosmos-embed1-224p
+```
+This example extends from the above example and adds an additional embedding stages using cosmos-embed1-224p model.
+
+**Scene-aware splitting with TransNetV2**:
+```bash
+python video_split_clip_example.py \
+  --video-dir "$VIDEO_DIR" \
+  --output-clip-path "$OUTPUT_DIR" \
+  --splitting-algorithm transnetv2 \
+  --transnetv2-threshold 0.4 \
+  --transnetv2-min-length-s 2.0 \
+  --transnetv2-max-length-s 10.0 \
+  --embedding-algorithm internvideo2 \
+  --transcode-encoder libopenh264 \
+  --verbose
+```
+This example demonstrates a more advanced workflow than the minimal example by using scene-aware splitting with the TransNetV2 algorithm (which detects scene boundaries instead of fixed intervals), applies the InternVideo2 embedding model to each clip, transcodes the output using the libopenh264 encoder, and enables verbose logging for more detailed output.
+
+**Note: Choosing Between InternVideo2 and Cosmos-Embed1 for Embeddings**
+
+Cosmos-Embed1 is generally better than InternVideo2 for most video embedding tasks, offering improved performance and quality. However, the optimal choice can vary depending on your specific use case and requirements. We recommend starting with Cosmos-Embed1 (`cosmos-embed1-224p`) for your initial experiments, as it typically provides superior results. If you find that Cosmos-Embed1 doesn't meet your specific needs or performance expectations, consider exploring InternVideo2 (`internvideo2`) as an alternative. This approach allows you to leverage the generally better-performing model first while keeping the option to experiment with InternVideo2 if needed.
+
+To install InternVideo2:
+
+InternVideo2 requires a specific installation process involving cloning the repository and applying patches:
+
+```bash
+# Run the InternVideo2 installation script from the Curator directory
+cd /path/to/Curator
+bash external/intern_video2_installation.sh
+
+uv add InternVideo/InternVideo2/multi_modality
+```
+
+After running this script, InternVideo2 will be available when you use `--embedding-algorithm internvideo2` in your video curation pipelines.
+
+
+
+**Full pipeline with captions and filtering**:
+```bash
+python video_split_clip_example.py \
+  --video-dir "$VIDEO_DIR" \
+  --output-clip-path "$OUTPUT_DIR" \
+  --splitting-algorithm fixed_stride \
+  --fixed-stride-split-duration 10.0 \
+  --embedding-algorithm cosmos-embed1-224p \
+  --generate-captions \
+  --aesthetic-threshold 3.5 \
+  --motion-filter enable
+```
+This example demonstrates the most comprehensive pipeline among the examples above. In addition to splitting videos and generating embeddings, it also generates captions for each clip and applies filtering based on aesthetic and motion scores. This means that only clips meeting the specified quality thresholds (e.g., `--aesthetic-threshold 3.5` and `--motion-filter enable`) will be kept, and captions will be generated for each valid clip. This workflow is useful for curating high-quality, captioned video datasets with automated quality control.
+
+
+## Output Structure
+
+The pipeline creates the following directory structure:
+
+```
+$OUTPUT_DIR/
+├── clips/                          # Encoded clip videos (.mp4)
+├── filtered_clips/                 # Filtered-out clips (.mp4)
+├── previews/                       # Preview images (.webp)
+├── metas/v0/                       # Per-clip metadata (.json)
+├── iv2_embd/                       # InternVideo2 embeddings (.pickle)
+├── ce1_embd/                       # Cosmos-Embed1 embeddings (.pickle)
+├── iv2_embd_parquet/               # InternVideo2 embeddings (Parquet)
+├── ce1_embd_parquet/               # Cosmos-Embed1 embeddings (Parquet)
+├── processed_videos/               # Video-level metadata
+└── processed_clip_chunks/          # Per-chunk statistics
+```
+
+## Metadata Schema
+
+Each clip generates a JSON metadata file in `metas/v0/` with the following structure:
+
+```json
+{
+  "span_uuid": "d2d0b3d1-...",
+  "source_video": "/path/to/source/video.mp4",
+  "duration_span": [0.0, 5.0],
+  "width_source": 1920,
+  "height_source": 1080,
+  "framerate_source": 30.0,
+  "clip_location": "/outputs/clips/d2/d2d0b3d1-....mp4",
+  "motion_score": {
+    "global_mean": 0.51,
+    "per_patch_min_256": 0.29
+  },
+  "aesthetic_score": 0.72,
+  "windows": [
+    {
+      "start_frame": 0,
+      "end_frame": 30,
+      "qwen_caption": "A person walks across a room",
+      "qwen_lm_enhanced_caption": "A person briskly crosses a bright modern room"
+    }
+  ],
+  "valid": true
+}
+```
+
+### Metadata Fields
+
+- **`span_uuid`**: Unique identifier for the clip
+- **`source_video`**: Path to the original video file
+- **`duration_span`**: Start and end times in seconds `[start, end]`
+- **`width_source`**, **`height_source`**, **`framerate_source`**: Original video properties
+- **`clip_location`**: Path to the encoded clip file
+- **`motion_score`**: Motion analysis scores (if motion filtering enabled)
+- **`aesthetic_score`**: Aesthetic quality score (if aesthetic filtering enabled)
+- **`windows`**: Caption windows with generated text (if captioning enabled)
+- **`valid`**: Whether the clip passed all filters
+
+## Embedding Formats
+
+### Parquet Files
+Embeddings are stored in Parquet format with two columns:
+- **`id`**: String UUID for the clip
+- **`embedding`**: List of float values (512 dimensions for InternVideo2, 768 for Cosmos-Embed1)
+
+### Pickle Files
+Individual clip embeddings are also saved as `.pickle` files for direct access.

tutorials/video/getting-started/video_read_example.py

@@ -283,7 +283,23 @@ def main(args: argparse.Namespace) -> None:
     parser = argparse.ArgumentParser()
     # General arguments
     parser.add_argument("--video-dir", type=str, required=True, help="Path to input video directory")
-    parser.add_argument("--model-dir", type=str, required=True, help="Path to model directory")
+    parser.add_argument(
+        "--model-dir",
+        type=str,
+        default="./models",
+        help=(
+            "Path to model directory containing required model weights. "
+            "Models will be automatically downloaded on first use if not present. "
+            "Required models depend on selected algorithms:\n"
+            "  - TransNetV2: For scene detection (--splitting-algorithm transnetv2)\n"
+            "  - InternVideo2: For embeddings (--embedding-algorithm internvideo2)\n"
+            "  - Cosmos-Embed1: For embeddings (--embedding-algorithm cosmos-embed1-*)\n"
+            "  - Qwen: For captioning (--generate-captions)\n"
+            "  - Aesthetic models: For filtering (--aesthetic-threshold)\n"
+            "Default: ./models\n"
+            "Example: --model-dir /path/to/models or --model-dir ./models"
+        )
+    )
     parser.add_argument("--video-limit", type=int, default=None, help="Limit the number of videos to read")
     parser.add_argument("--verbose", action="store_true", default=False)
     parser.add_argument("--output-clip-path", type=str, help="Path to output clips", required=True)
@@ -487,7 +503,7 @@ def main(args: argparse.Namespace) -> None:
         "--clip-extraction-target-res",
         type=int,
         default=-1,
-        help="Target resolution for clip extraction as (height, width). A value of -1 implies disables resize",
+        help="Target resolution for clip extraction as a square (height=width). A value of -1 disables resize",
     )
     # Aesthetic arguments
     parser.add_argument(