Fix: fix rare case extraction workflow

Author: WCJ-BERT

docs/algengine_usage.md

@@ -11,7 +11,9 @@ This guide covers how to use AlgEngine for training end-to-end autonomous drivin
 - [Fine-Tuning](#fine-tuning)
 - [Configuration](#configuration)
 - [Model Architectures](#model-architectures)
+- [Advanced Training](#advanced-training)
 - [Troubleshooting](#troubleshooting)
+- [Performance Optimization](#performance-optimization)
 
 ---
 
@@ -25,6 +27,9 @@ cd projects/AlgEngine
 # Open-loop evaluation
 ./scripts/e2e_dist_eval.sh <config> <checkpoint> <num_gpus>
 
+# Full train set evaluation (chunked)
+bash scripts/e2e_dist_eval_navtrain_chunked.sh <config> <checkpoint> <num_gpus> [num_chunks]
+
 # Rare case extraction
 python scripts/rare_case_sampling_by_pdms.py \
     --pdm-result <csv_file> \
@@ -248,7 +253,7 @@ conda activate algengine
 cd projects/AlgEngine
 
 python scripts/rare_case_sampling_by_pdms.py \
-    --pdm-result work_dirs/e2e_vadv2_50pct/navtrain.csv \
+    --pdm-result ${WORLDENGINE_ROOT}/experiments/worldengine/e2e_vadv2_50pct/navtrain.csv \
     --base-split configs/navsim_splits/navtrain_split/navtrain_50pct.yaml \
     --output-dir configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_rare
 ```
@@ -263,10 +268,10 @@ python scripts/rare_case_sampling_by_pdms.py \
 The script generates three rare case split files:
 
 ```
-configs/navsim_splits/navtest_split/e2e_vadv2_50pct_rare/
-├── navtest_collision.yaml         # Collision scenarios
-├── navtest_off_road.yaml          # Off-road scenarios
-└── navtest_ep_1pct.yaml           # Low ego-progress (bottom 1%)
+configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_rare/
+├── navtrain_50pct_collision.yaml      # Collision scenarios
+├── navtrain_50pct_off_road.yaml       # Off-road scenarios
+└── navtrain_50pct_ep_1.0pct.yaml      # Low ego-progress (bottom 1%)
 ```
 
 ### Using Custom Thresholds
@@ -287,10 +292,10 @@ ep_threshold = df['ego_progress'].quantile(0.05)  # From 0.01 (1% -> 5%)
 
 ```bash
 # Check how many scenarios were extracted
-wc -l configs/navsim_splits/navtest_split/e2e_vadv2_50pct_rare/*.yaml
+wc -l configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_rare/*.yaml
 
 # View first few scenarios
-head -20 configs/navsim_splits/navtest_split/e2e_vadv2_50pct_rare/navtest_collision.yaml
+head -20 configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_rare/navtrain_50pct_collision.yaml
 ```
 
 ---
@@ -303,7 +308,22 @@ Fine-tune a trained model on rare cases using reinforcement learning.
 
 **Important:** Rare case extraction and Rollout data must be generated by SimEngine before fine-tuning. This involves:
 
-1. **Run SimEngine Rollout** to generate trajectory data:
+1. **Convert nuPlan Data to SimEngine Format** — convert rare case scenarios to SimEngine scenario format:
+   ```bash
+   conda activate simengine
+
+   python projects/SimEngine/worldengine/utils/dataset_utils/nuplan/digitaltwin_nuplan_converter_navsim_filter.py \
+       --navsim-filters $ALGENGINE_ROOT/configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_rare/navtrain_50pct_collision.yaml \
+        $ALGENGINE_ROOT/configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_rare/navtrain_50pct_ep_1.0pct.yaml \
+        $ALGENGINE_ROOT/configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_rare/navtrain_50pct_off_road.yaml \
+       --out-dir data/sim_engine/scenarios/original/navtrain_vadv2_50pct_rare \
+       --num-processes 8
+   ```
+   **Output:** `data/sim_engine/scenarios/original/navtrain_vadv2_50pct_rare/all_scenarios.pkl`
+
+   For full parameter reference, see [SimEngine: Convert nuPlan Data](simengine_usage.md#convert-nuplan-data-to-simengine-format).
+
+2. **Run SimEngine Rollout** to generate trajectory data:
    ```bash
    # Multi-GPU distributed rollout (recommended for large datasets)
    export WORLDENGINE_ROOT=/path/to/WorldEngine
@@ -312,11 +332,11 @@ Fine-tune a trained model on rare cases using reinforcement learning.
        $WORLDENGINE_ROOT/projects/AlgEngine/configs/worldengine/e2e_vadv2_50pct.py \
        $WORLDENGINE_ROOT/data/alg_engine/ckpts/e2e_vadv2_50pct_ep8.pth \
        e2e_vadv2_50pct \
-       navtrain_50pct_collision \ 
+       navtrain_vadv2_50pct_rare \
        navtrain
    ```
 
-2. **Rollout Output** is saved to:
+3. **Rollout Output** is saved to:
    ```
    experiments/closed_loop_exps/e2e_vadv2_50pct/navtrain_NR/
    └── WE_output/
@@ -327,7 +347,7 @@ Fine-tune a trained model on rare cases using reinforcement learning.
            └── all_scenes_pdm_averages_NR.csv
    ```
 
-3. **Reorganize to AlgEngine Format** (creates `openscene-synthetic` dataset):
+4. **Reorganize to AlgEngine Format** (creates `openscene-synthetic` dataset):
    ```bash
    conda activate simengine
 
@@ -339,7 +359,7 @@ Fine-tune a trained model on rare cases using reinforcement learning.
 
    **Output location:** `data/alg_engine/openscene-synthetic/`
 
-4. **Verify Data Structure:**
+5. **Verify Data Structure:**
    ```bash
    data/alg_engine/openscene-synthetic/
    ├── sensor_blobs/              # Replayed scenario sensor data
@@ -444,10 +464,10 @@ configs/
     ├── navtrain_split/
     │   ├── navtrain.yaml               # Full training set
     │   ├── navtrain_50pct.yaml         # 50% subset
-    │   └── e2e_vadv2_50pct_ep8/        # Rare case splits
+    │   └── e2e_vadv2_50pct_rare/       # Rare case splits
     │       ├── navtrain_50pct_collision.yaml
     │       ├── navtrain_50pct_off_road.yaml
-    │       └── navtrain_50pct_ep_1pct.yaml
+    │       └── navtrain_50pct_ep_1.0pct.yaml
     └── navtest_split/
         ├── navtest.yaml                # Full test set
         └── navtest_failures.yaml       # Failure subset

docs/simengine_usage.md

@@ -84,7 +84,7 @@ bash scripts/run_ray_distributed_rollout.sh \
 1. `<config>`: Model configuration file
 2. `<checkpoint>`: Trained model checkpoint
 3. `<model_name>`: Experiment name (creates output folder)
-4. `<data_type>`: Scenario split (e.g., `navtrain_50pct_collision`) PS: **Default** original, please change it in `run_ray_distributed_rollout.sh`.
+4. `<data_type>`: Scenario split (e.g., `navtrain_50pct_collision`). Must match a directory under `data/sim_engine/scenarios/original/`
 5. `<asset_name>` (optional): Asset folder name under `data/sim_engine/assets/`. Defaults to `<data_type>` if not provided
 
 **Features:**
@@ -265,18 +265,19 @@ WE_output/openscene_format/
 
 ## Utility Scripts
 
-### Convert nuPlan Data to WorldEngine Format
+### Convert nuPlan Data to SimEngine Format
 
-Convert nuPlan dataset to WorldEngine scenario format with navsim filter-based scene selection.
+Convert nuPlan dataset to SimEngine scenario format with navsim filter-based scene selection.
 
 Example:
 ```bash
 conda activate simengine
 
 python projects/SimEngine/worldengine/utils/dataset_utils/nuplan/digitaltwin_nuplan_converter_navsim_filter.py \
-    --navsim-filters $ALGENGINE_ROOT/configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_rare/navtrain_collision.yaml \
-        $ALGENGINE_ROOT/configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_rare/navtrain_ep_1pct.yaml \
-    --out-dir data/sim_engine/scenarios/original/navtrain_hydramdp_failures \
+    --navsim-filters $ALGENGINE_ROOT/configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_rare/navtrain_50pct_collision.yaml \
+        $ALGENGINE_ROOT/configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_rare/navtrain_50pct_ep_1.0pct.yaml \
+        $ALGENGINE_ROOT/configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_rare/navtrain_50pct_off_road.yaml \
+    --out-dir data/sim_engine/scenarios/original/navtrain_vadv2_50pct_rare \
     --num-processes 8
 ```
 
@@ -291,7 +292,7 @@ python projects/SimEngine/worldengine/utils/dataset_utils/nuplan/digitaltwin_nup
 - `--num-processes`: Number of parallel processes
 
 **Output:**
-- `{out-dir}/combined/all_scenarios.pkl` - Converted WorldEngine scenario file
+- `{out-dir}/all_scenarios.pkl` - Converted WorldEngine scenario file
 
 The script extracts from Digital Twin config and nuPlan raw data:
 - Ego and agent vehicle trajectories