bugfix: Fix issues in MPEG sequence preprocessing script and README documentation

Author: JasonLSC

README.md

@@ -18,13 +18,15 @@ conda activate gscodec_studio
 
 ### Packages Installation
 
-Please install [Pytorch](https://pytorch.org/get-started/locally/) first.
-
-Then, you can install the gsplat library extended with more compression features from source code. In this way it will build the CUDA code during installation.
+Please install [Pytorch](https://pytorch.org/get-started/locally/) first. Then, you can install the gsplat library extended with more compression features from source code. In this way it will build the CUDA code during installation.
 
 ```bash
 pip install .
-# pip install -e . (develop mode)
+```
+
+If you want to do further development based on this framework, you use following command to install Python packages in editable mode.
+```bash
+pip install -e . # (develop)
 ```
 
 ## Examples
@@ -38,7 +40,10 @@ cd examples
 pip install -r requirements.txt
 # download mipnerf_360 benchmark data
 python datasets/download_dataset.py
-# or place other dataset under 'data' folder
+```
+You can download the Tanks and Temples dataset and Deep Blending dataset used in original 3DGS via [this link](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/datasets/input/tandt_db.zip) and place these datasets under 'data' folder.
+```bash
+# place other dataset, e.g Tanks and Temples dataset, under 'data' folder
 ln -s data/tandt /xxxx/Dataset/tandt
 ```
 
@@ -67,7 +72,7 @@ bash benchmarks/compression/final_exp/mcmc_tt_sim.sh
 
 **Dynamic Gaussian Splats Training and Compression**
 
-First, please follow the dataset preprocessing instruction described in the [file]((mpeg_gsc_utils/multiview_video_preprocess/README.md)) for training data prepration.
+First, please follow the dataset preprocessing instruction described in the [file](mpeg_gsc_utils/multiview_video_preprocess/README.md) for training data prepration.
 
 Next, run the script for dynamic gaussian splats training and compression.
 ```bash

examples/benchmarks/dyngs/dyngs.sh

@@ -1,16 +1,18 @@
 SCENE_DIR="data/GSC"
-SCENE_LIST="Bartender" # CBA Bartender
+SCENE_LIST="Cinema" # CBA Bartender Cinema
 
 declare -A TEST_VIEWS
 TEST_VIEWS=(
     ["CBA"]="7 22"
     ["Bartender"]="8 10 12"
+    ["Cinema"]="8 10 12"
 )
 
 declare -A START_FRAMES
 START_FRAMES=(
     ["CBA"]=0
     ["Bartender"]=50
+    ["Cinema"]=235
 )
 
 RESULT_DIR="results/dyngs"
@@ -25,22 +27,24 @@ run_single_scene() {
 
     echo "Running $SCENE START_FRAME @ ${START_FRAME}"
 
-    # CUDA_VISIBLE_DEVICES=$GPU_ID python simple_trainer_dyngs.py compression_sim \
-    #     --model_path $RESULT_DIR/$SCENE/ \
-    #     --data_dir $SCENE_DIR/$SCENE/colmap/colmap_${START_FRAME} \
-    #     --result_dir $RESULT_DIR/$SCENE/ \
-    #     --downscale_factor 1 \
-    #     --duration $NUM_FRAME \
-    #     --batch_size 2 \
-    #     --max_steps 60_000 \
-    #     --refine_start_iter 3_000 \
-    #     --refine_stop_iter 30_000 \
-    #     --refine_every 100 \
-    #     --reset_every 6_000 \
-    #     --pause_refine_after_reset 500 \
-    #     --strategy Modified_STG_Strategy \
-    #     --test_view_id $TEST_VIEW_IDS 
+    # execute training 
+    CUDA_VISIBLE_DEVICES=$GPU_ID python simple_trainer_dyngs.py compression_sim \
+        --model_path $RESULT_DIR/$SCENE/ \
+        --data_dir $SCENE_DIR/$SCENE/colmap/colmap_${START_FRAME} \
+        --result_dir $RESULT_DIR/$SCENE/ \
+        --downscale_factor 1 \
+        --duration $NUM_FRAME \
+        --batch_size 2 \
+        --max_steps 60_000 \
+        --refine_start_iter 3_000 \
+        --refine_stop_iter 30_000 \
+        --refine_every 100 \
+        --reset_every 6_000 \
+        --pause_refine_after_reset 500 \
+        --strategy Modified_STG_Strategy \
+        --test_view_id $TEST_VIEW_IDS 
 
+    # execute evaluation and compression 
     CUDA_VISIBLE_DEVICES=$GPU_ID python simple_trainer_dyngs.py default \
         --model_path $RESULT_DIR/$SCENE/ \
         --data_dir $SCENE_DIR/$SCENE/colmap/colmap_${START_FRAME} \
@@ -53,7 +57,7 @@ run_single_scene() {
         --test_view_id $TEST_VIEW_IDS 
 }
 
-GPU_LIST=(6)
+GPU_LIST=(0)
 GPU_COUNT=${#GPU_LIST[@]}
 
 SCENE_IDX=-1

mpeg_gsc_utils/multiview_video_preprocess/README.md

@@ -4,6 +4,7 @@
 **MPEG Multiview Video Dataset**
 Please organize the data structure as follows:
 
+```
 data/
 ├── GSC
 │   ├── scene1/
@@ -14,25 +15,43 @@ data/
 │   │   │   ├── v00_texture_1920x1080_yuv420p10le.yuv
 │   │   │   ├── v01_texture_1920x1080_yuv420p10le.yuv
 │   │   │   ├── ...
-│   ├── scene1/
+│   ├── scene2/
 │   │   ├── colmap
 │   │   ├── mp4
 │   │   ├── png
 │   │   ├── yuv
 │   │   │   ├── v00_texture_1920x1080_yuv420p10le.yuv
 │   │   │   ├── v01_texture_1920x1080_yuv420p10le.yuv
 │   │   │   ├── ...
+```
+
+## Preliminaries
+The following series of preprocessing operations for MPEG datasets requires **Colmap** and **ffmpeg** to be installed in advance.
 
 ## Operations
+1. Before you start, please open ``mpeg_gsc_utils/multiview_video_preprocess/scene_info.py`` and fill in some necessary metadata for the Scene you are going to convert.
+For now, we have already prepared the metadata for Bartender and Cinema.
+
 1. Convert multiview videos in "yuv" format to "mp4" format and "png" format
 ```bash
-python mpeg_gsc_utils/multiview_video_preprocess/video_preprocess.py
-```
+python mpeg_gsc_utils/multiview_video_preprocess/video_preprocess.py \
+    --scene Cinema 
+``` 
+
+For the script mentioned above, you must specify the scene name. Additionally, if you haven't placed your data under ``examples/data/GSC/{scene}``, you can manually specify your base_dir by passing ``--base_dir``.
+
 2. Obtain the camera intrinsic and extrinsic parameters and save as "poses_bds.npy" file
 ```bash
-python mpeg_gsc_utils/multiview_video_preprocess/gen_poses_bds.py
+python mpeg_gsc_utils/multiview_video_preprocess/gen_poses_bds_file.py \
+    --scene Cinema 
 ```
+
+For the script mentioned above, you must specify the scene name. Additionally, if you haven't placed your data under ``examples/data/GSC/{scene}``, you can manually specify your base_dir by passing ``--base_dir``.You can also select a specific number of frames by passing ``--frame_num``.
+
 3. Run colmap frame by frame to get per-frame SfM point clouds 
 ```bash
-python mpeg_gsc_utils/multiview_video_preprocess/run_per_frame_colmap.py
-```
+python mpeg_gsc_utils/multiview_video_preprocess/run_per_frame_colmap.py \
+    --scene Cinema 
+```
+
+For the script mentioned above, you must specify the scene name. Additionally, if you haven't placed your data under ``examples/data/GSC/{scene}``, you can manually specify your base_dir by passing ``--base_dir``.You can also select a specific number of frames by passing ``--frame_num``.

mpeg_gsc_utils/multiview_video_preprocess/gen_poses_bds_file.py

@@ -4,6 +4,9 @@
 from pathlib import Path
 import shutil
 from tqdm import trange
+import tyro
+from dataclasses import dataclass
+from typing import Optional
 
 from mpeg_gsc_utils import colmap_read_model as read_model
 from scene_info import DATASET_INFOS
@@ -211,42 +214,74 @@ def gen_poses(basedir, match_type, factors=None):
 
     return True
 
-if __name__ == "__main__":
-    SCENE = "Bartender"
-    BASE_DIR = f"/work/Users/lisicheng/Dataset/GSC/{SCENE}"
+@dataclass
+class FramePrepConfig:
+    """Configuration for preparing frame data and generating poses"""
+    scene: str
+    """Scene name (e.g., Bartender)"""
+    
+    base_dir: Optional[str] = None
+    """Base directory path. If not provided, defaults to examples/data/GSC/{scene}"""
+    
+    frame_num: int = 65
+    """Number of frames to process"""
+    
+    match_type: str = "exhaustive_matcher"
+    """Matcher type for pose generation"""
+
+def main(config: FramePrepConfig):
+    # Process parameters
+    SCENE = config.scene
+    BASE_DIR = config.base_dir if config.base_dir else f"examples/data/GSC/{SCENE}"
     COLMAP_DIR = BASE_DIR + "/colmap"
-    FRAME_NUM = 65
+    FRAME_NUM = config.frame_num
     START_FRAME = DATASET_INFOS[SCENE]["start_frame"]
 
-    # copy png to each sub colmap dir
-    imgs_source_dir = Path(BASE_DIR + f"/png")
+    # Create COLMAP directory if it doesn't exist
+    os.makedirs(COLMAP_DIR, exist_ok=True)
 
+    print(f"Processing scene {SCENE} with {FRAME_NUM} frames starting from {START_FRAME}")
+    print(f"Base directory: {BASE_DIR}")
+    print(f"Working directory: {COLMAP_DIR}")
+
+    # Copy png to each frame directory
+    imgs_source_dir = Path(BASE_DIR + f"/png")
+    
+    print("Creating frame directories and copying images...")
     for f_idx in trange(START_FRAME, START_FRAME+FRAME_NUM):
         colmap_frame_dir = COLMAP_DIR + f"/colmap_{f_idx}"
         os.makedirs(colmap_frame_dir, exist_ok=True)
-        imgs_target_dir = colmap_frame_img_dir = Path(colmap_frame_dir + f"/input")
+        imgs_target_dir = Path(colmap_frame_dir + f"/input")
         os.makedirs(imgs_target_dir, exist_ok=True)
 
         for file in sorted(imgs_source_dir.glob(f"*{f_idx+1:03d}.png")):
             new_filenanme = file.name.split("_")[0] + ".png" 
             shutil.copy2(file, imgs_target_dir / new_filenanme)
 
-        # just a work-around for generating poses_bounds.npy
+        # Just a work-around for generating poses_bounds.npy
         if f_idx == START_FRAME:
             colmap_frame_dir = COLMAP_DIR + f"/colmap_{f_idx}"
             os.makedirs(colmap_frame_dir, exist_ok=True)
-            imgs_target_dir = colmap_frame_img_dir = Path(colmap_frame_dir + f"/images")
+            imgs_target_dir = Path(colmap_frame_dir + f"/images")
             os.makedirs(imgs_target_dir, exist_ok=True)
 
             for file in sorted(imgs_source_dir.glob(f"*{f_idx+1:03d}.png")):
                 new_filenanme = file.name.split("_")[0] + ".png" 
                 shutil.copy2(file, imgs_target_dir / new_filenanme)
 
-    # # get "poses_bounds.npy"
+    # Generate poses_bounds.npy
+    print("Generating poses_bounds.npy using COLMAP...")
     colmap_first_frame_dir = COLMAP_DIR + f"/colmap_{START_FRAME}"
     if not os.path.exists(colmap_first_frame_dir + "/poses_bounds.npy"):
-        gen_poses(colmap_first_frame_dir, match_type="exhaustive_matcher")
+        gen_poses(colmap_first_frame_dir, match_type=config.match_type)
 
-    # # move to basedir of the specific scene
+    # Move poses_bounds.npy to the base directory
+    print("Copying poses_bounds.npy to base directory...")
     shutil.copy2(colmap_first_frame_dir + "/poses_bounds.npy", COLMAP_DIR)
-    shutil.rmtree(colmap_first_frame_dir + "/images")
+    shutil.rmtree(colmap_first_frame_dir + "/images")
+    
+    print("Frame preparation and pose generation completed!")
+
+if __name__ == "__main__":
+    config = tyro.cli(FramePrepConfig)
+    main(config)

mpeg_gsc_utils/multiview_video_preprocess/run_per_frame_colmap.py

@@ -3,6 +3,9 @@
 import numpy as np
 from pathlib import Path
 from tqdm import trange
+import tyro
+from dataclasses import dataclass
+from typing import Optional
 
 from scene_info import DATASET_INFOS
 from mpeg_gsc_utils.pre_colmap import COLMAPDatabase
@@ -205,18 +208,42 @@ def getcolmapsinglen3d(folder, offset):
         destination_file = os.path.join(folder, "sparse", "0", file)
         shutil.move(source_file, destination_file)
 
-if __name__ == "__main__":
-    SCENE = "Bartender"
-    BASE_DIR = f"/work/Users/lisicheng/Dataset/GSC/{SCENE}"
+@dataclass
+class ColmapProcessConfig:
+    """Configuration for processing frames with COLMAP"""
+    scene: str
+    """Scene name (e.g., Bartender)"""
+    
+    base_dir: Optional[str] = None
+    """Base directory path. If not provided, defaults to examples/data/GSC/{scene}"""
+    
+    frame_num: int = 65
+    """Number of frames to process"""
+
+def main(config: ColmapProcessConfig):
+    # Process parameters
+    SCENE = config.scene
+    BASE_DIR = config.base_dir if config.base_dir else f"examples/data/GSC/{SCENE}"
     COLMAP_DIR = BASE_DIR + "/colmap"
-    FRAME_NUM = 65
+    FRAME_NUM = config.frame_num
     START_FRAME = DATASET_INFOS[SCENE]["start_frame"]
 
-    # # make sure every frame share the same camera extrinsic and intrinsic 
-    # for frame in trange(START_FRAME, START_FRAME+FRAME_NUM):
-    #     convertdynerftocolmapdb(Path(COLMAP_DIR), frame, )
+    print(f"Processing scene {SCENE} with {FRAME_NUM} frames starting from {START_FRAME}")
+    print(f"Base directory: {BASE_DIR}")
+    print(f"COLMAP directory: {COLMAP_DIR}")
+
+    # Make sure every frame share the same camera extrinsic and intrinsic
+    print("Converting DyNeRF format to COLMAP database for each frame...")
+    for frame in trange(START_FRAME, START_FRAME+FRAME_NUM):
+        convertdynerftocolmapdb(Path(COLMAP_DIR), frame)
+    
+    # Run COLMAP for each frame to obtain initial point clouds
+    print("Running COLMAP for each frame to obtain initial point clouds...")
+    for frame in trange(START_FRAME, START_FRAME+FRAME_NUM):
+        getcolmapsinglen3d(Path(COLMAP_DIR), frame)
 
-    # # run colmap for each frame to obtain initial point clouds
-    # for frame in range(START_FRAME+1, START_FRAME+FRAME_NUM):
-    #     getcolmapsinglen3d(Path(COLMAP_DIR), frame, )
-    getcolmapsinglen3d(Path(COLMAP_DIR), 50, )
+    print("COLMAP processing completed!")
+
+if __name__ == "__main__":
+    config = tyro.cli(ColmapProcessConfig)
+    main(config)

mpeg_gsc_utils/multiview_video_preprocess/scene_info.py

@@ -12,5 +12,6 @@
     },
     "Cinema": {
         "start_frame": 235,
+        "resolution": "1920x1080"
     },
 }