⚖️ ScaleProtoSeg: WACV 2025

Prototype Learning: Grouping Mechanism

📑 Multi-Scale Grouped Prototypes for Interpretable Semantic Segmentation
✍️ Hugo Porta, Emanuele Dalsasso, Diego Marcos and Devis Tuia
WACV 2025

This repository is a fork of the code for ProtoSeg.

🛠️ Installation

Install python libraries and local code:

pip install -r requirements/requirements.txt

For using Deeplab it might be necessary to run:

pip install git+https://github.com/lucasb-eyer/pydensecrf.git --no-build-isolation

⚠️ The file .env_template is here to provide an example of environment file necessary to run this repository.

💽 Data

Downloading Data

• Cityscapes: Download at this link the images: leftImgbit_trainvaltest.zip, the labels: gtFine_trainvaltest.zip, and the parts annotations: gtFinePanopticParts_trainval.zip.

• Pascal: Follow the instructions detailed at this link for the images and labels.

  • Pascal data: VOC2012 can be downloaded here.
  • Augmented training data: SegmentationClassAug can be downloaded at this link.
  • The splits: SegmentationAug can be found here.
  • The data needs to be extended with the test set available here when registered to the benchmark, it is just necessary to copy the test images from JPEGImages in the corresponding folders for train, trainaug, and val.
  • The parts annotations can be found here and the parts.yaml should be copied from this file. All those data should be placed in the same folder as the rest of the Pascal benchmark. Rename the folders in labels to train and val.

• ADE20K: The scene parsing dataset can be downloaded here.

• EM: The dataset for the segmentation of neuronal structures in EM stacks challenge - ISBI 2012 can be found here

• COCO-Stuff: The COCO 2017 train, and validation images can be found at this link with the Stuff+thing annotations. In the target COCO-Stuff folder move the images from train2017.zip and val2017.zip folders to images/train and images/val. Similarly, move the annotations from stuffthingmaps_trainval2017.zip to annotations_png/train and annotations_png/val.

Preprocessing Data

Cityscapes

# replace <N> with the number of parallel jobs
python -m segmentation.data.preprocess_cityscapes <N> # for cityscapes
python -m segmentation.data.img_to_numpy --margin-size 0 --data-type cityscapes
python -m segmentation.data.preprocess_part_cityscapes preprocess-cityscapes <N> # for cityscapes

Pascal

# replace <N> with the number of parallel jobs
python -m segmentation.data.preprocess_pascal <N> # for pascal
python -m segmentation.data.img_to_numpy --margin-size 0 --data-type pascal
python -m segmentation.data.preprocess_part_pascal <N> # for pascal

ADE20K

# replace <N> with the number of parallel jobs
python -m segmentation.data.preprocess_ade <N> # for ade
python -m segmentation.data.img_to_numpy --margin-size 0 --data-type ade

EM

In EM preprocessing you can specify a seed to create different train/val splits from the original train data. In the paper we tested our models against two different random splits.

# replace <N> with the number of parallel jobs
python -m segmentation.data.preprocess_em <N> --seed 42 # for EM
python -m segmentation.data.img_to_numpy --margin-size 0 --data-type em

COCO-Stuff

# replace <N> with the number of parallel jobs
python -m segmentation.data.preprocess_coco <N> # for COCO
python -m segmentation.data.img_to_numpy --margin-size 0 --data-type coco

💪 Training

All the checkpoints for ScaleProtoSeg can be downloaded at the following Zenodo Link.

ScaleProtoSeg

# Example for cityscapes: Prototype Learning
python -m segmentation.train_wandb_multiscale scaleproto_cityscapes <your_training_run_name>

# Example for cityscapes: Grouping Mechanism
python -m segmentation.finetune_wandb_group group_scaleproto_cityscapes <your_training_run_name>

In the folder segmentation/configs all the config files follow a similar naming pattern than the one in the example above.

You need to specify for the grouping the path of the checkpoints from the first phase: train.start_checkpoint parameter.

ProtoSeg

# Example for cityscapes: Prototype Learning
python -m segmentation.train_wandb baseline_cityscapes <your_training_run_name>

# Example for cityscapes: Pruning
python -m segmentation.run_pruning baseline_cityscapes <your_training_run_name>

# Example for cityscapes: Finetuning
python -m segmentation.train_wandb baseline_cityscapes <your_training_run_name> --pruned

Please look at the file train_wandb.py and train_wandb_multiscale.py for manual modifications when training on EM.

🧪 Evaluation

ScaleProtoSeg

# Setting to zero the quasi non-used prototypes
python -m segmentation.analysis.threshold_save <your_training_run_name> <training_stage> <threshold>

# Evaluation on validation set
python -m segmentation.eval_valid_multiscale <your_training_run_name> <training_stage> <batch_size> <data_type>

# Generating predictions on cityscapes or pascal test set
python -m segmentation.eval_test  <your_training_run_name> <training_stage> <batch_size> <flag_pascal>

# Evaluation for consistency metric (same for stability)
python -m segmentation.analysis.consistency <prototype_training_run_name> <training_stage> <data_type> <quantile> <threshold> <group_training_run_name>

ProtoSeg

# Evaluation on validation set for Pascal or Cityscapes
python -m segmentation.eval_valid <your_training_run_name> <training_stage> <flag_pascal>

# Evaluation on validation set for ADE, COCO, or EM
python -m segmentation.eval_valid_multiscale <your_training_run_name> <training_stage> <batch_size> <data_type>

# Generating predictions on cityscapes or pascal test set
python -m segmentation.eval_test  <your_training_run_name> <training_stage> <batch_size> <flag_pascal>

# Evaluation for consistency metric (same for stability)
python -m segmentation.analysis.consistency <prototype_training_run_name> <training_stage> <data_type> <quantile> <threshold>

For testing it is necessary to then properly arrange the output folder to match the benchmarks requirements online.

🥇 Performance

On the validation sets, we report the results over 3 runs, while for the test sets, the results are based on ProtoSeg and our best ScaleProtoSeg validation run.

Method	Cityscapes (val)	Cityscapes (test)	Pascal (val)	Pascal (test)	ADE20K (val)
DeepLabv2 [Chen et al. 2017]	71.40	70.40	77.69	79.70	34.00
ProtoSeg [Sacha et al. 2023]	67.54 ± 0.22	67.04	71.98 ± 0.11	72.92	29.67 ± 0.23
ScaleProtoSeg	68.97 ± 0.25	68.52	71.80 ± 0.38	72.35	34.18 ± 0.18

Analysis

In the folder segmentation/analysis numerous scripts are available to investigate the interpretability of our model via different plotting strategy.

Prototype Maps: Grouping Mechanism

🖋️ Citation

@InProceedings{Porta_2025_WACV,
    author    = {Porta, Hugo and Dalsasso, Emanuele and Marcos, Diego and Tuia, Devis},
    title     = {Multi-Scale Grouped Prototypes for Interpretable Semantic Segmentation},
    booktitle = {Proceedings of the Winter Conference on Applications of Computer Vision (WACV)},
    month     = {February},
    year      = {2025},
    pages     = {2869-2880}
}