IllumiDiff: Indoor Illumination Estimation from a Single Image with Diffusion Model (TVCG 2025)

Shiyuan Shen^†, Zhongyun Bao^†, Wenju Xu, Chunxia Xiao

📘Paper · 📃PDF · 🏠HomePage · 🤗Demo

The current version builds on LDM.

Unlike the original paper, we retrain the models at all stages using 3,229 (training set) HDR panoramas without separating indoor and outdoor scenes.

The ControlNet implementation is available in the v-controlnet branch.

Structure

IllumiDiff/
├── ckpts/                      # downloaded checkpoints for inference
├── lighting_est/               # Stage 1 and Stage 3 models, datasets, and training
├── pano_ldm/                   # Stage 2 latent diffusion implementation
├── datasets/                   # preprocessing, fitting, splitting, and inpainting tools
├── utils.py                    # image, logging, runtime, and panorama utilities
├── inference.py                # full pipeline, ldr pers to hdr pano
└── inference_single.py         # parameterized SG + ASG lighting

Environment

conda create -n illumidiff python=3.11
conda activate illumidiff
pip install torch==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt

Checkpoints

Download the checkpoints from ☁️OneDrive or 🤗Buckets and place them in ./ckpts/:

id_net-step020k.ckpt
sg_net-step080k.ckpt
asg_net-step100k.ckpt
hdr_net-step038k.ckpt
ldm-step015k.ckpt

Inference

Full lighting estimation + panorama generation + inverse tonemapping inference:

python inference.py --input-path <path> --output-path <path>

Linear SG + ASG output:

python inference_single.py --input-path <path> --output-path <path>

Quantitative Comparison

Metric	IllumiDiff	DiffusionLight	LuxDiT
Base model	LDM	SDXL	CogVideo
Parameters	~ 0.7B	~ 3.5B	~ 5B
SI-RMSE	0.103 | 0.185 | 0.206	0.123 | 0.216 | 0.239	0.114 | 0.211 | 0.231
PSNR	17.43 | 13.99 | 12.62	15.54 | 12.84 | 11.72	16.70 | 12.95 | 12.02
RGB ang.	3.37 | 4.30 | 5.27	3.98 | 4.84 | 5.78	3.96 | 4.78 | 5.75
FID	114.67 | 29.09 | 44.60	22.57 | 34.78 | 87.23	94.69 | 32.52 | 81.21

Dataset

Start with a folder of HDR panoramas:

# Original HDR panoramas -> paired rotated pano/perspective HDR and LDR data
python datasets/build_dataset.py --input-dir <hdr-folder> --dataset-root <dataset_root>

# Fit parameterized lighting GT
python datasets/sg_fitting.py --dataset-root <dataset_root>
python datasets/asg_fitting.py --dataset-root <dataset_root>

# Preview a split before moving complete modality groups.
python datasets/split_testset.py --dataset-root <dataset_root>

Training

All networks are trained separately.

For id_net, sg_net, asg_net or hdr_net:

python -m lighting_est.train --config lighting_est/configs/<network>.toml

For LDM:

python -m pano_ldm.train_vae_1ch --config pano_ldm/configs/train_vae.toml
python -m pano_ldm.train --config pano_ldm/configs/train.toml --init-ckpt <init-path>

Acknowledgements

This project builds on ideas and components from LDM and Skylibs.

Citation

@article{shen2025illumidiff,
    title = {IllumiDiff: Indoor Illumination Estimation from a Single Image with Diffusion Model},
    author = {Shen, Shiyuan and Bao, Zhongyun and Xu, Wenju and Xiao, Chunxia},
    journal = {IEEE transactions on visualization and computer graphics},
    year = {2025},
    publisher = {IEEE}
}

Contact

For questions, please contact: syshen@whu.edu.cn