LIX: Implicitly Infusing Spatial Geometric Prior Knowledge into Visual Semantic Segmentation for Autonomous Driving (TIP-2025)

Paper | Project Page

This is the official implementation for LIX:

LIX: Implicitly Infusing Spatial Geometric Prior Knowledge into Visual Semantic Segmentation for Autonomous Driving

Sicen Guo, Ziwei Long, Zhiyuan Wu, Qijun Chen, Ioannis Pitas, Rui Fan

📋 Abstract

Despite the impressive performance achieved by data-fusion networks with duplex encoders for visual semantic segmentation, they become ineffective when spatial geometric data are not available. Implicitly infusing the spatial geometric prior knowledge acquired by a data-fusion teacher network into a single-modal student network is a practical, albeit less explored research avenue. This article delves into this topic and resorts to knowledge distillation approaches to address this problem. We introduce the Learning to Infuse "X" (LIX) framework, with novel contributions in both logit distillation and feature distillation aspects. We present a mathematical proof that underscores the limitation of using a single, fixed weight in decoupled knowledge distillation and introduce a logit-wise dynamic weight controller as a solution to this issue. Furthermore, we develop an adaptively-recalibrated feature distillation algorithm, including two novel techniques: feature recalibration via kernel regression and feature consistency quantification via centered kernel alignment. Extensive experiments conducted with intermediate-fusion and late-fusion networks across various public datasets provide both quantitative and qualitative evaluations, demonstrating the superior performance of our LIX framework when compared to other state-of-the-art approaches.

🪧 Overview

• We propose LIX, a knowledge distillation framework that implicitly infuses spatial geometric prior knowledge into visual semantic segmentation by distilling an RGBX data-fusion teacher network into a single-modal student network that operates solely on RGB images.

• We present the novel dynamically-weighted logit distillation algorithm, which extends the DKD algorithm, by assigning an appropriate weight to each logit, resulting in better performance compared to the baseline algorithm.

• We introduce the novel adaptively-recalibrated feature distillation algorithm that performs feature recalibration via kernel regression and feature consistency measurement leveraging HSIC-based CKA.

• We have conducted extensive experiments using representative RGB-X semantic segmentation networks on multiple public datasets to quantitatively and qualitatively validate the effectiveness of our introduced novel LD and FD techniques.

🏗️️ Setup

Follow the detailed Setup in SNE-RoadSeg.

💾 Dataset Preparation

nuImage Dataset can be downloaded from the nuImage Dataset. We randomly select 800 images with semantic annotations and generate dense depth maps using a pretrained Depth Anything network. They can be downloaded from here.

📸 Checkpoints

The official weights of the teacher model, vanilla student model, and distillated student model can be downloaded from here.

🏋 Training

We provide training scripts for LIX. It can be trained on a single Nvidia 3090 GPU.

python train_student_lix.py --name lix_nuimg --dataset nuimg --dataroot /datasets --use_disp --teacher /checkpoints/teacher

🗣️ Acknowledgements

Our code is based on SNE-RoadSeg and DKD. We sincerely appreciate their amazing works.

This research was supported by the National Natural Science Foundation of China under Grant 62473288, the National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi'an Jiaotong University (No. HMHAI-202406), the Fundamental Research Funds for the Central Universities, NIO University Programme (NIO UP), and the Xiaomi Young Talents Program. The research leading to these results has also received partial funding from the European Commission - European Union (under HORIZON EUROPE (HORIZON Research and Innovation Actions) under grant agreement 101093003 (TEMA) HORIZON-CL4-2022-DATA-01-01). Views and opinions expressed are, however, those of the authors only and do not necessarily reflect those of the European Union - European Commission. Neither the European Commission nor the European Union can be held responsible for them.