A curated list of balanced multimodal learning methods.
We have a discussion group about balanced multimodal learning. Anyone who may be interested is welcome! If the QR code expires, please add us on WeChat (ID: weike0409) and include a note saying “BML community.”
You are also welcome to join our Slack workspace: https://join.slack.com/t/balancedmulti-mhs9373/shared_invite/zt-3aj46comq-CNWrkqqGI9Z2kNn64aU9FQ
[CVPR-2022]
Balanced Multimodal Learning via On-the-fly Gradient Modulation
Code
[ICASSP-2023]
MMCosine: Multi-Modal Cosine Loss Towards Balanced Audio-Visual Fine-Grained Learning
Code
[ICLR-2024]
Quantifying and Enhancing Multi-modal Robustness with Modality Preference
Code
[CVPR-2024]
Enhancing Multimodal Cooperation via Sample-level Modality Valuation
Code
[ICML-2024]
MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance
Code
[ECCV-2024]
Diagnosing and Re-learning for Balanced Multimodal Learning
Code
[CVPR-2025]
Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisition
Code
- What Makes Training Multi-Modal Classification Networks Hard? [CVPR 2020]
- Learning to Balance the Learning Rates between Various Modalities via Adaptive Tracking Factor [IEEE Signal Processing Letters 2021]
- Audiovisual SlowFast Networks for Video Recognition [arXiv 2020]
- Joint Audio-Visual Deepfake Detection [ICCV 2021]
- Delving into Deep Imbalanced Regression [ICML2021]
- Characterizing and Overcoming the Greedy Nature of Learning in Multi-modal Deep Neural Networks [ICML 2022]
- MCL: A Contrastive Learning Method for Multimodal Data Fusion in Violence Detection [IEEE Signal Processing Letters 2022]
- Balanced MSE for Imbalanced Visual Regression[CVPR2022]
- On Uni-Modal Feature Learning in Supervised Multi-Modal Learning [ICML 2023]
- Calibrating Multimodal Learning [ICML 2023]
- Multimodal Pre-Training with Self-Distillation for Product Understanding in E-Commerce [WSDM 2023]
- PMR: Prototypical Modal Rebalance for Multimodal Learning [CVPR 2023]
- Graph Interactive Network with Adaptive Gradient for Multi-Modal Rumor Detection [ICMR 2023]
- Multimodal Imbalance-Aware Gradient Modulation for Weakly-supervised Audio-Visual Video Parsing [IEEE Transactions on Circuits and Systems for Video Technology 2023]
- Multimodal Temporal Attention in Sentiment Analysis [MuSe 2023]
- Utilizing Greedy Nature for Multimodal Conditional Image Synthesis in Transformers [TMM 2023]
- Variational Probabilistic Fusion Network for RGB-T Semantic Segmentation [arXiv 2023]
- Boosting Multi-modal Model Performance with Adaptive Gradient Modulation [ICCV 2023]
- Adaptive Mask Co-Optimization for Modal Dependence in Multimodal Learning [ICASSP 2024]
- Improving Multimodal Learning with Multi-Loss Gradient Modulation [arXiv 2024]
- Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworks [arXiv 2024]
- Suppress and Rebalance: Towards Generalized Multi-Modal Face Anti-Spoofing [CVPR 2024]
- C2KD: Bridging the Modality Gap for Cross-Modal Knowledge Distillation [CVPR 2024]
- Multimodal Representation Learning by Alternating Unimodal Adaptation [CVPR 2024]
- Balanced Multi-modal Federated Learning via Cross-Modal Infiltration [arXiv 2024]
- Balancing Multimodal Learning via Online Logit Modulation [IJCAI 2024]
- Enhancing Unimodal Features Matters: A Multimodal Framework for Building Extraction [TGRS 2024]
- Suppress and Rebalance: Towards Generalized Multi-Modal Face Anti-Spoofing [CVPR 2024]
- Understanding Unimodal Bias in Multimodal Deep Linear Networks [ICML 2024]
- Facilitating Multimodal Classification via Dynamically Learning Modality Gap [NIPS 2024]
- Ada2I: Enhancing Modality Balance for Multimodal Conversational Emotion Recognition [ACM MM 2024]
- Modality-Balanced Learning for Multimedia Recommendation [ACM MM 2024]
- ReconBoost: Boosting Can Achieve Modality Reconcilement. [ICML 2024]
- Classifier-guided Gradient Modulation for Enhanced Multimodal Learning [arXiv 2024]
- DynCIM: Dynamic Curriculum for Imbalanced Multimodal Learning [arXiv 2025]
- Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportion [arXiv 2025]
Methodology Classification (Refer to the paper BalanceBenchmark)
- Multimodal Representation Learning by Alternating Unimodal Adaptation [CVPR 2024]
- Characterizing and Overcoming the Greedy Nature of Learning in Multi-modal Deep Neural Networks [ICML 2022]
- Adaptive Mask Co-Optimization for Modal Dependence in Multimodal Learning [ICASSP 2024]
- MMCosine: Multi-Modal Cosine Loss Towards Balanced Audio-Visual Fine-Grained Learning [ICASSP 2023]
- On Uni-Modal Feature Learning in Supervised Multi-Modal Learning [ICML 2023]
- Multimodal Pre-Training with Self-Distillation for Product Understanding in E-Commerce [WSDM 2023]
- Calibrating Multimodal Learning [ICML 2023]
- MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance [ICML 2024]
- What Makes Training Multi-Modal Classification Networks Hard? [CVPR 2020]
- Facilitating Multimodal Classification via Dynamically Learning Modality Gap [NIPS 2024]
- Balanced Multimodal Learning via On-the-fly Gradient Modulation [CVPR2022]
- Boosting Multi-modal Model Performance with Adaptive Gradient Modulation [ICCV 2023]
- PMR: Prototypical Modal Rebalance for Multimodal Learning [CVPR 2023]
- Diagnosing and Re-learning for Balanced Multimodal Learning [ECCV2024]
- ReconBoost: Boosting Can Achieve Modality Reconcilement. [ICML 2024]