Clarification on Reconstruction Loss Implementation

[fahadskhan]

Hi, thanks a lot for your amazing work and for releasing this codebase!

I was going through the paper and the codebase, but I wasn’t able to locate the implementation of "Self-supervised Reconstruction Loss" in the repository. Could you please point me to the specific part of the code where this loss is defined, or clarify if it is included under a different name?

Thanks in advance for your help!

[HorizonWind2004]

Hi! BAGEL's reconstruction loss is implemented on BAGEL/data/t2i_dataset.py.

                    sequence_plan.append({
                        'type': 'vit_image',
                        'enable_cfg': 0,
                        'loss': 0,
                        'special_token_loss': 0,
                        'special_token_label': None,
                    })
                    
                    # Add text processing plan (prompt part)
                    sequence_plan.append({
                        'type': 'text',
                        'enable_cfg': 1,
                        'loss': 0,
                        'special_token_loss': 0,
                        'special_token_label': None,
                    })
                    
                    # Add image processing plan (reconstruction target)
                    sequence_plan.append({
                        'type': 'vae_image',
                        'enable_cfg': 0,
                        'loss': 1,
                        'special_token_loss': 0,
                        'special_token_label': None,
                    })

[HorizonWind2004]

Harmon's loss is implemented in Harmon/src.models/harmon_dev.py.

    def recon_loss(self, data_dict):
        ....

If it helps, please consider giving us a star~🌟