Whether to padding on right during evaluation process

[LieZiWind]

When I set do_causal_lm_eval: true and eval_causal_lm_metrics: ['chrf'] and eval_sample_packing: false in the config file, I kept get warnings about padding.

warning：A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set padding_side='left' when initializing the tokenizer.

Normally we train with right padding and inference with left padding, so I wonder if this is an issue.
I also try to fix this by using a new data collator called "LeftCollator" that explicitly set padding to left when processing the evaluation dataset.
Specifically, in /src/axolotl/core/trainer_builder.py, I set the collator to the "LeftCollator" when is_eval is true. I wonder if this is the correct practice.

[NanoCode012]

Hmm, thanks for lifting up a good point. I'm not sure as well how to best handle this. Another alternative is to temporarily set the tokenizer padding. Does lm_eval throw any more warnings when you set that collator?

If anyone coming across knows how to deal with this, do let us know!

[LieZiWind]

Hi, thanks for replying. I've been digging this for a while. I also tried to manually set the tokenizer padding size during the eval stage, but I am concerned that in a distributed training setting this would cause some synchronization problems. (Not sure though) That's why I try to customize a data collator.

After some investigation, it appears setting the collator does work. However, extra modifications are required. In fact, when eval_sample_packing == False, get_eval_dataloader() of the AxolotlTrainer will resort to super().get_eval_dataloader(eval_dataset) , which automatically uses the self.data_collator instead of self.eval_data_collator. After handling this, it seems to work. I manually checked the generated ids, outputs etc. and everything seems fine.

[NanoCode012]

How did you verify the outputs to be correct? Do you just see whether the padding output is always on left during eval but right during train?

[LieZiWind]

Yes that’s basically what I've checked. The decoded outputs also align with my fine-tune target.

Are there other stuffs that I should be concerned of?

[NanoCode012]

Do you only set it, for evaluating lm_eval or general eval? If the latter, do you perhaps have the eval loss comparisons ? If you're interested, you can make a draft PR on this.