What is the correct way to do DPO training on CHAT preference datasets?

[abhinand5]

Dataset Used: argilla/dpo-mix-7k

!! The fields are dict instead of strings. !!

So I have written a script to tokenize them and then save the tokenized prompts in a dataset, which I am trying to then load and do the training with custom dataset type.

rl: dpo
datasets:
  - path: dpo-mix-formatted.jsonl
    ds_type: json
    split: train
    type:
        field_prompt: prompt
        field_chosen: chosen
        field_rejected: rejected

The training is running currently, but I am not sure if this is the correct way to do this. Can anyone confirm?

[NanoCode012]

I haven't tried rlhf datasets yet. You may be able to run preprocess with --debug to see the processed outputs?