Dataset format for multi-turn conversations in DPO

[Kshitiz-Khandel]

Can you help on how to format the dataset for multi-turn dpo? Below is an example of the first record from my dataset containing three keys: system,chosen and rejected

{'system': 'You are a helpful and knowledgeable AI assisting in patient-doctor conversations.',
'chosen': [{'role': 'assistant',
'content': 'Hello, I am Dr. Smith. Can you tell me what brings you to the hospital today?'},
{'role': 'user',
'content': 'Yes, I have been feeling very weak and sick for the past two weeks. I have a persistent fever and dry cough.'},
{'role': 'assistant', 'content': 'I see. And how is your breathing?'},
{'role': 'user',
'content': "It's been shallow and rapid, especially when I am at rest. And I get severely breathless even with minor physical activities."},
{'role': 'assistant',
'content': 'Okay. I understand. You were given physical therapy, right?'},
{'role': 'user',
'content': 'Yes, they focused on educating me about dyspnea-relieving positions and the importance of regular mobilization and deep-breathing exercises.'}],
{'role': 'assistant', 'content': "That's good. },
'rejected': [{'role': 'assistant',
'content': "Hey, I'm Dr. Smith. What's up?"},
{'role': 'user',
'content': 'Yes, I have been feeling very weak and sick for the past two weeks. I have a persistent fever and dry cough.'},
{'role': 'assistant', 'content': 'So, breathing okay?'},
{'role': 'user',
'content': "It's been shallow and rapid, especially when I am at rest. And I get severely breathless even with minor physical activities."},
{'role': 'assistant', 'content': 'Right. You had physical therapy, yeah?'},
{'role': 'user',
'content': 'Yes, they focused on educating me about dyspnea-relieving positions and the importance of regular mobilization and deep-breathing exercises.'},
{'role': 'assistant', 'content': "That is good"},
}]}

[NanoCode012]

Hello, thanks for the question.

For DPO, the dataset is usually like so https://axolotl-ai-cloud.github.io/axolotl/docs/rlhf.html#chat_template.default

It would only have the last turn being different for the model to be able to choose chosen/rejected, not multi-turn difference.

Given your dataset, I would advise converting it into ones like above (splitting your dataset into smaller sections with the last message being the different one) or doing SFT instead.

[Kshitiz-Khandel]

Thank you! This helps

[ehartford]

Basically split the conversation into turns then for each turn have the history as the question and the last response as the answer

[Kshitiz-Khandel]

Hey @NanoCode012 .I wanted to clarify something regarding the dpo_dataset_format, Is it acceptable for the messages field to contain a full multi-turn conversation, with alternating messages between the user and the assistant, effectively capturing the full dialogue history?
From the documentation example, it seems like only user messages are included under the messages key, but I wanted to confirm whether assistant turns are also supported there. If not can you help me clarify how to prepare the dataset for multi turn chat?

{
"messages": [
{"role": "system", "content": "..."},
{"role": "user", "content": "..."},
{"role": "assistant", "content": "..."},
{"role": "user", "content": "..."},
{"role": "assistant", "content": "..."},
{"role": "user", "content": "..."}
],
"chosen": {"role": "assistant", "content": "..."},
"rejected": {"role": "assistant", "content": "..."}
}

[NanoCode012]

Is it acceptable for the messages field to contain a full multi-turn conversation, with alternating messages between the user and the assistant, effectively capturing the full dialogue history?

Yes :) I didn't make it too long in the docs as it was just a sample. You can do multiple turns with last assistant turn being in chosen/rejected.

[Kshitiz-Khandel]

Thank you so much!

Title: ValueError: num_samples=0 with custom .jsonl dataset for DPO + Deepspeed Training

I'm trying to fine-tune google/gemma-3-27b-it using Direct Preference Optimization (DPO) with my custom .jsonl dataset. However, I’m running into the following error at training time:

ValueError: num_samples should be a positive integer value, but got num_samples=0
File "/home/jupyter/DPO/axolotl/src/axolotl/train.py", line 198, in execute_training
trainer.train(resume_from_checkpoint=resume_from_checkpoint)
...
File "/torch/utils/data/sampler.py", line 165, in init
raise ValueError(
ValueError: num_samples should be a positive integer value, but got num_samples=0

## Below is the config i am using:

base_model: google/gemma-3-27b-it
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: True
load_in_4bit: False

chat_template: gemma3
rl: dpo
datasets:

• path: /home/jupyter/DPO/axolotl/examples/gemma3/axolotl_train.jsonl
  type: chat_template.default
  field_messages: "messages"
  field_chosen: "chosen"
  field_rejected: "rejected"
  message_property_mappings:
  role: role
  content: content
  roles:
  user: ["user"]
  assistant: ["assistant"]
  system: ["system"]

dataset_prepared_path:
val_set_size: 0.05
output_dir: ./outputs/dpo2/gemma3/exp1/

sequence_len: 256
sample_packing: false
pad_to_sequence_len: true

adapter: lora
lora_model_dir:
lora_r: 16
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 10
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002

bf16: auto
tf32: false

gradient_checkpointing: true
resume_from_checkpoint:
logging_steps: 1
flash_attention: true
deepspeed: deepspeed_configs/zero3.json

warmup_steps: 10
evals_per_epoch: 4
saves_per_epoch: 1
weight_decay: 0.0
shuffle: True

What I’ve tried:
Confirmed that the .jsonl file exists at the correct path and is non-empty.

Ask:

1. Could you help me identify the reason why the issue is persisting?

2. Is this configuration optimal to leverage Deepspeed (Zero3) with 8×A100 40GB for DPO tuning since I am training a large model?

[NanoCode012]

This may make more sense as a separate discussion, but I noticed your seq length being quite small. Could those samples have been dropped?

[NanoCode012]

Is this configuration optimal to leverage Deepspeed (Zero3) with 8×A100 40GB for DPO tuning since I am training a large model?

Not by itself. Enable the configs below for max improvements:

plugins:
  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
  - axolotl.integrations.liger.LigerPlugin

cut_cross_entropy: true

liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_layer_norm: true

gradient_checkpointing: "offload"

lora_mlp_kernel: true
lora_qkv_kernel: true
lora_o_kernel: true

We have an upcoming writeup on this.