DPO doesn't work with inference

[cinjon]

I've trained two models, one DPO and the other SFT. When I run inference on the first with python -m axolotl.cli.inference /path/to/sft/yml --base_model=/path/to/sft/model-dir, it works no problem.

When I run inference on the second with python -m axolotl.cli.inference /path/to/dpo/yml --base_model=/path/to/dpo/model-dir, it fails by just outputting @nate over and over again.

First:

<|begin_of_text|><|start_header_id|>user<|end_header_id|>

I'm going to Paris tomorrow.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Paris, the City of Light! How exciting! What are your plans while you're there? Have you been before, or is this your first time visiting?<|eot_id|>

Second:

<|begin_of_text|><|start_header_id|>user<|end_header_id|>

I'm going to Paris tomorrow.<|eot_id|><|start_header_id|>assistant<|end_header_id|>@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate@nate...

[winglian]

Can you share the yaml files you used for each model you trained?

[cinjon]

Here's an example sft one:

base_model: meta-llama/Meta-Llama-3-8B-Instruct
bf16: auto
counter: 3
data_seed: 1
datasets:
- path: /path/to/local/dataset.jsonl
  type: input_output
debug: null
deepspeed: null
early_stopping_patience: null
eval_max_new_tokens: 128
eval_table_size: null
evals_per_epoch: 2
flash_attention: true
fp16: null
fsdp: null
fsdp_config: null
gradient_accumulation_steps: 4
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
group_by_length: false
learning_rate: 2e-5
load_in_4bit: false
load_in_8bit: false
logging_steps: 1
lr_scheduler: cosine
micro_batch_size: 1
model_type: LlamaForCausalLM
num_epochs: 32
optimizer: paged_adamw_8bit
output_dir: /path/to/results/output/dir
pad_to_sequence_len: true
resume_from_checkpoint: null
sample_packing: false
saves_per_epoch: 1
seed: 1
sequence_len: 8192
special_tokens:
  pad_token: <|end_of_text|>
strict: false
tf32: false
tokenizer_type: AutoTokenizer
train_on_inputs: false
val_set_size: 0
wandb_entity: entityname
wandb_log_model: null
wandb_name: wandbname
wandb_project: projectname
wandb_run_id: null
wandb_watch: null
warmup_steps: 100
weight_decay: 0.0
xformers_attention: null

Here's an example dpo one:

base_model: meta-llama/Meta-Llama-3-8B-Instruct
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer
load_in_4bit: false
load_in_8bit: false
strict: false
output_dir: /path/to/results/output/dir
val_set_size: 0
sequence_len: 8192
sample_packing: false
pad_to_sequence_len: true
wandb_entity: entityname
wandb_log_model:
wandb_name: wandbname
wandb_project: projectname
wandb_run_id:
wandb_watch:
gradient_accumulation_steps: 8
micro_batch_size: 1
num_epochs: 1
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 2e-5
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_steps: 100
evals_per_epoch: 1
eval_table_size:
saves_per_epoch: 6
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
  pad_token: <|end_of_text|>
data_seed: 1
eval_max_new_tokens: 128
seed: 1
rl: dpo
datasets:
- path: /path/to/local/huggingface/dataset
  split: train
  type:
    field_prompt: prompt
    field_chosen: chosen
    field_rejected: rejected_key
    prompt_format: '{prompt}'
    chosen_format: '{chosen}'
    rejected_format: '{rejected}'
counter: 12
reject_key: rejected_key