Got Error on Fine-tuning TrOCR with LoRA

[bustamiyusoef]

I'm Fine-tuning TrOCR with LoRA, here's how I do it

model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-small-stage1")

config = LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
bias="none",
target_modules=["query","value"],
)

model = get_peft_model(model, config)
print_trainable_parameters(model)

I got:
trainable params: 294912 || all params: 61891584 || trainable %: 0.4764977415992456

it seems LoRA successfully injected to the Model.
When I trained the model using this way:

training_args = Seq2SeqTrainingArguments(
predict_with_generate=True,
evaluation_strategy="steps",
per_device_train_batch_size=8,
per_device_eval_batch_size=8,
fp16= True, # Set to False, if GPU not available
output_dir="./",
logging_steps=2,
save_steps=1000,
eval_steps=200,
num_train_epochs=3,
)

trainer = Seq2SeqTrainer(
model=model,
tokenizer=processor.feature_extractor,
args=training_args,
compute_metrics=compute_metrics,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
data_collator=default_data_collator,
)
trainer.train()

I got error like this:

'ValueError: The batch received was empty, your model won't be able to train on it. Double-check that your training dataset contains keys expected by the model: args,kwargs,label_ids,label.'

anyone can help?

[BenjaminBossan]

If your hardware allows it, could you check if you get the same error without PEFT? I.e. just doing a full fine-tune of the base model? You could try a lower batch size, SGD instead of Adam etc. to save on memory if necessary. The reason why I ask is because this looks like an issue with the dataset, not with PEFT.