No EOS token?

[SmartWashingMachine]

Hi, thanks for the example. But is this part right?

            caption = item["text"]
            messages = [{"role": "user", "content": self.task_prompt}]
            text_prompt = self.processor.apply_chat_template(
                messages,
                tokenize=False,
                add_generation_prompt=True
            )
            full_text = text_prompt + caption

If we print out the full_text we get:
<|begin_of_sentence|>User: <|IMAGE_START|><|IMAGE_PLACEHOLDER|><|IMAGE_END|>OCR: Assistant: 素直にあやまるしかあやまるわよ！

But no <|end_of_sentence|> token? So model would learn to repeat tokens indefinitely?

Looking at their chat template it seems that <|end_of_sentence|> was not added because there's no assistant role message.

So maybe we need this instead:

                messages = [{"role": "user", "content": self.task_prompt}, {"role": "assistant", "content": caption, }]
                text_prompt = self.processor.apply_chat_template(
                    messages,
                    tokenize=False,
                    add_generation_prompt=False
                )
                full_text = text_prompt

Which prints:
<|begin_of_sentence|>User: <|IMAGE_START|><|IMAGE_PLACEHOLDER|><|IMAGE_END|>OCR: Assistant: 素直にあやまるしかあやまるわよ！<|end_of_sentence|>

Thank you again for this example. It is very helpful.

[SmartWashingMachine]

In case it helps anyone: There is another issue that causes the model to break during training for me.

None of the inputs are masked for the OCR task. This means that the model will try to "predict" all the prefixes like the image tokens. This causes the training loss and gradients to spike dramatically.

To train on responses only I added:

            # ... old code
            # Mask padding tokens in labels (don't compute loss on padding)
            inputs["labels"][inputs["attention_mask"] == 0] = -100

            # new additional code below:

            assistant_role_text = "OCR:\nAssistant:"

            role_token_ids = self.processor.tokenizer.encode(
                assistant_role_text,
                add_special_tokens=False,
                return_tensors="pt"
            ).squeeze().tolist()

            for idx in range(inputs["labels"].shape[0]):
                role_tag_length = len(role_token_ids)

                label_row = inputs["labels"][idx]
                label_list = label_row.tolist()

                # NOTE: May break if multiple prefixes found as this only finds first.

                caption_start_index = -1

                for j in range(len(label_list) - role_tag_length):
                    if label_list[j:j + role_tag_length] == role_token_ids:
                        caption_start_index = j + role_tag_length
                        break # Found

                if caption_start_index != -1:
                    # Mask everything up to caption
                    inputs["labels"][idx, :caption_start_index] = -100
                else:
                    # hopefully never happens
                    print(f"Warning: Assistant role tag not found for item {idx}. Skipping loss masking.")

also for those using gradient accumulation > 1, maybe modify the modeling_paddleocr_vl.py loss function to calculate num_items_in_batch. I do not know if Huggingface properly handles this off the bat:

        if labels is not None:
            # ... standard loss computation
            # ... Just change CrossEntropyLoss to use "sum"
            ignore_index = kwargs.get("ignore_index", -100)

            shift_labels = labels[..., 1:].contiguous()
            hidden_states_slice = hidden_states[..., :-1, :].contiguous()

            shift_labels = shift_labels.view(-1)
            shift_labels = shift_labels.to(hidden_states_slice.device)

            num_items_in_batch = kwargs.get("num_items_in_batch", None)
            reduction = "sum" if num_items_in_batch is not None else "mean"
            loss_fn = LigerFusedLinearCrossEntropyLoss(reduction=reduction, ignore_index=ignore_index)

            loss = loss_fn(
                self.lm_head.weight,
                hidden_states_slice.view(-1, hidden_states_slice.size(-1)),
                shift_labels,
                self.lm_head.bias
            )

            # ... important new code to work with grad acc
            if reduction == "sum":
                # just in case users pass an int for num_items_in_batch, which could be the case for custom trainer
                if torch.is_tensor(num_items_in_batch):
                    num_items_in_batch = num_items_in_batch.to(loss.device)

                loss = loss / num_items_in_batch

(I used Liger Kernel for memory savings but similar code can be adapted for standard CrossEntropy)

So far training is progressing more smoothly. Hopefully remains so.

EDIT: Training still breaks, but takes longer before it does. It learns to repeat tokens that were already generated for a while - unstable gradients?

EDIT 2: The pretrained model does not actually have an error - the pretrained model seems to use </s> for EOS during generation, but the EOS mentioned during training in the chat template is <|end_of_sentence|> .

I set the trained model to use <|end_of_sentence|> token instead with model.generate(**inputs, max_new_tokens=64, do_sample=False, temperature=0.01, use_cache=True, eos_token_id=100272) and now it works.

But does this mean PaddleVL uses </s> or <|end_of_sentence|> for EOS normally? Which one should we train with...?

I think PaddleVL uses </s> by default, so I will try training with that instead. This can be done by modifying the tokenizer_config.json line "sep_token": "<|end_of_sentence|>", to "sep_token": "</s>",

[jzhang533]

@SmartWashingMachine thanks a lot for sharing your findings!

I dig more into SFT PaddleOCR-VL-0.9B recently, the code I used is in wip branch in this repository, currently.

Some key points I found are

1. use completion_only loss, i.e.: "train on responses only“ as you did, I have a similar approach here: https://github.com/jzhang533/paddleocr-vl-sft/blob/wip/custom_collator.py
2. transformers/trl respect to chat_template.json and tokenizer_config.json, so I think it use <|end_of_sentence|> as EOS, but the pretrained model actually is using </s> token. For me, I didn't change any config of the released model, instead, when inference after SFT, I am using both <|end_of_sentence|> and </s> token as EOS, as shown here : https://github.com/jzhang533/paddleocr-vl-sft/blob/wip/use_transformers.py#L70 . It works for me.
3. don‘t use truncation when training, otherwise it may truncate vision tokens, this will cause training crash.

BTW: I shared a synthetic dataset on huggingface: https://huggingface.co/datasets/jzhang533/manga-synthetic, planned to consolidate the findings, and share a tutorial using this dataset on how to SFT this model using transformers and trl library.

[Sunting78]

Thank you for your attention. In PaddleOCR-VL, <|end_of_sentence|> is actually the sep token, and is the eos. It seems the chat_template use the <|end_of_sentence|>. We will fix it as soon as possible. Your addition of eos_token_id=100272 is helpful in this training code. It is also correct to mask the image tokens you mentioned in the training code.

[SmartWashingMachine]

Thanks for clarifying! This model is the best OCR I've seen yet. Last notes in case it helps anyone else:

I fine-tuned two variants, one with </s> as EOS and one with <|end_of_sentence|> as EOS. The one with </s> has a slightly lower initial loss, but both end up converging at similar values.

This model is extremely strong as-is, and also sensitive to the learning rate. in my experiments (even with warmup). LR 7e-5 had a moderately higher loss and over-fitting rate than lower ones like 4e-5. Keeping it at values similar to their technical report (e.g: 1e-6) may be better.