Example: Making fine-tuning LFM2.5-VL-1.6B work for datasets with system prompt and vision layer fine-tuning.

[urbanspr1nter]

See: unslothai/unsloth#3938

Currently, unsloth seems to be breaking when trying to fine tune LiquidAI/LFM2.5-VL-1.6B.

Reasons:

1. Chat template for the model assumes system prompt is a string, not a list. Normalization in vision utils will assume array and will break the collation process.
2. Vision layers cannot be fine tuned because the encoder is SigLIP2.

[gemini-code-assist]

Summary of Changes

Hello @urbanspr1nter, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves critical compatibility issues encountered when attempting to fine-tune the LiquidAI/LFM2.5-VL-1.6B model. It specifically enables the fine-tuning of models with SigLIP2 vision encoders by extending the recognized vision configurations and corrects an erroneous system prompt normalization behavior that previously broke the collation process for this model. These changes ensure a smoother and functional fine-tuning experience for the specified model.

Highlights

• SigLIP2 Vision Encoder Support: Added support for Siglip2VisionConfig within the PEFT utilities, enabling proper fine-tuning of models utilizing the SigLIP2 vision encoder, which was previously not recognized.
• System Prompt Normalization for LFM2.5-VL-1.6B: Implemented a fix in vision utilities to correctly normalize system prompts for Lfm2VlProcessor. This addresses an issue where string-based system prompts were being incorrectly processed as lists, causing collation failures during fine-tuning.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds support for fine-tuning the LFM2.5-VL-1.6B model by enabling vision layer fine-tuning for Siglip2VisionConfig and handling string-based system prompts for Lfm2VlProcessor. The changes are generally in the right direction. However, I've identified a critical logic bug in unsloth_zoo/vision_utils.py where a new if statement should be an elif. This could cause incorrect data processing. My review includes a suggestion to fix this issue.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 10f9775a4a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[Datta0]

Hey @urbanspr1nter does this only impact LFM? What is the root cause of the said behaviour?

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dbcd787afd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Guard the LFM2 processor import

This hard import makes unsloth_zoo.vision_utils fail to import in environments where transformers does not export Lfm2VlProcessor (which is plausible given the repo’s wide supported transformers range), and that breaks all vision collation paths, not just LFM2 fine-tuning. A missing symbol here raises ImportError at module import time, so users cannot even run unaffected models unless this is wrapped in a compatibility guard (for example try/except with a fallback type check by class name).

Useful? React with 👍 / 👎.

[urbanspr1nter]

Hey @urbanspr1nter does this only impact LFM? What is the root cause of the said behaviour?

@Datta0 - I think this is most notable thing is the Siglip2VisionConfig addition. I have only tested this with LFM, but I would now assume model which system prompt isn't expected to be normalized in the way unsloth will do would be affected too.