Allow using PEFT (e.g. LoRA) in mask decoder with frozen image encoder

[YonghaoZhao722]

Hi, thank you for the great work on micro-sam!

Currently, the function get_trainable_sam_model raises an error when trying to use PEFT together with freeze_image_encoder=True:

raise ValueError("You cannot use PEFT & freeze the image encoder at the same time.")

However, in many fine-tuning scenarios (especially with limited data or compute), it's common to freeze the encoder and apply LoRA or other PEFT methods only to the decoder.

Would it be possible to:

• either remove this restriction, or
• allow it with a warning, leaving the responsibility to the user?

Thanks again for developing this tool!

[anwai98]

Hi @YonghaoZhao722,

Thanks for your interest in micro-sam!

Currently, the function get_trainable_sam_model raises an error when trying to use PEFT together with freeze_image_encoder=True:

That's right. Since the freezing and LoRA, both are currently only applied to the image encoder, it blocks you from doing so!

However, in many fine-tuning scenarios (especially with limited data or compute), it's common to freeze the encoder and apply LoRA or other PEFT methods only to the decoder.

That's right too. However, our current implementation does not apply any PEFT methods to the mask decoder.

If you'd like to use it, you would need to add a new PEFT-specific class, compatible with PEFT_Sam (https://github.com/computational-cell-analytics/micro-sam/blob/dev/micro_sam/models/peft_sam.py#L393) to use PEFT methods for the mask decoder!

[YonghaoZhao722]

Hi @anwai98, thanks for the clarification!

I’m interested in extending the current implementation by adding a PEFT-specific class for applying LoRA (or other PEFT methods) to the mask decoder while keeping the image encoder frozen, as you suggested.

If I implement this functionality in a compatible way with the current PEFT_Sam design, would you be open to reviewing and potentially merging it into the main repository?

Thanks!

[anwai98]

Hi @YonghaoZhao722,

Thanks for pitching it. I'd be happy to check out your PR.

Since you are already planning a try-out, I'll leave a few tips on how to go ahead with it:

• Create a new branch out of the latest dev branch and create the PR on dev only (we strictly follow a development cycle for micro-sam, atleast for things planned as proposed by you)
• Make sure to follow common python linting guidelines, and stick to adapting current LoRA_Surgery to the decoder inside PEFT_Sam (off the top of my head, this shouldn't be a lot of code required) and add an argument in a function signature which allows to use lora for decoder!

And once you are at this stage, we can iterate a bit via your PR to make this compatible with micro-sam!

Let me know if something's unclear (I'll reopen this issue so that we can continue any relevant discussions)

[YonghaoZhao722]

Hi @anwai98,

Just created a PR 6fa5956. Is it on dev ?

[anwai98]

Hi @YonghaoZhao722,

I switched it to dev. Will take a quick look at it in some!