Hi @anwai98, I have something to discuss.

Summary

Add LoRA support for SAM decoder transformer layers with decoder_lora=True parameter.

Changes

• Add SimpleLoRALayer for individual linear layer LoRA adaptation
• Add decoder_lora parameter to PEFT_Sam.__init__()
• Apply LoRA to decoder: self-attention, cross-attention, MLP layers

SimpleLoRALayer

Why I added SimpleLoRALayer instead of adapting LoRASurgery:

• Encoder: Uses combined qkv = nn.Linear(dim, dim*3) → splits output into q,k,v
• Decoder: Uses separate q_proj, k_proj, v_proj layers

Current LoRASurgery/AttentionLoRA (encoder):

# One projection layer for all Q, K, V
qkv = self.qkv_proj(x)
qkv = torch.cat([
    qkv[:, :, :, :self.dim] + new_q,
    qkv[:, :, :, self.dim:-self.dim] + new_k,
    qkv[:, :, :, -self.dim:] + new_v
], dim=-1)

Decoder needs (separate projections):

q = self.q_proj(x) + lora_q(x)
k = self.k_proj(x) + lora_k(x)
v = self.v_proj(x) + lora_v(x)

That's why I add a SimpleLoRALayer. What do you think?