Custom prompt encoder for soft prompt tuning (P-tuning)

[mmiakashs]

Hi Everyone,
I wanted to implement a custom prompt encoder to learn some soft prompt embedding which will pass thorough the pre-trained LLM along with input. Is there any resources or example script to implement a custom prompt encoder for PEFT base model to extract soft prompt embedding?
Any resources and suggestions are highly appreciated. Thanks.

[BenjaminBossan]

To clarify, do you mean you want to use the existing technique from PEFT and are looking for guidance how to use it or are you asking about a custom implementation that differs from what PEFT currently offers?

[mmiakashs]

I want to custom implement a prompt encoder. I think PEFT has a pre-defined prompt encoder. I was wondering is there any way overload/replace the existing prompt encoder by a custom prompt encoder. Specifically, I was thinking to override the implementation of PromptEncoder

[BenjaminBossan]

No, I believe it's not possible. You would have to adapt/patch the PEFT code itself to make such a change. If your method is something of more general interest (e.g. a based on a paper with good results), you could consider creating a PR to add the new method, we could help you with the integration.

[kanseaveg]

mark, anything update?