QLoRa and prompting combination

[cmarquesdasilva]

It is possible combines the two tecniques in order to train the virtual tokens with LoRa weights together?

I think this might work. Any thoughts?

peft_config = PromptTuningConfig(
    task_type=TaskType.CAUSAL_LM,
    prompt_tuning_init=PromptTuningInit.TEXT,
    num_virtual_tokens=128,
    prompt_tuning_init_text="Translate sentence",
    tokenizer_name_or_path=model_id,
)

prompting_model = get_peft_model(model, peft_config, adapter_name='prompting')

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["query_key_value"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

lora = inject_adapter_in_model(peft_config=lora_config, model=prompting_model)

[BenjaminBossan]

I see no principal reason why it cannot work. Did you try?