Adding custom MoE-PEFT adapter

[EomSooHwan]

Hi, thank you for sharing this repo. It has been really helpful and well-organized.

I'm currently trying to implement and experiment with my own MoE-PEFT adapter idea, and I thought this codebase would be a great starting point. I’ve followed the existing adapter structure and made the necessary additions to integrate a custom adapter, but I'm running into an issue where it seems like the adapter is not being executed at all.

What I’ve done so far:

• Created custom_adapter folder under moe_peft/adapters
• Added model.py with CustomAdapter class and CustomAdapterLoss class under moe_peft/adapters/custom_adpater
• Added config.py and __init__.py under moe_peft/adapters/custom_adpater according to my model
• Registered CustomAdapter class, CustomAdapterLoss class, and CustomAdapterConfig class in moe_peft/adapters/__init__.py
• Add custom_adapter_template.json file under templates folder

How I’m launching it:

python ./launch.py gen \
    --template custom_adapter \
    --tasks "arc-c;arc-e;boolq;obqa;piqa" \
    --multi_task True \
    --adapter_name custom_adapter

python moe_peft.py \
    --base_model /path/to/Llama-2-7b-hf \
    --config moe_peft.json

The Issue:

Despite setting a breakpoint with ipdb.set_trace() inside my model.py, it doesn't seem to trigger, which leads me to believe that the custom_adapter is not being used at all during execution.

Request:

Could you please provide a brief guideline or checklist on how to properly register and invoke a custom adapter within this framework?

In particular:

• How is the adapter selected and instantiated at runtime?
• Is there any additional step I might have missed beyond what I listed?
• How can I verify that my custom adapter is indeed being used?

Thanks again for your work on this project. Looking forward to your guidance!