Add LoRA linear definition #11044

lucylq · 2025-05-21T17:53:49Z

Stack from ghstack (oldest at bottom):

-> Add LoRA linear definition #11044

^
Add lora linear definition. Pull out linears from attention, and allow custom linear (eg. lora linear) to be passed in.

If none, construct linear (current behaviour).

Differential Revision: D73953776

^ Add lora linear definition. Pull out linears from attention, and allow custom linear (eg. lora linear) to be passed in. If none, construct linear (current behaviour). Differential Revision: [D73953776](https://our.internmc.facebook.com/intern/diff/D73953776/) [ghstack-poisoned]

pytorch-bot · 2025-05-21T17:53:52Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/11044

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 2 Unrelated Failures

As of commit 180fe7f with merge base 8da2ea6 ():

NEW FAILURES - The following jobs have failed:

pull / unittest / linux / linux-job (gh)
RuntimeError: Command docker exec -t fe5e1723b762aaeee0a961940d4bb686c09429802ff57dfa52ff762b28b5bc2d /exec failed with exit code 5
pull / unittest-editable / linux / linux-job (gh)
RuntimeError: Command docker exec -t c3a16bd211651c7e7e290c7813fb45d6cadb37f08d8118e9ee11f15669e85622 /exec failed with exit code 5

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / unittest / macos / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 5
pull / unittest-editable / macos / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 5

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2025-05-21T17:53:59Z

This pull request was exported from Phabricator. Differential Revision: D73953776

^ Add lora linear definition. Pull out linears from attention, and allow custom linear (eg. lora linear) to be passed in. If none, construct linear (current behaviour). Differential Revision: [D73953776](https://our.internmc.facebook.com/intern/diff/D73953776/) ghstack-source-id: 285402969 Pull Request resolved: pytorch/executorch#11044

^ Add lora linear definition. Pull out linears from attention, and allow custom linear (eg. lora linear) to be passed in. If none, construct linear (current behaviour). Differential Revision: [D73953776](https://our.internmc.facebook.com/intern/diff/D73953776/) [ghstack-poisoned]

facebook-github-bot · 2025-07-10T18:09:24Z

This pull request was exported from Phabricator. Differential Revision: D73953776

examples/models/llama/llama_transformer.py

JacobSzwejbka · 2025-07-18T17:03:43Z

examples/models/llama/llama_transformer.py

+            # Static attention constructs ModuleLists for qkvo and
+            # populates them with MHA attention layers; do not pass in
+            # wq, wk, wv, wo.
+            attention = cls(model_args, layer_id, rope)


Why are you doing this in llama_transformer at all? You have model args passed to the constructor couldnt you just construct these tensors in the constructor rather then passing them to init?

Not having to add attention specific logic to llama_transformer.py is the intention of the attention registry, so if we have to break that abstraction I think we need to elaborate why

The intention is to make building a transformer composable, e.g., you can pass in any type of attention, with any type of linears.

If construct them inside the transformer constructor, then the transformer has to understand what all the different options are. i.e., we didn't want to put LoRA logic inside MHA attention itself, as it isn't tied to MHA specifically.

Attention already has to understand what all the different options are. Thats why model args is passed into it no?

Attention already has to understand what all the different options are. Thats why model args is passed into it no?

As in its already possible to generate invalid configs where someone sticks model options in and then uses an attention that doesnt support those options.

LoRA specifically doesn't have to be coupled with the attention implementation (eg. MHA). I can see why it makes sense to do it inside if other optimizations are also applied within eg. MHA though ..

@JacobSzwejbka my original concern on this is that model config might be non-standardized. Now that we have Jack's new config system maybe this is less of a concern.

This set up is already dependent on model config though. The lora constructors use tons of args from the config

^ Add lora linear definition. Pull out linears from attention, and allow custom linear (eg. lora linear) to be passed in. If none, construct linear (current behaviour). Differential Revision: [D73953776](https://our.internmc.facebook.com/intern/diff/D73953776/) [ghstack-poisoned]

Pull Request resolved: #11044 ^ Add lora linear definition. Pull out linears from attention, and allow custom linear (eg. lora linear) to be passed in. If none, construct linear (current behaviour). ghstack-source-id: 297800340 @exported-using-ghexport Differential Revision: [D73953776](https://our.internmc.facebook.com/intern/diff/D73953776/)

facebook-github-bot · 2025-07-22T17:29:22Z

This pull request was exported from Phabricator. Differential Revision: D73953776

^ Add lora linear definition. Pull out linears from attention, and allow custom linear (eg. lora linear) to be passed in. If none, construct linear (current behaviour). Differential Revision: [D73953776](https://our.internmc.facebook.com/intern/diff/D73953776/) [ghstack-poisoned]

facebook-github-bot · 2025-07-22T17:51:31Z

This pull request was exported from Phabricator. Differential Revision: D73953776

^ Add lora linear definition. Pull out linears from attention, and allow custom linear (eg. lora linear) to be passed in. If none, construct linear (current behaviour). Differential Revision: [D73953776](https://our.internmc.facebook.com/intern/diff/D73953776/) [ghstack-poisoned]

facebook-github-bot · 2025-07-23T23:57:33Z

This pull request was exported from Phabricator. Differential Revision: D73953776

^ Add lora linear definition. Pull out linears from attention, and allow custom linear (eg. lora linear) to be passed in. If none, construct linear (current behaviour). Differential Revision: [D73953776](https://our.internmc.facebook.com/intern/diff/D73953776/) [ghstack-poisoned]

Pull Request resolved: #11044 ^ Add lora linear definition. Pull out linears from attention, and allow custom linear (eg. lora linear) to be passed in. If none, construct linear (current behaviour). ghstack-source-id: 298402065 @exported-using-ghexport Differential Revision: [D73953776](https://our.internmc.facebook.com/intern/diff/D73953776/)

facebook-github-bot · 2025-07-24T20:57:08Z

This pull request was exported from Phabricator. Differential Revision: D73953776

^ Add lora linear definition. Pull out linears from attention, and allow custom linear (eg. lora linear) to be passed in. If none, construct linear (current behaviour). Differential Revision: [D73953776](https://our.internmc.facebook.com/intern/diff/D73953776/) [ghstack-poisoned]

Pull Request resolved: #11044 ^ Add lora linear definition. Pull out linears from attention, and allow custom linear (eg. lora linear) to be passed in. If none, construct linear (current behaviour). ghstack-source-id: 298411928 @exported-using-ghexport Differential Revision: [D73953776](https://our.internmc.facebook.com/intern/diff/D73953776/)

facebook-github-bot · 2025-07-24T21:28:26Z

This pull request was exported from Phabricator. Differential Revision: D73953776

@lucylq

This PR was created by the merge bot to help merge the original PR into the main branch. ghstack PR number: #11044 by @lucylq ^ Please use this as the source of truth for the PR details, comments, and reviews ghstack PR base: https://github.com/pytorch/executorch/tree/gh/lucylq/82/base ghstack PR head: https://github.com/pytorch/executorch/tree/gh/lucylq/82/head Merge bot PR base: https://github.com/pytorch/executorch/tree/main Merge bot PR head: https://github.com/pytorch/executorch/tree/gh/lucylq/82/orig @diff-train-skip-merge Co-authored-by: lucylq <[email protected]>

lucylq requested a review from jackzhxng as a code owner May 21, 2025 17:53

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label May 21, 2025

lucylq mentioned this pull request May 21, 2025

Export a lora model #11045

Merged

facebook-github-bot added the fb-exported label May 21, 2025

lucylq added the release notes: llm Changes to llm utilities label May 21, 2025

lucylq mentioned this pull request Jul 15, 2025

Export and run lora llama3b with adapter weights #9580

Open

jackzhxng approved these changes Jul 15, 2025

View reviewed changes

examples/models/llama/llama_transformer.py Outdated Show resolved Hide resolved

JacobSzwejbka requested changes Jul 18, 2025

View reviewed changes

JacobSzwejbka approved these changes Jul 22, 2025

View reviewed changes

lucylq mentioned this pull request Jul 22, 2025

CI test for lora #12724

Merged

lucylq added 6 commits July 24, 2025 12:37

facebook-github-bot merged commit db8729a into gh/lucylq/82/base Jul 25, 2025
95 of 101 checks passed

facebook-github-bot deleted the gh/lucylq/82/head branch July 25, 2025 18:17

facebook-github-bot temporarily deployed to cherry-pick-bot July 25, 2025 18:17 — with GitHub Actions Inactive

pytorchbot mentioned this pull request Jul 25, 2025

Add LoRA linear definition #12861

Merged

Add LoRA linear definition #11044

Add LoRA linear definition #11044

Uh oh!

Conversation

lucylq commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/11044

❌ 2 New Failures, 2 Unrelated Failures

Uh oh!

facebook-github-bot commented May 21, 2025

Uh oh!

facebook-github-bot commented Jul 10, 2025

Uh oh!

Uh oh!

JacobSzwejbka Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lucylq Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JacobSzwejbka Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

JacobSzwejbka Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

lucylq Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

larryliu0820 Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

JacobSzwejbka Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Jul 22, 2025

Uh oh!

facebook-github-bot commented Jul 22, 2025

Uh oh!

facebook-github-bot commented Jul 23, 2025

Uh oh!

facebook-github-bot commented Jul 24, 2025

Uh oh!

facebook-github-bot commented Jul 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

lucylq commented May 21, 2025 •

edited

Loading

pytorch-bot bot commented May 21, 2025 •

edited

Loading

JacobSzwejbka Jul 18, 2025 •

edited

Loading

lucylq Jul 22, 2025 •

edited

Loading