Export LLM documentation revamp #12381

jackzhxng · 2025-07-10T23:44:34Z

Summary

Structure:

New Getting Started page
AOT (export)
- Old getting started page, which was the NanoGPT tutorial, is moved to export-custom-llm.md, with the runner sections removed to add to the run-with-c-plus-plus.md
- New export-llm.md page for exporting LLMs with export_llm API
Runtime
- iOS/Android app docs remain, they detail steps to take after the .pte is generated for running on-device
- Added a C++ runner page for @larryliu0820 to fill out with the new runner APIs
- Since the QNN Llama tutorial is highly custom, we are going to leave the export section in it as well instead of dividing like we did for the rest of the tutorials

pytorch-bot · 2025-07-10T23:44:38Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12381

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit ff92c9c with merge base dd4488d ():

NEW FAILURES - The following jobs have failed:

Lint / lintrunner / linux-job (gh)
>>> Lint for docs/source/llm/run-on-ios.md:
pull / test-eval_llama-mmlu-linux / linux-job (gh)
RuntimeError: Command docker exec -t 6d0c28fcc6d8124454177c11cd6f36f55b8e2854cf73b783f3a1b824b9273b7e /exec failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2025-07-10T23:45:13Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

docs/source/llm/export-custom-llm.md

docs/source/llm/export-llm.md

shoumikhin · 2025-07-16T23:21:54Z

Please make a placeholder for ObjC/Swift runtime APIs and i'll fill it out

jackzhxng · 2025-07-16T23:42:30Z

@shoumikhin I have left the existing llama-demo-ios.md untouched and linked to it from getting-started.md, feel free to make any changes to llama-demo-ios.md that you need

mergennachin

export-llm is a great page, thank you for working on this

docs/source/index.md

docs/source/llm/getting-started.md

docs/source/llm/export-llm.md

shoumikhin · 2025-07-18T23:16:20Z

Instead of llama-demo-ios.md we should probably just point to examples/demo-apps/apple_ios/LLaMA/README.md?
I meant some place in the docs where we describe the LLM runtime API in C++ and how to use it. I can append some info on ObjC/Swift API there.

jackzhxng · 2025-07-19T00:22:33Z

@shoumikhin sure, feel free to open a PR against this!

docs/source/index.md

digantdesai · 2025-07-21T11:36:26Z

docs/source/llm/export-custom-llm.md

+
+Create a file called export_nanogpt.py with the following contents:
+
+```python


so much code (I know you are just moving here) in an .md is just waiting to get outdated. Is there an alternative like a notebook or something (which is also in CI)?

Yeah, it'd be good to put into a notebook maybe

docs/source/llm/run-with-c-plus-plus.md

Co-authored-by: Anthony Shoumikhin <[email protected]>

### Summary [PLEASE REMOVE] See [CONTRIBUTING.md's Pull Requests](https://github.com/pytorch/executorch/blob/main/CONTRIBUTING.md#pull-requests) for ExecuTorch PR guidelines. [PLEASE REMOVE] If this PR closes an issue, please add a `Fixes #<issue-id>` line. [PLEASE REMOVE] If this PR introduces a fix or feature that should be the upcoming release notes, please add a "Release notes: <area>" label. For a list of available release notes labels, check out [CONTRIBUTING.md's Pull Requests](https://github.com/pytorch/executorch/blob/main/CONTRIBUTING.md#pull-requests). ### Test plan [PLEASE REMOVE] How did you test this PR? Please write down any manual commands you used and note down tests that you have written if applicable.

jackzhxng · 2025-07-23T19:39:48Z

@pytorchbot cherry-pick --onto release/0.7 -c docs

(cherry picked from commit 01b7e7d)

pytorchbot · 2025-07-23T19:42:02Z

Cherry picking #12381

The cherry pick PR is at #12757 The following tracker issues are updated:

[v0.7.0] Release Tracker #11075 (comment)

Details for Dev Infra team

Raised by workflow job

Merge #12381 from side branch into main. --------- Co-authored-by: Gasoonjia <[email protected]> Co-authored-by: Mergen Nachin <[email protected]>

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 10, 2025

jackzhxng force-pushed the jz/export-llm-docs branch 20 times, most recently from 89e52e9 to 0ffe50a Compare July 11, 2025 22:13

jackzhxng requested a review from larryliu0820 July 11, 2025 22:15

jackzhxng force-pushed the jz/export-llm-docs branch from 0ffe50a to dc267a8 Compare July 14, 2025 18:36

jackzhxng changed the base branch from main to export-llm-docs July 14, 2025 18:36

jackzhxng marked this pull request as ready for review July 14, 2025 18:36

jackzhxng requested review from Gasoonjia, JacobSzwejbka and mergennachin as code owners July 14, 2025 18:36

metascroy reviewed Jul 16, 2025

View reviewed changes

docs/source/llm/export-custom-llm.md Show resolved Hide resolved

metascroy reviewed Jul 16, 2025

View reviewed changes

docs/source/llm/export-custom-llm.md Show resolved Hide resolved

metascroy reviewed Jul 16, 2025

View reviewed changes

docs/source/llm/export-llm.md Outdated Show resolved Hide resolved

Update LLM documentation

1b2fd42

jackzhxng force-pushed the jz/export-llm-docs branch from 6fda1ef to 1b2fd42 Compare July 16, 2025 23:23

Scott comments

659db4f

mergennachin approved these changes Jul 17, 2025

View reviewed changes

Mergen comments

1a3e422

jackzhxng force-pushed the jz/export-llm-docs branch from 46cfe37 to 1a3e422 Compare July 19, 2025 00:21

shoumikhin reviewed Jul 19, 2025

View reviewed changes

docs/source/index.md Outdated Show resolved Hide resolved

digantdesai reviewed Jul 21, 2025

View reviewed changes

docs/source/llm/run-with-c-plus-plus.md Outdated Show resolved Hide resolved

jackzhxng and others added 6 commits July 21, 2025 15:55

Anthony comment

5fafb3a

Co-authored-by: Anthony Shoumikhin <[email protected]>

Update

09c24de

Add run on iOS page

1a7e797

Update run-on-ios.md

78b9f3f

Update index.md

ff92c9c

jackzhxng merged commit 01b7e7d into export-llm-docs Jul 23, 2025
93 of 95 checks passed

jackzhxng deleted the jz/export-llm-docs branch July 23, 2025 19:37

pytorchbot pushed a commit that referenced this pull request Jul 23, 2025

Export LLM documentation revamp (#12381)

5ac5c7d

(cherry picked from commit 01b7e7d)

pytorchbot mentioned this pull request Jul 23, 2025

[v0.7.0] Release Tracker #11075

Closed

jackzhxng mentioned this pull request Jul 24, 2025

Export LLM documentation revamp (#12381) #12800

Merged

jackzhxng added a commit that referenced this pull request Jul 29, 2025

Export LLM documentation revamp (#12381) (#12800)

08c7636

Merge #12381 from side branch into main. --------- Co-authored-by: Gasoonjia <[email protected]> Co-authored-by: Mergen Nachin <[email protected]>


		Create a file called export_nanogpt.py with the following contents:

		```python

Export LLM documentation revamp #12381

Export LLM documentation revamp #12381

Uh oh!

Conversation

jackzhxng commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

pytorch-bot bot commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12381

❌ 2 New Failures

Uh oh!

github-actions bot commented Jul 10, 2025

This PR needs a release notes: label

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shoumikhin commented Jul 16, 2025

Uh oh!

jackzhxng commented Jul 16, 2025

Uh oh!

mergennachin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shoumikhin commented Jul 18, 2025

Uh oh!

jackzhxng commented Jul 19, 2025

Uh oh!

Uh oh!

digantdesai Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

jackzhxng Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jackzhxng commented Jul 23, 2025

Uh oh!

pytorchbot commented Jul 23, 2025

Cherry picking #12381

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

jackzhxng commented Jul 10, 2025 •

edited

Loading

pytorch-bot bot commented Jul 10, 2025 •

edited

Loading

This PR needs a `release notes:` label