Serialize NamedDataStoreOutput into PTD. #9758

pytorchbot · 2025-03-29T18:33:54Z

This PR was created by the merge bot to help merge the original PR into the main branch.
ghstack PR number: #9125 by @lucylq
^ Please use this as the source of truth for the PR details, comments, and reviews
ghstack PR base: https://github.com/pytorch/executorch/tree/gh/lucylq/59/base
ghstack PR head: https://github.com/pytorch/executorch/tree/gh/lucylq/59/head
Merge bot PR base: https://github.com/pytorch/executorch/tree/main
Merge bot PR head: https://github.com/pytorch/executorch/tree/gh/lucylq/59/orig
@diff-train-skip-merge

Pull Request resolved: #9125 Update PTD serialization to account for blobs from the NamedDataStoreOutput. Something we can do in the future is to consolidate tensors (that go through the emitter) and blobs (that come from the NamedDataStore). ghstack-source-id: 274509070 @exported-using-ghexport Differential Revision: [D70939807](https://our.internmc.facebook.com/intern/diff/D70939807/)

pytorch-bot · 2025-03-29T18:33:58Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9758

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 72 New Failures

As of commit e26fe91 with merge base 976fe48 ():

NEW FAILURES - The following jobs have failed:

Check Labels / Check labels (gh)
RuntimeError: Error checking labels: PR does not have required labels
Lint / lintrunner / linux-job (gh)
RuntimeError: Command docker exec -t 9d9af47b8ad56b06cf64e1039bbc3824e340906467364a6836295200a1690fe8 /exec failed with exit code 1
pull / android / build-llm-demo / linux-job (gh)
RuntimeError: Command docker exec -t 40fc7bc73b4dc0519aa9b9adccf223b292a1d4fd461a9b7a56f0720ec77ebd70 /exec failed with exit code 1
pull / test-binary-size-linux / linux-job (gh)
RuntimeError: Command docker exec -t 953c0ed823f9eb167ccafe26cc6080c347ca62e99854818c7450266435da6cb8 /exec failed with exit code 1
pull / test-binary-size-linux-gcc / linux-job (gh)
RuntimeError: Command docker exec -t e2e9d50e7f87a0363d6bdca82849803afc9b7f091629740f62dd2cbef32775a8 /exec failed with exit code 1
pull / test-custom-ops-linux / linux-job (gh)
RuntimeError: Command docker exec -t 607d587d2dd8ac89116ec95e3e482bc2398aaf7df5e8a38eae1318f4fe980510 /exec failed with exit code 1
pull / test-eval_llama-mmlu-linux / linux-job (gh)
RuntimeError: Command docker exec -t c62f58afb6e894361bac1240a3838144127c41232128598bbf2d7d5b2f8171c1 /exec failed with exit code 1
pull / test-eval_llama-wikitext-linux / linux-job (gh)
RuntimeError: Command docker exec -t b3a0fe7fbfd897bec98eb6dc7814283e0fc2c419334a68ebaaf1ceb90afe62dc /exec failed with exit code 1
pull / test-llama_runner_eager-linux / linux-job (gh)
RuntimeError: Command docker exec -t 23df9bc4b4f3b608e8ade7ea020d066a0552371ffec26871dc0bad9f276de991 /exec failed with exit code 1
pull / test-llama-runner-linux (bf16, custom, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
RuntimeError: Command docker exec -t 62ff46aa5508c6cca3b0b3c19036b9dbe73682ffd989384bb536514a61e4f04a /exec failed with exit code 1
pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
RuntimeError: Command docker exec -t 2fea5bd7f319e9a0cc6ea4868a67214fb516f58988d19a900e594ef29380d5c6 /exec failed with exit code 1
pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.arm64.2xlarge, executorch-ubuntu-22.04-gc... / linux-job (gh)
RuntimeError: Command docker exec -t d691f57061407fdcec37844d27d52b74ef327463d039b538d2e27693d6b8b677 /exec failed with exit code 1
pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04... / linux-job (gh)
RuntimeError: Command docker exec -t 4fd7beab5563d3b462a2a079cd580570b4c2d98f333e4d8563e5c244c6efd45d /exec failed with exit code 1
pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu... / linux-job (gh)
RuntimeError: Command docker exec -t f95594660e8816796c385ba3c4da5f3522f3baa43f0af100f13995964e9f00ff /exec failed with exit code 1
pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
RuntimeError: Command docker exec -t 6d3f7073112825137c6e59c44a1b2f865db59519f182f00ecf6a2002d004ed69 /exec failed with exit code 1
pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu-22.04-... / linux-job (gh)
RuntimeError: Command docker exec -t 38e4c5c2c42eed8d03383bdc7d2bc0c342b6c2480fde6539819e89affe42aa93 /exec failed with exit code 1
pull / test-llama-runner-linux-android / linux-job (gh)
RuntimeError: Command docker exec -t f05366d99b3ddc820376e6dddc33de0f65d0bc53a93bb5f5be8af4e5aa35120a /exec failed with exit code 1
pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
RuntimeError: Command docker exec -t fb3498eda30ee275da98d47399396d6cc567c122662e65bf622151d4ba4699aa /exec failed with exit code 1
pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
RuntimeError: Command docker exec -t 9af83cdebcabe5e12081bc965a555998ccdc03ead78e37f60cf21860c241bb46 /exec failed with exit code 1
pull / test-llava-runner-linux / linux-job (gh)
RuntimeError: Command docker exec -t 7f977352280bf519ffb5e0fe527a4162b251a902c6202094537a7361a60bbcbb /exec failed with exit code 1
pull / test-models-linux (add_mul, portable, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t cf31b1518e1f7ca926024d80ba9bd5e9cac4ea6ef78648a4436a007fb09add49 /exec failed with exit code 1
pull / test-models-linux (add_mul, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 9d8f385e33d6cb71c9489168d50a393eb36e6173d46b8014e9a4ef2330158e55 /exec failed with exit code 1
pull / test-models-linux (add, portable, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 9a2c434ba2dee0bb3f697d3faecc542483a0bfb76e8d28b6d393f0df528a3789 /exec failed with exit code 1
pull / test-models-linux (add, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 80326d9e2c0c76eccfec94782e2a2c2015fc662527a0d8720608b8b2ce2b3beb /exec failed with exit code 1
pull / test-models-linux (emformer_join, portable, linux.4xlarge.memory) / linux-job (gh)
RuntimeError: Command docker exec -t 0e2d35e51a3bbe68d2e8e53a575d01f2f9ac1eb6a6e86d0563cef184962a3348 /exec failed with exit code 1
pull / test-models-linux (emformer_join, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
RuntimeError: Command docker exec -t dfd5030a7f0d05727e5e740bd6e1955fd0ab9f4c1b1d093a970c9de0a759f2ff /exec failed with exit code 1
pull / test-models-linux (emformer_transcribe, portable, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 255f52cf32d74a2d2aebb5bff46ba7064ab295691008674c60c7c7e5d9663307 /exec failed with exit code 1
pull / test-models-linux (emformer_transcribe, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 675d97b388e97e8a26c92a8e913ce545c4322216a631fb5533ea5ffdb5825e3a /exec failed with exit code 1
pull / test-models-linux (ic3, portable, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 20c031b57d551d8404b0c72b375a30dc9433fb8aa56d909332b6e1f06b5068fb /exec failed with exit code 1
pull / test-models-linux (ic3, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 1f364541a32815413bdc85ad0587ca9ee171a14f71e9cea970baee421cd69f4c /exec failed with exit code 1
pull / test-models-linux (ic4, portable, linux.4xlarge.memory) / linux-job (gh)
RuntimeError: Command docker exec -t 70ef6ebcb9d4337b6a639e3a53d26f4b8af656ede6d7a371af398c82958c22f0 /exec failed with exit code 1
pull / test-models-linux (ic4, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
RuntimeError: Command docker exec -t 25b382188956e054f6a0a2a6cb8ecf4609fa55f24d5872e85bc32fbfd1a627f3 /exec failed with exit code 1
pull / test-models-linux (linear, portable, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t a8ec780cca8437cfe8633a6b34e19b42754b3c2850bce5f2aa49823941369fc8 /exec failed with exit code 1
pull / test-models-linux (linear, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 8ddb21bdf2de77f0843508ca6a1144a451edc9af13777d1f58e689d150ed6aca /exec failed with exit code 1
pull / test-models-linux (llama3_2_vision_encoder, portable, linux.4xlarge.memory) / linux-job (gh)
RuntimeError: Command docker exec -t 266b245f219e3ac481df0036db17aa6a1f09957629be92e5d78b58db559362e0 /exec failed with exit code 1
pull / test-models-linux (mobilebert, portable, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 59592413dee4d83cb0c3bacebfe6cf09444d161f1b258ed6a3fcf3df08c6ffdf /exec failed with exit code 1
pull / test-models-linux (mobilebert, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 2f074cb84e529b7dbf5ed4649a4ef121982dfd9e3c2152572ea6aa8c2c4660b7 /exec failed with exit code 1
pull / test-models-linux (mv2, portable, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 700d03bf318bf2b99895f9f571a198a78c0e07412337b0dd6caca8549cad49e2 /exec failed with exit code 1
pull / test-models-linux (mv2, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t f796fb99748f5029304238a1108c552e111633ca63af20aca1333fa2f36eb3ea /exec failed with exit code 1
pull / test-models-linux (phi_4_mini, portable, linux.4xlarge.memory) / linux-job (gh)
RuntimeError: Command docker exec -t 0d9857f7be10ad3ae7b92e2ef4df55deae1cac70beeb8a6baf641f0d7fea4fbc /exec failed with exit code 1
pull / test-models-linux (resnet18, portable, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t e453c4858ae263fca9d67e93d57a9893e3178b5c7254bbd83c1c8aff10333f61 /exec failed with exit code 1
pull / test-models-linux (resnet18, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 3c714dd70a1692cf28951b877066dcaeb6171aeeae4fafc38a4a055ff478b0a3 /exec failed with exit code 1
pull / test-models-linux (resnet50, portable, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 730350146f4e0e543da386fb1e828815ffc3a2e9275acbafb1f64aefb09336ef /exec failed with exit code 1
pull / test-models-linux (resnet50, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t c36d7c2048af75a187ddfb81d14c8ec6bd0270b1939b822f80e8360a03aa1392 /exec failed with exit code 1
pull / test-models-linux (w2l, portable, linux.4xlarge.memory) / linux-job (gh)
RuntimeError: Command docker exec -t dbcd42c7a313716c562b6bbbbd22f16e712e7f1a6faf1fdc934d0f50e4610e5c /exec failed with exit code 1
pull / test-models-linux-basic (mv3, portable, buck2, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
RuntimeError: Command docker exec -t 18281b5112fb9ad010005008cc1247a63a12759d7cd23330647a63cf9beb8132 /exec failed with exit code 1
pull / test-models-linux-basic (mv3, portable, cmake, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
RuntimeError: Command docker exec -t 2423f655dcd55ebeed2bd9456d30f27b2404457054499b5a405c69136831e2bd /exec failed with exit code 1
pull / test-models-linux-basic (mv3, portable, cmake, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11... / linux-job (gh)
RuntimeError: Command docker exec -t 7006c5ff7755ac517ae35c8745a449d9ad6defb1534c526e6bccb379d320e5cd /exec failed with exit code 1
pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t 3d1cd8595a1e662ce923892c59416f4670691b3cb3774a21dde807b10a74571c /exec failed with exit code 1
pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t d48979a9a17b77db66dd253980ab5ef07ca3110c62596907be5f080e52bda25a /exec failed with exit code 1
pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
RuntimeError: Command docker exec -t f36e6ca94f8290509d9f318e9d61efb9897b45911b1d40b372852a4e926f8f14 /exec failed with exit code 1
pull / test-models-linux-basic (vit, portable, buck2, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
RuntimeError: Command docker exec -t e3bd6c02041a654834c6529c9af047b1a9df4d9413df5a0569cb4c227c4b213a /exec failed with exit code 1
pull / test-models-linux-basic (vit, portable, cmake, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
RuntimeError: Command docker exec -t 181d75ef1000cae2d8a3a111b35da1566fb7e24a591b1540e88a7e6fb7978fc7 /exec failed with exit code 1
pull / test-models-linux-basic (vit, portable, cmake, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11... / linux-job (gh)
RuntimeError: Command docker exec -t 5b3d780305ddf88fedbb4a9e3f1ccf60455ef898aebf77b7f0b171e7b15384f4 /exec failed with exit code 1
pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t 97a24f74a8149ee03791f662d9963746e9b61b95e82bdf44f1d029105801648c /exec failed with exit code 1
pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t 8f1bcf817fbdc3d7d6fe9f1ce4a5cf666a9d92748aeefb372957d19d0e3078d9 /exec failed with exit code 1
pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
RuntimeError: Command docker exec -t 2cdfa3697b80a1bcbdb3eca0a7469b773e9708e125abfd60c56102d21fd5b693 /exec failed with exit code 1
pull / test-moshi-linux / linux-job (gh)
RuntimeError: Command docker exec -t a3c66cf62a3cc725f81fda68fdf7497f721753163c35fdf45548749f6a71c51e /exec failed with exit code 1
pull / test-openvino-linux / linux-job (gh)
RuntimeError: Command docker exec -t 26ab60e831b1f9c51936a67621dcbad1d99dcdb8508bf8b5953f8b89d6e5a527 /exec failed with exit code 127
pull / test-phi-3-mini-runner-linux / linux-job (gh)
RuntimeError: Command docker exec -t b5884512ef08020fcb3d16431623cdea75ed3b8290c701d26eabb2cc3e1ddb72 /exec failed with exit code 1
pull / test-pybind-build-linux / linux-job (gh)
RuntimeError: Command docker exec -t 718ebe2bc049170b979d738091f986d114b1271256e153f56685ef8c77832342 /exec failed with exit code 1
pull / test-quantized-aot-lib-linux / linux-job (gh)
RuntimeError: Command docker exec -t cfad8b9505c9808df54673b089c899b1deee3da12717cc1e6418b2e95eb493f9 /exec failed with exit code 1
pull / test-selective-build-linux / linux-job (gh)
RuntimeError: Command docker exec -t 4463bddd026857f97684128c4b8f26fdeaca430306d5c6c7a980f0f15e3de258 /exec failed with exit code 1
pull / test-setup-linux-gcc / linux-job (gh)
RuntimeError: Command docker exec -t 1a5ec58ecfdc1a09c37b5a68e56cc344334ecb8a4f33a3699dda8451faa25cc0 /exec failed with exit code 1
pull / test-static-llama-qnn-linux / linux-job (gh)
RuntimeError: Command docker exec -t 5ce1d8138a352ab3c640cbea8f1c4d1f9dc5d089e22e4f361a58e395b1d9ae0a /exec failed with exit code 1
pull / unittest / linux / linux-job (gh)
RuntimeError: Command docker exec -t 6435b31a6d673fd1871b6b18b49b4aa6ce796a86460a5c2f6c8670decea0dfb8 /exec failed with exit code 1
pull / unittest / macos / macos-job (gh)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
pull / unittest-arm / linux-job (gh)
RuntimeError: Command docker exec -t a0687fbf5b973237792a542f1b64127e507d3432ffd5cd32f45f09b4dcb9febd /exec failed with exit code 1
pull / unittest-buck / linux / linux-job (gh)
RuntimeError: Command docker exec -t bea4648463b4ccf587622a4e5247be75360caffeff9c2751bc1b2ec384bc3a1e /exec failed with exit code 1
pull / unittest-buck / macos / macos-job (gh)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
pull / unittest-editable / linux / linux-job (gh)
RuntimeError: Command docker exec -t c58fdc5e576ec0b239be41e011946cfdba0b13fde595580eeed5b7078f58fa69 /exec failed with exit code 1
pull / unittest-editable / macos / macos-job (gh)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2025-03-29T18:35:06Z

This PR needs a `release notes:` label

If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Update PTD serialization to account for blobs from the NamedDataStoreOutput. Something we can do in the future is to consolidate tensors (that go through the emitter) and blobs (that come from the NamedDataStore). Differential Revision: [D70939807](https://our.internmc.facebook.com/intern/diff/D70939807/)

pytorchbot requested review from JacobSzwejbka, larryliu0820, lucylq and tarun292 as code owners March 29, 2025 18:33

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Mar 29, 2025

kirklandsign approved these changes Mar 31, 2025

View reviewed changes

kirklandsign merged commit cc231cf into main Mar 31, 2025
10 of 83 checks passed

kirklandsign deleted the gh/lucylq/59/orig branch March 31, 2025 16:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Serialize NamedDataStoreOutput into PTD. #9758

Serialize NamedDataStoreOutput into PTD. #9758

Uh oh!

pytorchbot commented Mar 29, 2025

Uh oh!

pytorch-bot bot commented Mar 29, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Mar 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Serialize NamedDataStoreOutput into PTD. #9758

Serialize NamedDataStoreOutput into PTD. #9758

Uh oh!

Conversation

pytorchbot commented Mar 29, 2025

Uh oh!

pytorch-bot bot commented Mar 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9758

❌ 72 New Failures

Uh oh!

github-actions bot commented Mar 29, 2025

This PR needs a release notes: label

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pytorch-bot bot commented Mar 29, 2025 •

edited

Loading

This PR needs a `release notes:` label