Skip to content

Add flat tensor to cmake #8332

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
lucylq wants to merge 41 commits into from
Closed

Add flat tensor to cmake #8332

lucylq wants to merge 41 commits into from

Conversation

@lucylq
Copy link
Contributor

@lucylq lucylq commented Feb 10, 2025

Summary

Add flat tensor to cmake

Test plan

PYTHON_EXECUTABLE=python EXECUTORCH_BUILD_PYBIND=ON CMAKE_ARGS="-DEXECUTORCH_BUILD_XNNPACK=ON -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON" .ci/scripts/setup-linux.sh cmake

sh test/run_oss_cpp_tests.sh extension/flat_tensor/test

lucylq added 30 commits December 4, 2024 22:18
@lucylq
[executorch][schema] Add 'EXTERNAL' to DataLocation in schema
3753000 
To indicate if a tensor is external to the PTE file or not.

Currently, we can also use the existence of 'fqn' to determine if a tensor is external or not.
I think it's better to have a specific location field as fqn may be required for cases besides external tensor storage.

Differential Revision: [D66523171](https://our.internmc.facebook.com/intern/diff/D66523171/)

[ghstack-poisoned]
@lucylq
[executorch][emitter] Emit FQNs
4ad792e 
Emit FQNs for external tensors. In the emitter, store external tensors as:
```
// list of unique tensors
external_constants_buffer: List[bytes]

// map of {constant_tag: {fqn: index into external_constant_buffer}}
// constant_tag: may want to save multiple external constant files; group them together via the tag.
// {fqn: index}; there may be multiple fqns pointing to the same data buffer. This is for deduplication.
external_constants_map: [Dict[str, Dict[str, int]]
```

Differential Revision: [D66523226](https://our.internmc.facebook.com/intern/diff/D66523226/)

[ghstack-poisoned]
@lucylq
[executorch][passes] Add config and pass to tag constants for externa…
2ae5e12 
…l file

- Add config 'external_constants' to ExecutorchBackendConfig.
- When set to True, run the 'external_constants_pass'
- This tags all constants as external, and moves them into a separate buffer to be serialized outside of the PTE file.

Note: users can write their own passes to tag weights to specific files / multiple files.

TODO: write example pass and test for the case where we have two constant files.

Differential Revision: [D66560903](https://our.internmc.facebook.com/intern/diff/D66560903/)

[ghstack-poisoned]
@lucylq
[executorch][serialization] Data serialization interface
320fc1c 
Introduce data serialization interface.

Differential Revision: [D65947145](https://our.internmc.facebook.com/intern/diff/D65947145/)

[ghstack-poisoned]
@lucylq
Update base for Update on "[executorch][serialization] Data serializa…
fcf2aa8 
…tion interface"

Introduce data serialization interface.

Differential Revision: [D65947145](https://our.internmc.facebook.com/intern/diff/D65947145/)

[ghstack-poisoned]
@lucylq
Update on "[executorch][serialization] Data serialization interface"
c764b19 
Introduce data serialization interface.

Differential Revision: [D65947145](https://our.internmc.facebook.com/intern/diff/D65947145/)

[ghstack-poisoned]
@lucylq
[executorch][serialization] Refactor flatbuffer utils into separate file
e0e2c33 
For usage in extension/flat_tensor/serialize.

Differential Revision: [D66854756](https://our.internmc.facebook.com/intern/diff/D66854756/)

[ghstack-poisoned]
@lucylq
[executorch][flat_tensor] Generate flatc files
c0982a3 
1. Move schema files under flat_tensor/serialize
2. Add targets to generate schema files

Differential Revision: [D66903492](https://our.internmc.facebook.com/intern/diff/D66903492/)

[ghstack-poisoned]
@lucylq
[executorch][flat_tensor] Serialize flat tensor
e4b3ca1 
Serialize a flat tensor file. The resulting file looks like:

Header with
- flatbuffer offset and size
- segment data offset and size

Flatbuffer
Tensor data (in segment)

Differential Revision: [D66374253](https://our.internmc.facebook.com/intern/diff/D66374253/)

[ghstack-poisoned]
@lucylq
[executorch][flat_tensor] Serialize flat tensor tests
4ec0731 
More comprehensive testing for flat tensor serialization.

Differential Revision: [D67007821](https://our.internmc.facebook.com/intern/diff/D67007821/)

[ghstack-poisoned]
@lucylq
[executorch][serialization] Serialize PTD files.
aa13c87 
Introduce top-level serialization file that calls:
- serialize_pte_binary for PTE file
- FlatTensor.serialize_tensors for PTD files.

Differential Revision: [D66523267](https://our.internmc.facebook.com/intern/diff/D66523267/)

[ghstack-poisoned]
@lucylq
Update base for Update on "[executorch][serialization] Serialize PTD …
062d47b 
…files."

Introduce top-level serialization file that calls:
- serialize_pte_binary for PTE file
- FlatTensor.serialize_tensors for PTD files.

Differential Revision: [D66523267](https://our.internmc.facebook.com/intern/diff/D66523267/)

[ghstack-poisoned]
@lucylq
Update on "[executorch][serialization] Serialize PTD files."
103ac70 
Introduce top-level serialization file that calls:
- serialize_pte_binary for PTE file
- FlatTensor.serialize_tensors for PTD files.

Differential Revision: [D66523267](https://our.internmc.facebook.com/intern/diff/D66523267/)

[ghstack-poisoned]
@lucylq
Update base for Update on "[executorch][serialization] Serialize PTD …
34c4841 
…files."

Introduce top-level serialization file that calls:
- serialize_pte_binary for PTE file
- FlatTensor.serialize_tensors for PTD files.

Differential Revision: [D66523267](https://our.internmc.facebook.com/intern/diff/D66523267/)

[ghstack-poisoned]
@lucylq
Update on "[executorch][serialization] Serialize PTD files."
eae2b11 
Introduce top-level serialization file that calls:
- serialize_pte_binary for PTE file
- FlatTensor.serialize_tensors for PTD files.

Differential Revision: [D66523267](https://our.internmc.facebook.com/intern/diff/D66523267/)

[ghstack-poisoned]
@lucylq
Update base for Update on "[executorch][serialization] Serialize PTD …
6cc5085 
…files."

Introduce top-level serialization file that calls:
- serialize_pte_binary for PTE file
- FlatTensor.serialize_tensors for PTD files.

Differential Revision: [D66523267](https://our.internmc.facebook.com/intern/diff/D66523267/)

[ghstack-poisoned]
@lucylq
Update on "[executorch][serialization] Serialize PTD files."
d25ae36 
Introduce top-level serialization file that calls:
- serialize_pte_binary for PTE file
- FlatTensor.serialize_tensors for PTD files.

Differential Revision: [D66523267](https://our.internmc.facebook.com/intern/diff/D66523267/)

[ghstack-poisoned]
@lucylq
Update base for Update on "[executorch][serialization] Serialize PTD …
2992d47 
…files."

Introduce top-level serialization file that calls:
- serialize_pte_binary for PTE file
- FlatTensor.serialize_tensors for PTD files.

Differential Revision: [D66523267](https://our.internmc.facebook.com/intern/diff/D66523267/)

[ghstack-poisoned]
@lucylq
Update on "[executorch][serialization] Serialize PTD files."
3bedb8d 
Introduce top-level serialization file that calls:
- serialize_pte_binary for PTE file
- FlatTensor.serialize_tensors for PTD files.

Differential Revision: [D66523267](https://our.internmc.facebook.com/intern/diff/D66523267/)

[ghstack-poisoned]
@lucylq
[executorch][core] Add TensorLayout to core
46ee760 
Introduce TensorLayout class, used to describe external tensors. Currently contains:
- scalar_type
- sizes
- dim_order

Differential Revision: [D67048723](https://our.internmc.facebook.com/intern/diff/D67048723/)

[ghstack-poisoned]
@lucylq
[executorch][core] NamedDataMap interface
c81d8cf 
Add NamedDataMap interface to runtime.

Differential Revision: [D66834552](https://our.internmc.facebook.com/intern/diff/D66834552/)

[ghstack-poisoned]
@lucylq
Update base for Update on "[executorch][core] NamedDataMap interface"
20f5706 
Add NamedDataMap interface to runtime.

Differential Revision: [D66834552](https://our.internmc.facebook.com/intern/diff/D66834552/)

[ghstack-poisoned]
@lucylq
Update on "[executorch][core] NamedDataMap interface"
907d9f2 
Add NamedDataMap interface to runtime.

Differential Revision: [D66834552](https://our.internmc.facebook.com/intern/diff/D66834552/)

[ghstack-poisoned]
@lucylq
[executorch][flat_tensor] DataMap implementation
87eded3 
DataMap implementation that
* Loads a flat_tensor file
* Populates a map with {fqn: tensor} and {fqn: TensorLayout}.
* Makes tensor information available via the named_data_map.h interface.

For now, DataMap doesn't store the DataLoader.
- If/when tensors are in their own segments, DataMap should also store a DataLoader.

Differential Revision: [D67064580](https://our.internmc.facebook.com/intern/diff/D67064580/)

[ghstack-poisoned]
@lucylq
Update on "[executorch][flat_tensor] DataMap implementation"
b8f4a78 
DataMap implementation that
* Loads a flat_tensor file
* Populates a map with {fqn: tensor} and {fqn: TensorLayout}.
* Makes tensor information available via the named_data_map.h interface.

For now, DataMap doesn't store the DataLoader.
- If/when tensors are in their own segments, DataMap should also store a DataLoader.

Differential Revision: [D67064580](https://our.internmc.facebook.com/intern/diff/D67064580/)

[ghstack-poisoned]
@lucylq
Update base for Update on "[executorch][flat_tensor] DataMap implemen…
968e982 
…tation"

DataMap implementation that
* Loads a flat_tensor file
* Populates a map with {fqn: tensor} and {fqn: TensorLayout}.
* Makes tensor information available via the named_data_map.h interface.

For now, DataMap doesn't store the DataLoader.
- If/when tensors are in their own segments, DataMap should also store a DataLoader.

Differential Revision: [D67064580](https://our.internmc.facebook.com/intern/diff/D67064580/)

[ghstack-poisoned]
@lucylq
Update on "[executorch][flat_tensor] DataMap implementation"
eb49548 
DataMap implementation that
* Loads a flat_tensor file
* Populates a map with {fqn: tensor} and {fqn: TensorLayout}.
* Makes tensor information available via the named_data_map.h interface.

For now, DataMap doesn't store the DataLoader.
- If/when tensors are in their own segments, DataMap should also store a DataLoader.

Differential Revision: [D67064580](https://our.internmc.facebook.com/intern/diff/D67064580/)

[ghstack-poisoned]
@lucylq
Update base for Update on "[executorch][flat_tensor] DataMap implemen…
c74a135 
…tation"

DataMap implementation that
* Loads a flat_tensor file
* Populates a map with {fqn: tensor} and {fqn: TensorLayout}.
* Makes tensor information available via the named_data_map.h interface.

For now, DataMap doesn't store the DataLoader.
- If/when tensors are in their own segments, DataMap should also store a DataLoader.

Differential Revision: [D67064580](https://our.internmc.facebook.com/intern/diff/D67064580/)

[ghstack-poisoned]
@lucylq
Update on "[executorch][flat_tensor] DataMap implementation"
54c2137 
DataMap implementation that
* Loads a flat_tensor file
* Populates a map with {fqn: tensor} and {fqn: TensorLayout}.
* Makes tensor information available via the named_data_map.h interface.

For now, DataMap doesn't store the DataLoader.
- If/when tensors are in their own segments, DataMap should also store a DataLoader.

Differential Revision: [D67064580](https://our.internmc.facebook.com/intern/diff/D67064580/)

[ghstack-poisoned]
@lucylq
Update base for Update on "[executorch][flat_tensor] DataMap implemen…
965a986 
…tation"

DataMap implementation that
* Loads a flat_tensor file
* Populates a map with {fqn: tensor} and {fqn: TensorLayout}.
* Makes tensor information available via the named_data_map.h interface.

For now, DataMap doesn't store the DataLoader.
- If/when tensors are in their own segments, DataMap should also store a DataLoader.

Differential Revision: [D67064580](https://our.internmc.facebook.com/intern/diff/D67064580/)

[ghstack-poisoned]
lucylq added 10 commits January 24, 2025 11:26
@lucylq
Update on "[executorch][flat_tensor] DataMap implementation"
f24fcae 
DataMap implementation that
* Loads a flat_tensor file
* Populates a map with {fqn: tensor} and {fqn: TensorLayout}.
* Makes tensor information available via the named_data_map.h interface.

For now, DataMap doesn't store the DataLoader.
- If/when tensors are in their own segments, DataMap should also store a DataLoader.

Differential Revision: [D67064580](https://our.internmc.facebook.com/intern/diff/D67064580/)

[ghstack-poisoned]
@lucylq
Update base for Update on "[executorch][flat_tensor] DataMap implemen…
fa29404 
…tation"

DataMap implementation that
* Loads a flat_tensor file
* Populates a map with {fqn: tensor} and {fqn: TensorLayout}.
* Makes tensor information available via the named_data_map.h interface.

For now, DataMap doesn't store the DataLoader.
- If/when tensors are in their own segments, DataMap should also store a DataLoader.

Differential Revision: [D67064580](https://our.internmc.facebook.com/intern/diff/D67064580/)

[ghstack-poisoned]
@lucylq
Update on "[executorch][flat_tensor] DataMap implementation"
b2a592f 
DataMap implementation that
* Loads a flat_tensor file
* Populates a map with {fqn: tensor} and {fqn: TensorLayout}.
* Makes tensor information available via the named_data_map.h interface.

For now, DataMap doesn't store the DataLoader.
- If/when tensors are in their own segments, DataMap should also store a DataLoader.

Differential Revision: [D67064580](https://our.internmc.facebook.com/intern/diff/D67064580/)

[ghstack-poisoned]
@lucylq
Update base for Update on "[executorch][flat_tensor] DataMap implemen…
85564f5 
…tation"

DataMap implementation that
* Loads a flat_tensor file
* Populates a map with {fqn: tensor} and {fqn: TensorLayout}.
* Makes tensor information available via the named_data_map.h interface.

For now, DataMap doesn't store the DataLoader.
- If/when tensors are in their own segments, DataMap should also store a DataLoader.

Differential Revision: [D67064580](https://our.internmc.facebook.com/intern/diff/D67064580/)

[ghstack-poisoned]
@lucylq
Update on "[executorch][flat_tensor] DataMap implementation"
6783e4e 
DataMap implementation that
* Loads a flat_tensor file
* Populates a map with {fqn: tensor} and {fqn: TensorLayout}.
* Makes tensor information available via the named_data_map.h interface.

For now, DataMap doesn't store the DataLoader.
- If/when tensors are in their own segments, DataMap should also store a DataLoader.

Differential Revision: [D67064580](https://our.internmc.facebook.com/intern/diff/D67064580/)

[ghstack-poisoned]
@lucylq
[executorch][runtime] Add NamedDataMap to method load
ed41d21 
Add NamedDataMap as an arg to:
- Method
- load_method
- parseTensor

Use NamedDataMap to resolve external tensors in parseTensor.

Test that the PTE + PTD file run well inside method_test.

Differential Revision: [D67127327](https://our.internmc.facebook.com/intern/diff/D67127327/)

[ghstack-poisoned]
@lucylq
Update base for Update on "[executorch][runtime] Add NamedDataMap to …
fda77c5 
…method load"

Add NamedDataMap as an arg to:
- Method
- load_method
- parseTensor

Use NamedDataMap to resolve external tensors in parseTensor.

Test that the PTE + PTD file run well inside method_test.

Differential Revision: [D67127327](https://our.internmc.facebook.com/intern/diff/D67127327/)

[ghstack-poisoned]
@lucylq
Update on "[executorch][runtime] Add NamedDataMap to method load"
4674508 
Add NamedDataMap as an arg to:
- Method
- load_method
- parseTensor

Use NamedDataMap to resolve external tensors in parseTensor.

Test that the PTE + PTD file run well inside method_test.

Differential Revision: [D67127327](https://our.internmc.facebook.com/intern/diff/D67127327/)

[ghstack-poisoned]
@lucylq
Update base for Update on "[executorch][runtime] Add NamedDataMap to …
fa0faee 
…method load"

Add NamedDataMap as an arg to:
- Method
- load_method
- parseTensor

Use NamedDataMap to resolve external tensors in parseTensor.

Test that the PTE + PTD file run well inside method_test.

Differential Revision: [D67127327](https://our.internmc.facebook.com/intern/diff/D67127327/)

[ghstack-poisoned]
@lucylq
Update on "[executorch][runtime] Add NamedDataMap to method load"
974373e 
Add NamedDataMap as an arg to:
- Method
- load_method
- parseTensor

Use NamedDataMap to resolve external tensors in parseTensor.

Test that the PTE + PTD file run well inside method_test.

Differential Revision: [D67127327](https://our.internmc.facebook.com/intern/diff/D67127327/)

[ghstack-poisoned]
@pytorch-bot
Copy link

pytorch-bot bot commented Feb 10, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/8332

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 2 Cancelled Jobs

As of commit 03569f7 with merge base 8ee637e (image):

NEW FAILURE - The following job has failed:

CANCELLED JOBS - The following jobs were cancelled. Please retry:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 10, 2025
@github-actions
Copy link

github-actions bot commented Feb 10, 2025

This PR needs a release notes: label

If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@lucylq lucylq force-pushed the lfq.flat-tensor branch 4 times, most recently from d62d1b4 to 17fe113 Compare February 10, 2025 21:28
@lucylq lucylq closed this Mar 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@larryliu0820 larryliu0820 Awaiting requested review from larryliu0820 larryliu0820 will be requested when the pull request is marked ready for review larryliu0820 is a code owner

@kirklandsign kirklandsign Awaiting requested review from kirklandsign kirklandsign will be requested when the pull request is marked ready for review kirklandsign is a code owner

@JacobSzwejbka JacobSzwejbka Awaiting requested review from JacobSzwejbka JacobSzwejbka will be requested when the pull request is marked ready for review JacobSzwejbka is a code owner

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@lucylq @facebook-github-bot