Update base for Update on "[ET-VK] Use shared pointer for vTensorStorage"

## Changes

* In `vTensor`, store the `storage_` member using a shared pointer instead of a direct `vTensorStorage` object
* Remove the ability to construct a tensor view with a buffer offset

## Motivation

> * In `vTensor`, store the `storage_` member using a shared pointer instead of a direct `vTensorStorage` object

Previously, to support the ability to create tensor views I implemented copy constructors for `VulkanImage`, `VulkanBuffer`, and `vTensorStorage`. The idea was that the copied object instance would have the same handles as the original, but would not own the handle (and therefore not be responsible for destroying it).

However, the problem with this approach is that in order to construct pipeline barriers and  image layout transitions properly, we must know the details of how a resource was last accessed. Since tensor views make copies, accessing the original tensor will not update the access information of any copies it may have. Therefore, if the copy is then accessed, the last access information it stores would then be out of date.

I originally tried to solve this problem with crude assumptions in the `transition()` function, but unfortunately the solution was not robust enough. I discovered validation errors such as

```
[ RUN      ] VulkanComputeGraphOpsTest.test_transpose_with_mm
VUID-VkImageMemoryBarrier-oldLayout-01197(ERROR / SPEC): msgNum: 307231540 - Validation Error: [ VUID-VkImageMemoryBarrier-oldLayout-01197 ] Object 0: handle = 0x21eba07e160, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x35643f0000000111, type = VK_OBJECT_TYPE_IMAGE; | MessageID = 0x124ffb34 | vkCmdPipelineBarrier(): pImageMemoryBarriers[2].image (VkImage 0x35643f0000000111[]) cannot transition the layout of aspect=1, level=0, layer=0 from VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL when the previous known layout is VK_IMAGE_LAYOUT_GENERAL.
The Vulkan spec states: If srcQueueFamilyIndex and dstQueueFamilyIndex define a queue family ownership transfer or oldLayout and newLayout define an image layout transition, oldLayout must be VK_IMAGE_LAYOUT_UNDEFINED or the current layout of the image subresources affected by the barrier (https://vulkan.lunarg.com/doc/view/1.3.296.0/windows/1.3-extensions/vkspec.html#VUID-VkImageMemoryBarrier-oldLayout-01197)
    Objects: 2
        [0] 0x21eba07e160, type: 6, name: NULL
        [1] 0x35643f0000000111, type: 10, name: NULL
UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout(ERROR / SPEC): msgNum: 1303270965 - Validation Error: [ UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout ] Object 0: handle = 0x21eba07e160, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x35643f0000000111, type = VK_OBJECT_TYPE_IMAGE; | MessageID = 0x4dae5635 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[0] command buffer VkCommandBuffer 0x21eba07e160[] expects VkImage 0x35643f0000000111[] (subresource: aspectMask VK_IMAGE_ASPECT_COLOR_BIT array layer 0, mip level 0) to be in layout VK_IMAGE_LAYOUT_GENERAL--instead, current layout is VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL.
    Objects: 2
        [0] 0x21eba07e160, type: 6, name: NULL
        [1] 0x35643f0000000111, type: 10, name: NULL
VUID-VkImageMemoryBarrier-oldLayout-01197(ERROR / SPEC): msgNum: 307231540 - Validation Error: [ VUID-VkImageMemoryBarrier-oldLayout-01197 ] Object 0: handle = 0x21eba07e160, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x35643f0000000111, type = VK_OBJECT_TYPE_IMAGE; | MessageID = 0x124ffb34 | vkCmdPipelineBarrier(): pImageMemoryBarriers[2].image (VkImage 0x35643f0000000111[]) cannot transition the layout of aspect=1, level=0, layer=0 from VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL when the previous known layout is VK_IMAGE_LAYOUT_GENERAL.
The Vulkan spec states: If srcQueueFamilyIndex and dstQueueFamilyIndex define a queue family ownership transfer or oldLayout and newLayout define an image layout transition, oldLayout must be VK_IMAGE_LAYOUT_UNDEFINED or the current layout of the image subresources affected by the barrier (https://vulkan.lunarg.com/doc/view/1.3.296.0/windows/1.3-extensions/vkspec.html#VUID-VkImageMemoryBarrier-oldLayout-01197)
    Objects: 2
        [0] 0x21eba07e160, type: 6, name: NULL
        [1] 0x35643f0000000111, type: 10, name: NULL
UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout(ERROR / SPEC): msgNum: 1303270965 - Validation Error: [ UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout ] Object 0: handle = 0x21eba07e160, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x35643f0000000111, type = VK_OBJECT_TYPE_IMAGE; | MessageID = 0x4dae5635 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[0] command buffer VkCommandBuffer 0x21eba07e160[] expects VkImage 0x35643f0000000111[] (subresource: aspectMask VK_IMAGE_ASPECT_COLOR_BIT array layer 0, mip level 0) to be in layout VK_IMAGE_LAYOUT_GENERAL--instead, current layout is VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL.
    Objects: 2
        [0] 0x21eba07e160, type: 6, name: NULL
        [1] 0x35643f0000000111, type: 10, name: NULL
VUID-VkImageMemoryBarrier-oldLayout-01197(ERROR / SPEC): msgNum: 307231540 - Validation Error: [ VUID-VkImageMemoryBarrier-oldLayout-01197 ] Object 0: handle = 0x21eba07e160, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x35643f0000000111, type = VK_OBJECT_TYPE_IMAGE; | MessageID = 0x124ffb34 | vkCmdPipelineBarrier(): pImageMemoryBarriers[2].image (VkImage 0x35643f0000000111[]) cannot transition the layout of aspect=1, level=0, layer=0 from VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL when the previous known layout is VK_IMAGE_LAYOUT_GENERAL.
The Vulkan spec states: If srcQueueFamilyIndex and dstQueueFamilyIndex define a queue family ownership transfer or oldLayout and newLayout define an image layout transition, oldLayout must be VK_IMAGE_LAYOUT_UNDEFINED or the current layout of the image subresources affected by the barrier (https://vulkan.lunarg.com/doc/view/1.3.296.0/windows/1.3-extensions/vkspec.html#VUID-VkImageMemoryBarrier-oldLayout-01197)
    Objects: 2
        [0] 0x21eba07e160, type: 6, name: NULL
        [1] 0x35643f0000000111, type: 10, name: NULL
UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout(ERROR / SPEC): msgNum: 1303270965 - Validation Error: [ UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout ] Object 0: handle = 0x21eba07e160, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x35643f0000000111, type = VK_OBJECT_TYPE_IMAGE; | MessageID = 0x4dae5635 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[0] command buffer VkCommandBuffer 0x21eba07e160[] expects VkImage 0x35643f0000000111[] (subresource: aspectMask VK_IMAGE_ASPECT_COLOR_BIT array layer 0, mip level 0) to be in layout VK_IMAGE_LAYOUT_GENERAL--instead, current layout is VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL.
    Objects: 2
        [0] 0x21eba07e160, type: 6, name: NULL
        [1] 0x35643f0000000111, type: 10, name: NULL
VUID-VkImageMemoryBarrier-oldLayout-01197(ERROR / SPEC): msgNum: 307231540 - Validation Error: [ VUID-VkImageMemoryBarrier-oldLayout-01197 ] Object 0: handle = 0x21eba07e160, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x35643f0000000111, type = VK_OBJECT_TYPE_IMAGE; | MessageID = 0x124ffb34 | vkCmdPipelineBarrier(): pImageMemoryBarriers[2].image (VkImage 0x35643f0000000111[]) cannot transition the layout of aspect=1, level=0, layer=0 from VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL when the previous known layout is VK_IMAGE_LAYOUT_GENERAL.
The Vulkan spec states: If srcQueueFamilyIndex and dstQueueFamilyIndex define a queue family ownership transfer or oldLayout and newLayout define an image layout transition, oldLayout must be VK_IMAGE_LAYOUT_UNDEFINED or the current layout of the image subresources affected by the barrier (https://vulkan.lunarg.com/doc/view/1.3.296.0/windows/1.3-extensions/vkspec.html#VUID-VkImageMemoryBarrier-oldLayout-01197)
    Objects: 2
        [0] 0x21eba07e160, type: 6, name: NULL
        [1] 0x35643f0000000111, type: 10, name: NULL
UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout(ERROR / SPEC): msgNum: 1303270965 - Validation Error: [ UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout ] Object 0: handle = 0x21eba07e160, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x35643f0000000111, type = VK_OBJECT_TYPE_IMAGE; | MessageID = 0x4dae5635 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[0] command buffer VkCommandBuffer 0x21eba07e160[] expects VkImage 0x35643f0000000111[] (subresource: aspectMask VK_IMAGE_ASPECT_COLOR_BIT array layer 0, mip level 0) to be in layout VK_IMAGE_LAYOUT_GENERAL--instead, current layout is VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL.
    Objects: 2
        [0] 0x21eba07e160, type: 6, name: NULL
        [1] 0x35643f0000000111, type: 10, name: NULL
VUID-VkImageMemoryBarrier-oldLayout-01197(ERROR / SPEC): msgNum: 307231540 - Validation Error: [ VUID-VkImageMemoryBarrier-oldLayout-01197 ] Object 0: handle = 0x21eba07e160, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x35643f0000000111, type = VK_OBJECT_TYPE_IMAGE; | MessageID = 0x124ffb34 | vkCmdPipelineBarrier(): pImageMemoryBarriers[2].image (VkImage 0x35643f0000000111[]) cannot transition the layout of aspect=1, level=0, layer=0 from VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL when the previous known layout is VK_IMAGE_LAYOUT_GENERAL.
The Vulkan spec states: If srcQueueFamilyIndex and dstQueueFamilyIndex define a queue family ownership transfer or oldLayout and newLayout define an image layout transition, oldLayout must be VK_IMAGE_LAYOUT_UNDEFINED or the current layout of the image subresources affected by the barrier (https://vulkan.lunarg.com/doc/view/1.3.296.0/windows/1.3-extensions/vkspec.html#VUID-VkImageMemoryBarrier-oldLayout-01197)
    Objects: 2
        [0] 0x21eba07e160, type: 6, name: NULL
        [1] 0x35643f0000000111, type: 10, name: NULL
UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout(ERROR / SPEC): msgNum: 1303270965 - Validation Error: [ UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout ] Object 0: handle = 0x21eba07e160, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x35643f0000000111, type = VK_OBJECT_TYPE_IMAGE; | MessageID = 0x4dae5635 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[0] command buffer VkCommandBuffer 0x21eba07e160[] expects VkImage 0x35643f0000000111[] (subresource: aspectMask VK_IMAGE_ASPECT_COLOR_BIT array layer 0, mip level 0) to be in layout VK_IMAGE_LAYOUT_GENERAL--instead, current layout is VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL.
    Objects: 2
        [0] 0x21eba07e160, type: 6, name: NULL
        [1] 0x35643f0000000111, type: 10, name: NULL
[       OK ] VulkanComputeGraphOpsTest.test_transpose_with_mm (26 ms)
```

when using tensor views. The simplest solution is to use a shared pointer to store the tensor storage object, that way tensor views can point to the same underlying object instance, and last access metadata will be consistent among all tensor views.

> * Remove the ability to construct a tensor view with a buffer offset

Removed this because it turns out that buffer properties cannot have arbitrary offsets; there are restrictions for what offsets can be used. The validation errors say it all:

```
[ RUN      ] VulkanComputeGraphTest.test_simple_graph_with_view
VUID-VkWriteDescriptorSet-descriptorType-00328(ERROR / SPEC): msgNum: -368569266 - Validation Error: [ VUID-VkWriteDescriptorSet-descriptorType-00328 ] | MessageID = 0xea08144e | vkUpdateDescriptorSets(): pDescriptorWrites[1].pBufferInfo[0].offset (84) must be a multiple of device limit minStorageBufferOffsetAlignment 16 when descriptor type is VK_DESCRIPTOR_TYPE_STORAGE_BUFFER.
The Vulkan spec states: If descriptorType is VK_DESCRIPTOR_TYPE_STORAGE_BUFFER or VK_DESCRIPTOR_TYPE_STORAGE_BUFFER_DYNAMIC, the offset member of each element of pBufferInfo must be a multiple of VkPhysicalDeviceLimits::minStorageBufferOffsetAlignment (https://vulkan.lunarg.com/doc/view/1.3.296.0/windows/1.3-extensions/vkspec.html#VUID-VkWriteDescriptorSet-descriptorType-00328)
VUID-VkDescriptorBufferInfo-range-00342(ERROR / SPEC): msgNum: -371195848 - Validation Error: [ VUID-VkDescriptorBufferInfo-range-00342 ] Object 0: handle = 0xee647e0000000009, type = VK_OBJECT_TYPE_BUFFER; | MessageID = 0xe9e00038 | vkUpdateDescriptorSets(): pDescriptorWrites[1].pBufferInfo[0].range (224) is larger than buffer size (224) + offset (84).
The Vulkan spec states: If range is not equal to VK_WHOLE_SIZE, range must be less than or equal to the size of buffer minus offset (https://vulkan.lunarg.com/doc/view/1.3.296.0/windows/1.3-extensions/vkspec.html#VUID-VkDescriptorBufferInfo-range-00342)
    Objects: 1
        [0] 0xee647e0000000009, type: 9, name: NULL
VUID-VkBufferMemoryBarrier-size-01189(ERROR / SPEC): msgNum: -1238074894 - Validation Error: [ VUID-VkBufferMemoryBarrier-size-01189 ] Object 0: handle = 0x25d7d900870, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0xb63479f2 | vkCmdPipelineBarrier(): pBufferMemoryBarriers[0].size VkBuffer 0xee647e0000000009[] has offset 0x54 and size 0xe0 whose sum is greater than total size 0xe0.
The Vulkan spec states: If size is not equal to VK_WHOLE_SIZE, size must be less than or equal to than the size of buffer minus offset (https://vulkan.lunarg.com/doc/view/1.3.296.0/windows/1.3-extensions/vkspec.html#VUID-VkBufferMemoryBarrier-size-01189)
    Objects: 1
        [0] 0x25d7d900870, type: 6, name: NULL
[       OK ] VulkanComputeGraphTest.test_simple_graph_with_view (8 ms)
```

## Impact

* Heap allocation upon tensor construction
* Increased pointer chasing when accessing `vTensor` members

I will validate that the impact of this change does not regress load/inference latency.

Differential Revision: [D76047204](https://our.internmc.facebook.com/intern/diff/D76047204/)

[ghstack-poisoned]

Author: SS-JIA

.ci/scripts/build_llama_android.sh

@@ -42,6 +42,7 @@ build_llama_runner() {
     popd
     ANDROID_ABI=arm64-v8a
     cmake -DBUCK2="${BUCK2}" \
+    -DBUILD_TESTING=OFF \
     -DCMAKE_TOOLCHAIN_FILE="$ANDROID_NDK"/build/cmake/android.toolchain.cmake  \
     -DANDROID_ABI="${ANDROID_ABI}" \
     -DCMAKE_INSTALL_PREFIX=cmake-android-out \

.ci/scripts/test_llama.sh

@@ -169,6 +169,7 @@ cmake_build_llama_runner() {
     popd
     dir="examples/models/llama"
     retry cmake \
+        -DBUILD_TESTING=OFF \
         -DCMAKE_INSTALL_PREFIX=cmake-out \
         -DCMAKE_BUILD_TYPE="$CMAKE_BUILD_TYPE" \
         -Bcmake-out/${dir} \

.ci/scripts/test_llama_torchao_lowbit.sh

@@ -40,6 +40,7 @@ cmake --build cmake-out -j16 --target install --config Release
 
 # Install llama runner with torchao
 cmake -DPYTHON_EXECUTABLE=python \
+    -DBUILD_TESTING=OFF \
     -DCMAKE_BUILD_TYPE=Release \
     -DEXECUTORCH_BUILD_KERNELS_CUSTOM=ON \
     -DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \

.ci/scripts/test_llava.sh

@@ -64,9 +64,10 @@ cmake_install_executorch_libraries_for_android() {
 
 
 LLAVA_COMMON_CMAKE_ARGS="                        \
+        -DBUILD_TESTING=OFF                      \
         -DPYTHON_EXECUTABLE="$PYTHON_EXECUTABLE" \
         -DCMAKE_INSTALL_PREFIX=${BUILD_DIR}      \
-        -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}         \
+        -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}   \
         -DEXECUTORCH_BUILD_KERNELS_CUSTOM=ON     \
         -DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON  \
         -DEXECUTORCH_BUILD_XNNPACK=ON"

.github/workflows/trunk.yml

@@ -262,7 +262,7 @@ jobs:
         output=$(ls -la ${elf})
         arr=($output)
         size=${arr[4]}
-        threshold="102400" # 100KiB
+        threshold="103068" # ~100KiB
         echo "size: $size, threshold: $threshold"
         if [[ "$size" -le "$threshold" ]]; then
           echo "Success $size <= $threshold"

backends/arm/_passes/annotate_channels_last_dim_order_pass.py

@@ -35,7 +35,10 @@
 def _transpose_impl(*args, **kwargs):
     # Validate length of dim_order array
     dim = args[1]
-    assert len(dim) in (4, 5)
+    if len(dim) != 4 and len(dim) != 5:
+        raise ValueError(
+            f"Dim order length must be either 4 or 5, got {len(dim)}: {dim}"
+        )
     # Pass-through in edge-IR
     return args[0]
 

backends/arm/_passes/convert_split_to_slice.py

@@ -41,9 +41,14 @@ def call(self, graph_module: torch.fx.GraphModule):
             dim = split_node.args[2] if len(split_node.args) > 2 else 0
             dim = (dim + rank) % rank
 
-            assert (
-                sum(split_lengths) == shape[dim]
-            ), "Given split lengths don't sum up to the size of the dimension."
+            # Validate that split lengths cover the entire dimension
+            length_sum = sum(split_lengths)
+            dim_size = shape[dim]
+            if length_sum != dim_size:
+                raise ValueError(
+                    f"Split sizes {split_lengths} sum to {length_sum}, "
+                    f"but dimension {dim} has size {dim_size}"
+                )
 
             # Convert split argument 'split_lengths' to slice arguments start and end.
             starts = [0] * len(split_lengths)

backends/arm/_passes/fold_qdq_with_annotated_qparams_pass.py

@@ -120,7 +120,9 @@ def fold_and_annotate_arg(
         if input_qparams is not None:
             node.meta["input_qparams"][i] = input_qparams
             for n in nodes_to_remove:
-                assert n.target == dq_op
+                if n.target != dq_op:
+                    raise RuntimeError(f"Expected {dq_op} dq_op, got {n.target}")
+
                 n.replace_all_uses_with(n.args[0])  # type: ignore[arg-type]
                 graph_module.graph.erase_node(n)
 
@@ -136,14 +138,16 @@ def call(self, graph_module: GraphModule) -> PassResult:
                 continue
 
             # Make sure we haven't already set qparams meta information on the node
-            assert "input_qparams" not in n.meta, (
-                f'Unexpected key "input_qparams" found in meta for node {n}. '
-                "input_qparams should not have been set at this point"
-            )
-            assert "output_qparams" not in n.meta, (
-                f'Unexpected key "output_qparams" found in meta for node {n}. '
-                "output_qparams should not have been set at this point"
-            )
+            if "input_qparams" in n.meta:
+                raise RuntimeError(
+                    f'Unexpected key "input_qparams" found in meta for node {n}. '
+                    "input_qparams should not have been set at this point"
+                )
+            if "output_qparams" in n.meta:
+                raise RuntimeError(
+                    f'Unexpected key "output_qparams" found in meta for node {n}. '
+                    "output_qparams should not have been set at this point"
+                )
 
             # for the inputs and outputs search the graph for quantization info and
             # store the information in a dict with order of the _tensor_ inputs as key,

backends/arm/_passes/insert_table_ops.py

@@ -240,8 +240,17 @@ def call(self, graph_module: GraphModule) -> PassResult:
                     args=(node.args[0],),
                 )
                 output_node = table_node
-                assert len(input_qparams) == 1
-                assert len(output_qparams) == 1
+                # Expect exactly one quantization parameter for input and output
+                if len(input_qparams) != 1:
+                    raise ValueError(
+                        f"InsertTableOpsPass expected exactly one input quantization parameter, "
+                        f"got {len(input_qparams)} for node {node.name}"
+                    )
+                if len(output_qparams) != 1:
+                    raise ValueError(
+                        f"InsertTableOpsPass expected exactly one output quantization parameter, "
+                        f"got {len(output_qparams)} for node {node.name}"
+                    )
 
                 # Generate table buffer and how much to lshift the table output.
                 buffer, lshift = self.generate_table_values(

backends/arm/_passes/remove_clone_pass.py

@@ -17,5 +17,8 @@ def call_operator(self, op, args, kwargs, meta):
         if op != exir_ops.edge.aten.clone.default:
             return super().call_operator(op, args, kwargs, meta)
 
-        assert len(args) == 1
+        if len(args) != 1:
+            raise ValueError(
+                f"clone operator expects exactly one argument, got {len(args)}"
+            )
         return args[0]