Skip to content

Commit e4b080a

Browse files
committed
Update on "[ET-VK][ez] Enable dynamic shape support when using push constants"
## Changes * Call `encode_execute()` upon resize in `VulkanBackend.cpp` * Minor update to `DispatchNode` to store push constant data array as a persistent member of the class ## Motivation Passing in tensor metadata (i.e. sizes, strides) via push constants is typically more performant than passing them via a UBO (uniform buffer object). However, currently dynamic shapes do not work when push constants are used as I realized that the tensor metadata contained in the push constants do not get updated. It appears that that `vkCmdPushConstants` sets the push constants when encoding the command buffer, however the push constants will not be updated if the command buffer is submitted for execution multiple times. Therefore, to update push constant values **the command buffer needs to be re-encoded**. ## Performance Impact This may add a small performance overhead (i.e. re-encoding the command buffer) when executing models with dynamic shapes. Models that do not trigger tensor resizing will not be impacted. However, I measured the impact on a llama 3.2 1B model and the impact of re-encoding a command buffer appears to be negligible. In any case, re-encoding the command buffer is a "necessary evil" when working with dynamic shapes, otherwise the tensor metadata seen by shaders may never get updated. Furthermore, re-encoding the command buffer can allow an opportunity to adjust global work group sizing to match current tensor sizes, which may have a huge performance impact when maximum tensor sizes far exceeds what tensor sizes will realistically be during inference (one instance of this is for transformer models when the max sequence length is very long). Differential Revision: [D75686051](https://our.internmc.facebook.com/intern/diff/D75686051/) [ghstack-poisoned]
1 parent 85726c4 commit e4b080a

File tree

2 files changed

+1
-1
lines changed

2 files changed

+1
-1
lines changed

backends/vulkan/runtime/VulkanBackend.cpp

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -569,7 +569,6 @@ class VulkanBackend final : public ::executorch::runtime::BackendInterface {
569569

570570
if (should_propagate_resize) {
571571
compute_graph->propagate_resize();
572-
compute_graph->encode_execute();
573572
}
574573
compute_graph->execute();
575574

backends/vulkan/runtime/graph/ComputeGraph.cpp

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -696,6 +696,7 @@ void ComputeGraph::propagate_resize() {
696696
for (std::unique_ptr<ExecuteNode>& node : execute_nodes_) {
697697
node->trigger_resize(this);
698698
}
699+
encode_execute();
699700
}
700701

701702
} // namespace vkcompute

0 commit comments

Comments
 (0)