Update on "[ET-VK] Add kInt8x4 dtype and GPUMemoryLayouts for packed quantized tensors"

## Motivation

Lay the foundations for being able to execute statically quantized CNNs with ET-VK. Unlike with dynamic quantization, static quantization allows the output of quantized operators to stay in integer representation and be fed directly to the next quantized operator.

## Context

Typically, int8 quantized tensors can be represented by simply having the tensor use the int8 data type. While this is possible in ET-VK, in practice quantized operators expect int8 quantized tensors to be packed so that 16 8-bit values are packed into each `ivec4`, such that quantized int8 tensors will load/store with a granularity of 16 elements.

The reason for this is twofold:
* Support for shader int8 / storage buffer int8 extension is not guaranteed, meaning some devices do not allow using int8 types in shaders
* We have found that load/store from storage buffers/textures that use int8 data types sometimes results in worse memory load performance, due to vectorized load/store instructions not being used.

Therefore, in ET-VK we need a way to mark that a quantized tensor should

1. Use int32 as the underlying data type for the storage buffer/texture
2. Account for the block-packing that may be used

## Changes

First, introduce the `Int8x4` dtype that can be used for packed int8 tensors. This dtype is functionally the same as `Int`, but denotes that each int32 actually contains 4 packed 8-bit values.

Second, introduce new memory layouts: `kPackedInt8_4W4C` and `kPackedInt8_4H4W`. The former will be used for convolution, whil the latter will be used for matrix multiplication. See the inline comments for more details about these memory layouts.

Then, update `QuantizedConvolution.cpp` and `QuantizedLinear.cpp` to use the new data type and memory layouts for the packed int8 input tensor.

Differential Revision: [D82542336](https://our.internmc.facebook.com/intern/diff/D82542336/)

[ghstack-poisoned]

Author: ssjia

backends/cadence/aot/TARGETS

@@ -142,8 +142,8 @@ executorch_generated_lib(
     platforms = CXX,
     visibility = ["PUBLIC"],
     deps = [
-        "//executorch/backends/cadence/reference/kernels:cadence_kernels",
-        "//executorch/backends/cadence/reference/operators:cadence_cpu_ops",
+        "//executorch/backends/cadence/generic/kernels:cadence_kernels",
+        "//executorch/backends/cadence/generic/operators:cadence_generic_ops",
         "//executorch/kernels/portable:executorch_all_ops",
         "//executorch/kernels/portable:operators",
     ],

backends/cadence/aot/functions.yaml

@@ -20,17 +20,17 @@
 - op: _softmax.out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::_softmax_out
+      kernel_name: impl::G3::_softmax_out
 
 - op: add.out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::add_out
+      kernel_name: impl::G3::add_out
 
 - op: add.Scalar_out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::add_scalar_out
+      kernel_name: impl::G3::add_scalar_out
 
 - op: bmm.out
   kernels:
@@ -40,18 +40,18 @@
 - op: cat.out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::cat_out
+      kernel_name: impl::G3::cat_out
 
 - op: clamp.out
   cpp_no_default_args: ['min']
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::clamp_out
+      kernel_name: impl::G3::clamp_out
 
 - op: clamp.Tensor_out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::clamp_Tensor_out
+      kernel_name: impl::G3::clamp_Tensor_out
 
 - op: clone.out
   kernels:
@@ -61,12 +61,12 @@
 - op: div.out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::div_out
+      kernel_name: impl::G3::div_out
 
 - op: div.out_mode
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::div_out_mode
+      kernel_name: impl::G3::div_out_mode
 
 - op: embedding.out
   kernels:
@@ -81,41 +81,41 @@
 - op: lt.Scalar_out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::lt_Scalar_out
+      kernel_name: impl::G3::lt_Scalar_out
 
 - op: lt.Tensor_out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::lt_Tensor_out
+      kernel_name: impl::G3::lt_Tensor_out
 
 - op: mul.out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::mul_out
+      kernel_name: impl::G3::mul_out
 - op: mul.Scalar_out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::mul_scalar_out
+      kernel_name: impl::G3::mul_scalar_out
 
 - op: permute_copy.out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::permute_copy_out
+      kernel_name: impl::G3::permute_copy_out
 
 - op: rsqrt.out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::rsqrt_out
+      kernel_name: impl::G3::rsqrt_out
 
 - op: sigmoid.out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::sigmoid_out
+      kernel_name: impl::G3::sigmoid_out
 
 - op: slice_copy.Tensor_out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::slice_copy_Tensor_out
+      kernel_name: impl::G3::slice_copy_Tensor_out
 
 - op: split_with_sizes_copy.out
   kernels:
@@ -125,27 +125,27 @@
 - op: sqrt.out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::sqrt_out
+      kernel_name: impl::G3::sqrt_out
 
 - op: sub.out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::sub_out
+      kernel_name: impl::G3::sub_out
 
 - op: sub.Scalar_out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::sub_scalar_out
+      kernel_name: impl::G3::sub_scalar_out
 
 - op: tanh.out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::tanh_out
+      kernel_name: impl::G3::tanh_out
 
 - op: transpose_copy.int_out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::transpose_copy_int_out
+      kernel_name: impl::G3::transpose_copy_int_out
 
 - op: view_copy.out
   kernels:
@@ -155,37 +155,37 @@
 - op: where.self_out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::where_self_out
+      kernel_name: impl::G3::where_self_out
 
 - op: native_layer_norm.out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::native_layer_norm_out
+      kernel_name: impl::G3::native_layer_norm_out
 
 - op: mean.out
   kernels:
     - arg_meta: null
-      kernel_name:  cadence::impl::G3::mean_out
+      kernel_name:  impl::G3::mean_out
 
 - op: exp.out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::exp_out
-      
+      kernel_name: impl::G3::exp_out
+
 - op: hardtanh.out
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::hardtanh_out
+      kernel_name: impl::G3::hardtanh_out
 
 # custom ops
 - func: cadence::quantize_per_tensor.out(Tensor input, float scale, int zero_point, int quant_min, int quant_max, ScalarType dtype, *, Tensor(a!) out) -> Tensor(a!)
   variants: function
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::native::quantize_per_tensor_out
+      kernel_name: impl::G3::native::quantize_per_tensor_out
 
 - func: cadence::dequantize_per_tensor.out(Tensor input, float scale, int zero_point, int quant_min, int quant_max, ScalarType dtype, *, Tensor(a!) out) -> Tensor(a!)
   variants: function
   kernels:
     - arg_meta: null
-      kernel_name: cadence::impl::G3::native::dequantize_per_tensor_out
+      kernel_name: impl::G3::native::dequantize_per_tensor_out