GH-47572: [C++][Parquet] Uniform unpack interface (#47573)

### Rationale for this change
Refactor the unpack function code generator and dispatcher to accommodate more use cases:
- ~`uint16_t`~ and `uint64_t` new sizes
- ~A *dispatch once* function returning a function pointer to the correct bit width~

*Update 2025-10-13*: the dispatch once and `uint16_t` implementation were removed as they turned out to be slower.

### What changes are included in this PR?
The diff are hard to look at but the important files to look at are:
- The two python files for code generation accommodate new sizes (~with the exception of `uint16` on Avx512 for which the algorithms assumptions break~);
- The two code generators have a uniform structure for the "batch unpackers" they generate: each one of them is a specialization of a struct template `unpack29_32` > `Unpacker<uint32_t, 29>::unpack`
  - Using specialization instead of hard coded number in function names makes it possible to use them in more generic code
  - Wrapping the functions in a struct makes it possible to carry information along the function (such as the number of values that said function unpacks) and to leave the door open for partial specialization for future improvements.
- The public functions in `bpacking_internal.h` are also template specialization `unpack32` -> `unpack<uint32_t>`
- The large `switch` statements and for loops used to dispatch the bit width to its appropriate implementation are now all generic with a `constexpr` generated jump table. The logic is in `bpacking_dispatch_internal.h`

From a performance perspective:
- there is no improvement to the individual "batch unpacker" generated
  - The SIMD code is actually doing worst that scalar on SSE4.2 OR `uint16_t`
- there are new sizes that can bring improvements
  - `unpack<uint64_t>` has an SIMD implementation that should benefit `DeltaBitPackDecoder`
  -  ~Novel `unpack<uint16_t>` should benefit the level decoders~
  - ~The dispatch once mechanism should benefit all repeated calls to unpack functions (still need mixing with dynamic dispatch, but see `get_unpack_fn` for the building block).~
  
*Update 2025-10-13*:
- Added an unpack implementation for `uint8_t` and `uint16_t` that call the `uint32_t` version on a local buffer combined with a `static_cast` loop (what was done on the call site before).
- The performances should remain on par with previous implementations. The PR as it is now mainly changes the interface of unpack functions for future iterations and cleaner use.

### Are these changes tested?
Very much.

### Are there any user-facing changes?

* GitHub Issue: #47572

Lead-authored-by: AntoinePrv <[email protected]>
Co-authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>

Author: AntoinePrv

.gitattributes

@@ -1,4 +1,4 @@
-cpp/src/arrow/util/bpacking_*_generated.h linguist-generated=true
+cpp/src/arrow/util/bpacking_*_generated_internal.h linguist-generated=true
 cpp/src/parquet/chunker_*_generated.h linguist-generated=true
 cpp/src/generated/*.cpp linguist-generated=true
 cpp/src/generated/*.h linguist-generated=true

cpp/apidoc/Doxyfile

@@ -1095,6 +1095,7 @@ EXCLUDE_PATTERNS       = *-test.cc \
                          *test* \
                          *_generated.h \
                          *-benchmark.cc \
+                         *_codegen.py \
                          *internal*
 
 # The EXCLUDE_SYMBOLS tag can be used to specify one or more symbol names

cpp/src/arrow/CMakeLists.txt

@@ -490,6 +490,8 @@ set(ARROW_UTIL_SRCS
     util/bitmap_builders.cc
     util/bitmap_ops.cc
     util/bpacking.cc
+    util/bpacking_scalar.cc
+    util/bpacking_simd_default.cc
     util/byte_size.cc
     util/byte_stream_split_internal.cc
     util/cancel.cc
@@ -533,11 +535,8 @@ set(ARROW_UTIL_SRCS
 
 append_runtime_avx2_src(ARROW_UTIL_SRCS util/byte_stream_split_internal_avx2.cc)
 
-append_runtime_avx2_src(ARROW_UTIL_SRCS util/bpacking_avx2.cc)
-append_runtime_avx512_src(ARROW_UTIL_SRCS util/bpacking_avx512.cc)
-if(ARROW_HAVE_NEON)
-  list(APPEND ARROW_UTIL_SRCS util/bpacking_neon.cc)
-endif()
+append_runtime_avx2_src(ARROW_UTIL_SRCS util/bpacking_simd_avx2.cc)
+append_runtime_avx512_src(ARROW_UTIL_SRCS util/bpacking_simd_avx512.cc)
 
 if(ARROW_WITH_BROTLI)
   list(APPEND ARROW_UTIL_SRCS util/compression_brotli.cc)

cpp/src/arrow/util/bit_stream_utils_internal.h

@@ -296,6 +296,18 @@ inline bool BitReader::GetValue(int num_bits, T* v) {
   return GetBatch(num_bits, v, 1) == 1;
 }
 
+namespace internal_bit_reader {
+template <typename T>
+struct unpack_detect {
+  using type = std::make_unsigned_t<T>;
+};
+
+template <>
+struct unpack_detect<bool> {
+  using type = bool;
+};
+}  // namespace internal_bit_reader
+
 template <typename T>
 inline int BitReader::GetBatch(int num_bits, T* v, int batch_size) {
   ARROW_DCHECK(buffer_ != NULL);
@@ -323,47 +335,12 @@ inline int BitReader::GetBatch(int num_bits, T* v, int batch_size) {
     }
   }
 
-  if (sizeof(T) == 4) {
-    int num_unpacked =
-        internal::unpack32(buffer + byte_offset, reinterpret_cast<uint32_t*>(v + i),
-                           batch_size - i, num_bits);
-    i += num_unpacked;
-    byte_offset += num_unpacked * num_bits / 8;
-  } else if (sizeof(T) == 8 && num_bits > 32) {
-    // Use unpack64 only if num_bits is larger than 32
-    // TODO (ARROW-13677): improve the performance of internal::unpack64
-    // and remove the restriction of num_bits
-    int num_unpacked =
-        internal::unpack64(buffer + byte_offset, reinterpret_cast<uint64_t*>(v + i),
-                           batch_size - i, num_bits);
-    i += num_unpacked;
-    byte_offset += num_unpacked * num_bits / 8;
-  } else {
-    // TODO: revisit this limit if necessary
-    ARROW_DCHECK_LE(num_bits, 32);
-    const int buffer_size = 1024;
-    uint32_t unpack_buffer[buffer_size];
-    while (i < batch_size) {
-      int unpack_size = std::min(buffer_size, batch_size - i);
-      int num_unpacked =
-          internal::unpack32(buffer + byte_offset, unpack_buffer, unpack_size, num_bits);
-      if (num_unpacked == 0) {
-        break;
-      }
-      for (int k = 0; k < num_unpacked; ++k) {
-#ifdef _MSC_VER
-#  pragma warning(push)
-#  pragma warning(disable : 4800)
-#endif
-        v[i + k] = static_cast<T>(unpack_buffer[k]);
-#ifdef _MSC_VER
-#  pragma warning(pop)
-#endif
-      }
-      i += num_unpacked;
-      byte_offset += num_unpacked * num_bits / 8;
-    }
-  }
+  using unpack_t = typename internal_bit_reader::unpack_detect<T>::type;
+
+  int num_unpacked = ::arrow::internal::unpack(
+      buffer + byte_offset, reinterpret_cast<unpack_t*>(v + i), batch_size - i, num_bits);
+  i += num_unpacked;
+  byte_offset += num_unpacked * num_bits / 8;
 
   buffered_values =
       detail::ReadLittleEndianWord(buffer + byte_offset, max_bytes - byte_offset);