Merge remote-tracking branch 'origin/main' into feat/flight-sql-client

Author: HackPoint

.github/pull_request_template.md

@@ -1,61 +1,20 @@
-
-<!--
 Thanks for opening a pull request!
-If this is your first pull request you can find detailed information on how 
-to contribute here:
-  * [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
-  * [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
-
-
-If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
-
-Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
-
-Then could you also rename the pull request title in the following format?
-
-    GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
 
-or
+If this is your first pull request you can find detailed information on how to contribute here:
 
-    MINOR: [${COMPONENT}] ${SUMMARY}
+  * [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
+  * [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
 
--->
+Please remove this line and the above text before creating your pull request.
 
 ### Rationale for this change
 
-<!--
- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
- Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
--->
-
 ### What changes are included in this PR?
 
-<!--
-There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
--->
-
 ### Are these changes tested?
 
-<!--
-We typically require tests for all PRs in order to:
-1. Prevent the code from being accidentally broken by subsequent changes
-2. Serve as another way to document the expected behavior of the code
-
-If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
--->
-
 ### Are there any user-facing changes?
 
-<!--
-If there are user-facing changes then we may require documentation to be updated before approving the PR.
--->
-
-<!--
-If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking.
--->
-<!-- **This PR includes breaking changes to public APIs.** -->
+**This PR includes breaking changes to public APIs.** (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)
 
-<!--
-Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious.
--->
-<!-- **This PR contains a "Critical Fix".** -->
+**This PR contains a "Critical Fix".** (If the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld), please provide explanation. If not, you can remove this.)

.github/workflows/r.yml

@@ -361,6 +361,7 @@ jobs:
           working-directory: 'r'
           extra-packages: |
             any::rcmdcheck
+            any::cyclocomp
       # TODO(ARROW-17149): figure out why the GCS tests are hanging on Windows
       # - name: Install Google Cloud Storage Testbench
       #   shell: bash
@@ -405,9 +406,7 @@ jobs:
           )
           # we use pak for package installation since it is faster, safer and more convenient
           pak::local_install()
-          # Pin the lintr version to avoid breaking changes with newer 3.2.0 version
-          # See: https://github.com/apache/arrow/pull/45524
-          pak::pak("[email protected]")
+          pak::pak("lintr")
           lintr::expect_lint_free()
       - name: Dump install logs
         shell: cmd

LICENSE.txt

@@ -2257,5 +2257,36 @@ SOFTWARE.
 java/vector/src/main/java/org/apache/arrow/vector/util/IntObjectHashMap.java
 java/vector/src/main/java/org/apache/arrow/vector/util/IntObjectMap.java
 
-These file are derived from code from Netty, which is made available under the
+These files are derived from code from Netty, which is made available under the
 Apache License 2.0.
+
+--------------------------------------------------------------------------------
+cpp/src/arrow/util/math_internal.cc (some portions)
+
+Some portions of this file are derived from
+
+https://github.com/ankane/dist-rust/
+
+which is made available under the MIT license
+
+The MIT License (MIT)
+
+Copyright (c) 2021-2023 Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

ci/conda_env_cpp.txt

@@ -37,6 +37,7 @@ libprotobuf
 libutf8proc
 lz4-c
 make
+meson
 ninja
 nodejs
 orc

ci/scripts/PKGBUILD

@@ -28,7 +28,6 @@ depends=("${MINGW_PACKAGE_PREFIX}-bzip2"
          "${MINGW_PACKAGE_PREFIX}-curl" # for google-cloud-cpp bundled build
          "${MINGW_PACKAGE_PREFIX}-libutf8proc"
          "${MINGW_PACKAGE_PREFIX}-re2"
-         "${MINGW_PACKAGE_PREFIX}-thrift"
          "${MINGW_PACKAGE_PREFIX}-snappy"
          "${MINGW_PACKAGE_PREFIX}-zlib"
          "${MINGW_PACKAGE_PREFIX}-lz4"
@@ -85,6 +84,19 @@ build() {
   # and does not work with newer versions of CMake. See comments:
   # https://github.com/apache/arrow/pull/44989/files#r1901428998
 
+  # We use the bundled Apache Thrift instead of the MINGW one because
+  # the upstream one on rtools packages is unmaintained. Apache Thrift
+  # still have the following problem:
+  #
+  #   https://github.com/apache/thrift/pull/2725
+  #
+  # The original MSYS2 package has another fix:
+  #
+  #   https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-thrift/002-fix-pkgconfig-paths.patch
+  #
+  # But one on rtools packages doesn't have the fix. So we can't use
+  # the MINGW one.
+
   # MSYS2_ARG_CONV_EXCL is needed to prevent autoconverting CMAKE_INSTALL_PREFIX
   # to Windows paths. See https://www.msys2.org/docs/filesystem-paths/#process-arguments
 
@@ -128,7 +140,8 @@ build() {
     -DCMAKE_BUILD_TYPE="release" \
     -DCMAKE_INSTALL_PREFIX=${MINGW_PREFIX} \
     -DCMAKE_UNITY_BUILD=OFF \
-    -DCMAKE_VERBOSE_MAKEFILE=ON
+    -DCMAKE_VERBOSE_MAKEFILE=ON \
+    -DThrift_SOURCE=BUNDLED
 
   make -j3
   popd

ci/scripts/cpp_build.sh

@@ -109,7 +109,13 @@ if [ "${ARROW_OFFLINE}" = "ON" ]; then
   echo > /etc/resolv.conf
 fi
 
-if [ "${ARROW_EMSCRIPTEN:-OFF}" = "ON" ]; then
+if [ "${ARROW_USE_MESON:-OFF}" = "ON" ]; then
+  meson setup \
+    --prefix=${MESON_PREFIX:-${ARROW_HOME}} \
+    --buildtype=${ARROW_BUILD_TYPE:-debug} \
+    . \
+    ${source_dir}
+elif [ "${ARROW_EMSCRIPTEN:-OFF}" = "ON" ]; then
   if [ "${UBUNTU}" = "20.04" ]; then
     echo "arrow emscripten build is not supported on Ubuntu 20.04, run with UBUNTU=22.04"
     exit -1
@@ -141,7 +147,6 @@ else
     -DARROW_BUILD_BENCHMARKS=${ARROW_BUILD_BENCHMARKS:-OFF} \
     -DARROW_BUILD_EXAMPLES=${ARROW_BUILD_EXAMPLES:-OFF} \
     -DARROW_BUILD_INTEGRATION=${ARROW_BUILD_INTEGRATION:-OFF} \
-    -DARROW_BUILD_OPENMP_BENCHMARKS=${ARROW_BUILD_OPENMP_BENCHMARKS:-OFF} \
     -DARROW_BUILD_SHARED=${ARROW_BUILD_SHARED:-ON} \
     -DARROW_BUILD_STATIC=${ARROW_BUILD_STATIC:-ON} \
     -DARROW_BUILD_TESTS=${ARROW_BUILD_TESTS:-OFF} \
@@ -244,8 +249,12 @@ else
     ${source_dir}
 fi
 
-export CMAKE_BUILD_PARALLEL_LEVEL=${CMAKE_BUILD_PARALLEL_LEVEL:-$[${n_jobs} + 1]}
-time cmake --build . --target install
+if [ "${ARROW_USE_MESON:-OFF}" = "ON" ]; then
+  time meson install
+else
+  export CMAKE_BUILD_PARALLEL_LEVEL=${CMAKE_BUILD_PARALLEL_LEVEL:-$[${n_jobs} + 1]}
+  time cmake --build . --target install
+fi
 
 # Save disk space by removing large temporary build products
 find . -name "*.o" -delete

ci/scripts/cpp_test.sh

@@ -89,14 +89,21 @@ pushd ${build_dir}
 if [ -z "${PYTHON}" ] && ! which python > /dev/null 2>&1; then
   export PYTHON="${PYTHON:-python3}"
 fi
-ctest \
+if [ "${ARROW_USE_MESON:-OFF}" = "ON" ]; then
+  ARROW_BUILD_EXAMPLES=OFF # TODO: Remove this
+  meson test \
+    --print-errorlogs \
+    "$@"
+else
+  ctest \
     --label-regex unittest \
     --output-on-failure \
     --parallel ${n_jobs} \
     --repeat until-pass:3 \
     --timeout ${ARROW_CTEST_TIMEOUT:-300} \
     "${ctest_options[@]}" \
     "$@"
+fi
 
 if [ "${ARROW_BUILD_EXAMPLES}" == "ON" ]; then
     examples=$(find ${binary_output_dir} -executable -name "*example")

ci/scripts/install_python.sh

@@ -28,9 +28,9 @@ declare -A versions
 versions=([3.9]=3.9.13
           [3.10]=3.10.11
           [3.11]=3.11.9
-          [3.12]=3.12.5
-          [3.13]=3.13.0
-          [3.13t]=3.13.0)
+          [3.12]=3.12.9
+          [3.13]=3.13.2
+          [3.13t]=3.13.2)
 
 if [ "$#" -ne 2 ]; then
   echo "Usage: $0 <platform> <version>"
@@ -47,17 +47,11 @@ full_version=${versions[$2]}
 if [ $platform = "macOS" ]; then
     echo "Downloading Python installer..."
 
-    if [ "$version" = "3.13" ] || [ "$version" = "3.13t" ];
+    if [ "$(uname -m)" = "x86_64" ] && [ "$version" = "3.9" ];
     then
-        fname="python-${full_version}rc2-macos11.pkg"
-    elif [ "$(uname -m)" = "arm64" ] || \
-         [ "$version" = "3.10" ] || \
-         [ "$version" = "3.11" ] || \
-         [ "$version" = "3.12" ];
-    then
-        fname="python-${full_version}-macos11.pkg"
-    else
         fname="python-${full_version}-macosx10.9.pkg"
+    else
+        fname="python-${full_version}-macos11.pkg"
     fi
     wget "https://www.python.org/ftp/python/${full_version}/${fname}"
 

ci/scripts/r_windows_build.sh

@@ -66,23 +66,23 @@ if [ -d mingw64/lib/ ]; then
   # Move the 64-bit versions of libarrow into the expected location
   mv mingw64/lib/*.a $DST_DIR/lib/x64
   # These are from https://dl.bintray.com/rtools/mingw{32,64}/
-  cp $MSYS_LIB_DIR/mingw64/lib/lib{thrift,snappy,zstd,lz4,brotli*,bz2,crypto,curl,ss*,utf8proc,re2,nghttp2}.a $DST_DIR/lib/x64
+  cp $MSYS_LIB_DIR/mingw64/lib/lib{snappy,zstd,lz4,brotli*,bz2,crypto,curl,ss*,utf8proc,re2,nghttp2}.a $DST_DIR/lib/x64
 fi
 
 # Same for the 32-bit versions
 if [ -d mingw32/lib/ ]; then
   ls $MSYS_LIB_DIR/mingw32/lib/
   mkdir -p $DST_DIR/lib/i386
   mv mingw32/lib/*.a $DST_DIR/lib/i386
-  cp $MSYS_LIB_DIR/mingw32/lib/lib{thrift,snappy,zstd,lz4,brotli*,bz2,crypto,curl,ss*,utf8proc,re2,nghttp2}.a $DST_DIR/lib/i386
+  cp $MSYS_LIB_DIR/mingw32/lib/lib{snappy,zstd,lz4,brotli*,bz2,crypto,curl,ss*,utf8proc,re2,nghttp2}.a $DST_DIR/lib/i386
 fi
 
 # Do the same also for ucrt64
 if [ -d ucrt64/lib/ ]; then
   ls $MSYS_LIB_DIR/ucrt64/lib/
   mkdir -p $DST_DIR/lib/x64-ucrt
   mv ucrt64/lib/*.a $DST_DIR/lib/x64-ucrt
-  cp $MSYS_LIB_DIR/ucrt64/lib/lib{thrift,snappy,zstd,lz4,brotli*,bz2,crypto,curl,ss*,utf8proc,re2,nghttp2}.a $DST_DIR/lib/x64-ucrt
+  cp $MSYS_LIB_DIR/ucrt64/lib/lib{snappy,zstd,lz4,brotli*,bz2,crypto,curl,ss*,utf8proc,re2,nghttp2}.a $DST_DIR/lib/x64-ucrt
 fi
 
 # Create build artifact

cpp/CMakePresets.json

@@ -41,7 +41,6 @@
       "cacheVariables": {
         "ARROW_BUILD_BENCHMARKS": "ON",
         "ARROW_BUILD_BENCHMARKS_REFERENCE": "ON",
-        "ARROW_BUILD_OPENMP_BENCHMARKS": "ON",
         "ARROW_BUILD_DETAILED_BENCHMARKS": "OFF",
         "CMAKE_BUILD_TYPE": "RelWithDebInfo"
       }