apacheGH-49422: [CI][Integration][Ruby] Add the Ruby implementation (apache#49423)

### Rationale for this change

There are some missing features in the Ruby implmentaion for now but we can pass them by skipping some tests in our integration tests.

### What changes are included in this PR?

Archery:

* Add `--with-ruby` to `archery integration`
* Add `archery.integration.tester_ruby.RubyTester`
* Add `no_map_field_names_validate` quirk for apacheGH-49415
* Show environment variables too on external command failure because Ruby tester uses environment variables not command line arguments to pass information to integration tester
* Use `ARCHERY_INTEGRATION_WITH_CPP=1` instead of `ARROW_INTEGRATION_CPP=ON` like other implementations such as `ARCHERY_INTEGRATION_WITH_GO`

Ruby:

* Add `red-arrow-format-integration-test` as the test driver
  * This is not included in `.gem` because this is only for development
* Add `ruby/red-arrow-format/lib/arrow-format/integration/` as helpers of the test driver
  * This is not included in `.gem` because this is only for development
* Add `ArrowFormat::Array#empty?`
* Add `ArrowFormat::RecordBatch#empty?`
* Add `ArrowFormat::NullArray#n_nulls`
* `ArrowFormat::*Array#to_a`: Add support for empty case
* Fix Apache Arrow decimal <-> `BigDecimal` conversion
* `ArrowFormat::Bitmap#each`: Fix a bug that one bit is ignored
* Move dictionary ID to `ArrowFormat::DictionaryType` from `ArrowFormat::Field`
* Add support for V4 union that has validity bitmap
* Add support for no continuation token message for backward compatibility
* `ArrowFormat::StreamingReader`: Add support for reading schema without calling `#each`
* `ArrowFormat::MapType`: Add support for keys sorted
* `ArrowFormat::MapType`: Always use "key"/"value"/"entries" for field names

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* GitHub Issue: apache#49422

Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>

Author: kou

.github/workflows/integration.yml

@@ -33,6 +33,7 @@ on:
       - 'integration/**'
       - 'cpp/**'
       - 'format/**'
+      - 'ruby/red-arrow-format/**'
   pull_request:
     paths:
       - '.dockerignore'
@@ -43,6 +44,7 @@ on:
       - 'integration/**'
       - 'cpp/**'
       - 'format/**'
+      - 'ruby/red-arrow-format/**'
 
 concurrency:
   group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}

ci/docker/conda-integration.dockerfile

@@ -42,6 +42,7 @@ RUN mamba install -q -y \
         nodejs=${node} \
         yarn=${yarn} \
         openjdk=${jdk} \
+        ruby \
         zstd && \
     mamba clean --yes --all --force-pkgs-dirs
 

ci/scripts/integration_arrow.sh

@@ -24,9 +24,14 @@ build_dir=${2}
 
 gold_dir=$arrow_dir/testing/data/arrow-ipc-stream/integration
 
+# For backward compatibility.
 : "${ARROW_INTEGRATION_CPP:=ON}"
+: "${ARCHERY_INTEGRATION_WITH_CPP:=$([ "${ARROW_INTEGRATION_CPP}" = "ON" ] && echo "1" || echo "0")}"
+export ARCHERY_INTEGRATION_WITH_CPP
+: "${ARCHERY_INTEGRATION_WITH_RUBY:=1}"
+export ARCHERY_INTEGRATION_WITH_RUBY
 
-: "${ARCHERY_INTEGRATION_TARGET_IMPLEMENTATIONS:=cpp}"
+: "${ARCHERY_INTEGRATION_TARGET_IMPLEMENTATIONS:=cpp,ruby}"
 export ARCHERY_INTEGRATION_TARGET_IMPLEMENTATIONS
 
 . "${arrow_dir}/ci/scripts/util_log.sh"
@@ -57,14 +62,11 @@ export PYTHONFAULTHANDLER=1
 export GOMEMLIMIT=200MiB
 export GODEBUG=gctrace=1,clobberfree=1
 
-ARCHERY_WITH_CPP=$([ "$ARROW_INTEGRATION_CPP" == "ON" ] && echo "1" || echo "0")
-
 # Rust can be enabled by exporting ARCHERY_INTEGRATION_WITH_RUST=1
 time archery integration \
     --run-c-data \
     --run-ipc \
     --run-flight \
-    --with-cpp="${ARCHERY_WITH_CPP}" \
     --gold-dirs="$gold_dir/0.14.1" \
     --gold-dirs="$gold_dir/0.17.1" \
     --gold-dirs="$gold_dir/1.0.0-bigendian" \

ci/scripts/integration_arrow_build.sh

@@ -22,7 +22,10 @@ set -e
 arrow_dir=${1}
 build_dir=${2}
 
+# For backward compatibility.
 : "${ARROW_INTEGRATION_CPP:=ON}"
+: "${ARCHERY_INTEGRATION_WITH_CPP:=$([ "${ARROW_INTEGRATION_CPP}" = "ON" ] && echo "1" || echo "0")}"
+: "${ARCHERY_INTEGRATION_WITH_RUBY:=1}"
 
 . "${arrow_dir}/ci/scripts/util_log.sh"
 
@@ -41,7 +44,7 @@ fi
 github_actions_group_end
 
 github_actions_group_begin "Integration: Build: C++"
-if [ "${ARROW_INTEGRATION_CPP}" == "ON" ]; then
+if [ "${ARCHERY_INTEGRATION_WITH_CPP}" -gt "0" ]; then
     "${arrow_dir}/ci/scripts/cpp_build.sh" "${arrow_dir}" "${build_dir}"
 fi
 github_actions_group_end
@@ -69,3 +72,9 @@ if [ "${ARCHERY_INTEGRATION_WITH_JS}" -gt "0" ]; then
     cp -a "${arrow_dir}/js" "${build_dir}/js"
 fi
 github_actions_group_end
+
+github_actions_group_begin "Integration: Build: Ruby"
+if [ "${ARCHERY_INTEGRATION_WITH_RUBY}" -gt "0" ]; then
+    rake -C "${arrow_dir}/ruby/red-arrow-format" install
+fi
+github_actions_group_end

dev/archery/archery/cli.py

@@ -667,7 +667,8 @@ def _set_default(opt, default):
 @click.option('--random-seed', type=int, default=12345,
               help="Seed for PRNG when generating test data")
 @click.option('--with-cpp', type=bool, default=False,
-              help='Include C++ in integration tests')
+              help='Include C++ in integration tests',
+              envvar="ARCHERY_INTEGRATION_WITH_CPP")
 @click.option('--with-dotnet', type=bool, default=False,
               help='Include .NET in integration tests',
               envvar="ARCHERY_INTEGRATION_WITH_DOTNET")
@@ -683,6 +684,9 @@ def _set_default(opt, default):
 @click.option('--with-nanoarrow', type=bool, default=False,
               help='Include nanoarrow in integration tests',
               envvar="ARCHERY_INTEGRATION_WITH_NANOARROW")
+@click.option('--with-ruby', type=bool, default=False,
+              help='Include Ruby in integration tests',
+              envvar="ARCHERY_INTEGRATION_WITH_RUBY")
 @click.option('--with-rust', type=bool, default=False,
               help='Include Rust in integration tests',
               envvar="ARCHERY_INTEGRATION_WITH_RUST")

dev/archery/archery/integration/datagen.py

@@ -1937,13 +1937,15 @@ def get_generated_json_files(tempdir=None):
         .skip_tester('Java')
         .skip_tester('JS')
         .skip_tester('nanoarrow')
+        .skip_tester('Ruby')
         .skip_tester('Rust')
         .skip_tester('Go'),
 
         generate_decimal64_case()
         .skip_tester('Java')
         .skip_tester('JS')
         .skip_tester('nanoarrow')
+        .skip_tester('Ruby')
         .skip_tester('Rust')
         .skip_tester('Go'),
 
@@ -1993,32 +1995,37 @@ def get_generated_json_files(tempdir=None):
         .skip_tester('nanoarrow')
         .skip_tester('Java')  # TODO(ARROW-7779)
         # TODO(https://github.com/apache/arrow/issues/38045)
-        .skip_format(SKIP_FLIGHT, '.NET'),
+        .skip_format(SKIP_FLIGHT, '.NET')
+        .skip_tester('Ruby'),
 
         generate_run_end_encoded_case()
         .skip_tester('.NET')
         .skip_tester('JS')
         # TODO(https://github.com/apache/arrow-nanoarrow/issues/618)
         .skip_tester('nanoarrow')
+        .skip_tester('Ruby')
         .skip_tester('Rust'),
 
         generate_binary_view_case()
         .skip_tester('JS')
         # TODO(https://github.com/apache/arrow-nanoarrow/issues/618)
         .skip_tester('nanoarrow')
+        .skip_tester('Ruby')
         .skip_tester('Rust'),
 
         generate_list_view_case()
         .skip_tester('.NET')     # Doesn't support large list views
         .skip_tester('JS')
         # TODO(https://github.com/apache/arrow-nanoarrow/issues/618)
         .skip_tester('nanoarrow')
+        .skip_tester('Ruby')
         .skip_tester('Rust'),
 
         generate_extension_case()
         .skip_tester('nanoarrow')
         # TODO(https://github.com/apache/arrow/issues/38045)
-        .skip_format(SKIP_FLIGHT, '.NET'),
+        .skip_format(SKIP_FLIGHT, '.NET')
+        .skip_tester('Ruby'),
     ]
 
     generated_paths = []

dev/archery/archery/integration/runner.py

@@ -196,9 +196,11 @@ def _gold_tests(self, gold_dir):
                 skip_testers.add(".NET")
                 skip_testers.add("Java")
                 skip_testers.add("JS")
+                skip_testers.add("Ruby")
                 skip_testers.add("Rust")
             if prefix == '2.0.0-compression':
                 skip_testers.add("JS")
+                skip_testers.add("Ruby")
             if prefix == '2.0.0-compression' and 'lz4' in name:
                 # https://github.com/apache/arrow-nanoarrow/issues/621
                 skip_testers.add("nanoarrow")
@@ -590,9 +592,9 @@ def get_static_json_files():
 
 
 def select_testers(with_cpp=True, with_java=True, with_js=True,
-                   with_dotnet=True, with_go=True, with_rust=False,
-                   with_nanoarrow=False, target_implementations="",
-                   **kwargs):
+                   with_dotnet=True, with_go=True, with_ruby=False,
+                   with_rust=False, with_nanoarrow=False,
+                   target_implementations="", **kwargs):
     target_implementations = (target_implementations.split(",")
                               if target_implementations else [])
 
@@ -629,6 +631,10 @@ def append_tester(implementation, tester):
         from .tester_nanoarrow import NanoarrowTester
         append_tester("nanoarrow", NanoarrowTester(**kwargs))
 
+    if with_ruby:
+        from .tester_ruby import RubyTester
+        append_tester("ruby", RubyTester(**kwargs))
+
     if with_rust:
         from .tester_rust import RustTester
         append_tester("rust", RustTester(**kwargs))

dev/archery/archery/integration/tester_ruby.py

@@ -0,0 +1,78 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+
+from .tester import Tester
+from .util import run_cmd, log
+from ..utils.source import ARROW_ROOT_DEFAULT
+
+
+_EXE_PATH = os.path.join(
+    ARROW_ROOT_DEFAULT, "ruby/red-arrow-format/bin/red-arrow-format-integration-test")
+
+
+class RubyTester(Tester):
+    PRODUCER = True
+    CONSUMER = True
+
+    name = "Ruby"
+
+    def _run(self, env):
+        command_line = [_EXE_PATH]
+        if self.debug:
+            command_line_string = ""
+            for key, value in env.items:
+                command_line_string += f"{key}={value} "
+            command_line_string += " ".join(command_line)
+            log(command_line_string)
+        run_cmd(command_line, env=os.environ | env)
+
+    def validate(self, json_path, arrow_path, quirks=None):
+        env = {
+            "ARROW": arrow_path,
+            "COMMAND": "validate",
+            "JSON": json_path,
+        }
+        if quirks:
+            for quirk in quirks:
+                env[f"QUIRK_{quirk.upper()}"] = "true"
+        self._run(env)
+
+    def json_to_file(self, json_path, arrow_path):
+        env = {
+            "ARROW": arrow_path,
+            "COMMAND": "json-to-file",
+            "JSON": json_path,
+        }
+        self._run(env)
+
+    def stream_to_file(self, stream_path, file_path):
+        env = {
+            "ARROW": file_path,
+            "ARROWS": stream_path,
+            "COMMAND": "stream-to-file",
+        }
+        self._run(env)
+
+    def file_to_stream(self, file_path, stream_path):
+        env = {
+            "ARROW": file_path,
+            "ARROWS": stream_path,
+            "COMMAND": "file-to-stream",
+        }
+        self._run(env)

dev/archery/archery/integration/util.py

@@ -17,6 +17,7 @@
 
 import contextlib
 import io
+import os
 import random
 import socket
 import subprocess
@@ -137,7 +138,13 @@ def run_cmd(cmd, **kwargs):
     except subprocess.CalledProcessError as e:
         # this avoids hiding the stdout / stderr of failed processes
         sio = io.StringIO()
-        print('Command failed:', " ".join(cmd), file=sio)
+        command_line_string = ''
+        env = kwargs.get('env', {})
+        for key in env.keys() - os.environ.keys():
+            value = env[key]
+            command_line_string += f'{key}={value} '
+        command_line_string += ' '.join(cmd)
+        print(f'Command failed: {command_line_string}', file=sio)
         print('With output:', file=sio)
         print('--------------', file=sio)
         print(frombytes(e.output), file=sio)

ruby/red-arrow-format/bin/red-arrow-format-integration-test

@@ -0,0 +1,64 @@
+#!/usr/bin/env ruby
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+require_relative "../lib/arrow-format"
+require_relative "../lib/arrow-format/integration/options"
+
+options = ArrowFormat::Integration::Options.singleton
+case options.command
+when "validate"
+  require_relative "../lib/arrow-format/integration/validate"
+when "json-to-file"
+  require_relative "../lib/arrow-format/integration/json-reader"
+  File.open(options.json, "r") do |input|
+    reader = ArrowFormat::Integration::JSONReader.new(input)
+    File.open(options.arrow, "wb") do |output|
+      writer = ArrowFormat::FileWriter.new(output)
+      writer.start(reader.schema)
+      reader.each do |record_batch|
+        writer.write_record_batch(record_batch)
+      end
+      writer.finish
+    end
+  end
+when "stream-to-file"
+  File.open(options.arrows, "rb") do |input|
+    reader = ArrowFormat::StreamingReader.new(input)
+    File.open(options.arrow, "wb") do |output|
+      writer = ArrowFormat::FileWriter.new(output)
+      writer.start(reader.schema)
+      reader.each do |record_batch|
+        writer.write_record_batch(record_batch)
+      end
+      writer.finish
+    end
+  end
+when "file-to-stream"
+  File.open(options.arrow, "rb") do |input|
+    reader = ArrowFormat::FileReader.new(input)
+    File.open(options.arrows, "wb") do |output|
+      writer = ArrowFormat::StreamingWriter.new(output)
+      writer.start(reader.schema)
+      reader.each do |record_batch|
+        writer.write_record_batch(record_batch)
+      end
+      writer.finish
+    end
+  end
+end