### Rationale for this change

Other file systems such as GCS file system and S3 file system are enabled.

### What changes are included in this PR?

Enabled Azure file system.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* GitHub Issue: #44347

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>

…n wheel-manylinux-*-cp313t-* (#44362)

### Rationale for this change

If interactive deb configuration is used, CI is blocked by waiting an user input.

### What changes are included in this PR?

Disable interactive deb configuration.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* GitHub Issue: #44355

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>

…ckaging (#44148)

## WIP

Creating the PR to validate if / what are the CI failures 

### Rationale for this change
Once the issue has been merged:
- #25118

We do not require numpy as a pyarrow dependency.

### What changes are included in this PR?

Remove numpy as a required dependency

### Are these changes tested?

Via CI

### Are there any user-facing changes?

Yes, pyarrow won't install numpy as a required dependency.
* GitHub Issue: #43846

Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>

…test wheels on Python 3.13 (#44193)

### Rationale for this change

Now that numpy, cython and pandas all have releases that support Python 3.13, we can remove the extra index and ensure that we are building our wheels with the released version instead of nightly dev version.

* GitHub Issue: #43519

Authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>

…test_dataset.py (#44370)

### Rationale for this change

If pyarrow has been built without dataset test collection errors due to pytest.mark.parametrize requiring dataset.

### What changes are included in this PR?

Move dataset requirement outside of test collection by removing it from the parametrization.

### Are these changes tested?

Via archery

### Are there any user-facing changes?

No
* GitHub Issue: #44369

Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>

…ve macOS 12 pyarrow wheels (#44315)

### Rationale for this change

We did just bump MACOSX_DEPLOYMENT_TARGET from 10 to 12 for pyarrow wheels on 18.0.0, we probably should wait a little before dropping.

### What changes are included in this PR?

Revert moving macOS deployment target to 12

### Are these changes tested?

Will trigger wheel jobs on archery

### Are there any user-facing changes?

Yes, wheels will be available for macOS 12.
* GitHub Issue: #44314

Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>

…4374)

### Rationale for this change

Java-jars are currently failing on macOS to install Python 3.13 because `python@ 3.13` (`python` is an alias of `python@ 3.13` now) isn't installed yet. Our link failure workaround is updating already installed `python@*` with `--overwrite` explicitly. `python@ 3.13` (`python`) isn't installed on GitHub Actions runner yet. So `brew bundle` installs `python` without `--overwrite`.

### What changes are included in this PR?

Install `python` with `--overwrite` explicitly before `brew bundle`.

### Are these changes tested?

Via CI

### Are there any user-facing changes?

No
* GitHub Issue: #44373

Lead-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>

### Rationale for this change

When adding Python 3.13 support (#43539), conda was not yet ready to support Python 3.13. But over the last weeks most conda packages having being updated, so trying again.

* GitHub Issue: #43519

Authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>

…cation jobs (#44382)

### Rationale for this change

Verification job is currently failing for Ruby.

### What changes are included in this PR?

We can't use Python 3.13 and `glib==2.80.5`. So we use old Python as a workaround.

### Are these changes tested?

Via archery

### Are there any user-facing changes?

No
* GitHub Issue: #44381

Lead-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>

…fication when using Conda (#44388)

### Rationale for this change

Conda Python 3.13 fails when running verification tasks for Java. It's related to JPype.

See also:
* GH-44389
* https://discuss.python.org/t/api-for-python-3-13-prevents-use-of-3rd-party-gc-allocators/62709/5

### What changes are included in this PR?

Pin Python 3.12 on Conda integration jobs because JPype doesn't work with Python 3.13. We can use Python 3.13 again once JPype supports Python 3.13.

### Are these changes tested?

Yes via CI

### Are there any user-facing changes?

No
* GitHub Issue: #44386

Lead-authored-by: Raúl Cumplido <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>

### Rationale for this change

jemalloc may have a problem on ARM.
See also: #44342

### What changes are included in this PR?

* Disable jemalloc by default on ARM.
* Disable jemalloc for manylinux wheel for ARM.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.

* GitHub Issue: #44342

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>

…rification script (#44407)

### Rationale for this change

We can remove these pins now that conda-forge/glib-feedstock#191 has been closed. I've tested on my own crossbow and will test here momentarily. This effectively reverts #44270 and #44268.

### What changes are included in this PR?

Tweaks to release release verification script, specifically removing recently-added pins.

### Are these changes tested?

Yes: https://github.com/amoeba/crossbow/actions/runs/11335070956/job/31522511285.

### Are there any user-facing changes?

No.

Closes #44415.
* GitHub Issue: #44415

Authored-by: Bryce Mecum <[email protected]>
Signed-off-by: Bryce Mecum <[email protected]>

…44425)

### Rationale for this change

It seems that the current `quay.io/centos/centos:stream9` for `linux/aarch64` is broken.

We use it for building and testing but testing is only broken for now. Because we use cached (a bit old) `quay.io/centos/centos:stream9` for now. Building will be failed when the cache is expired.

### What changes are included in this PR?

Change upload phase to after the build from after the test. Test is still failing but we can proceed our release process by this.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* GitHub Issue: #44422

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>

### Rationale for this change

See #44353 

`UnionMapWriter` does not override `map()` and falls back to implementation in `UnionListWriter` which causes exception when used.

### What changes are included in this PR?

Implement `map()` for `UnionMapWriter`.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Changed behavior of the related function. 

* GitHub Issue: #44353

Authored-by: 0xderek <[email protected]>
Signed-off-by: David Li <[email protected]>

…44377)

### Rationale for this change

See #44361. This allows testing compatibility of the .NET Flight implementation with other Flight implementations.

### What changes are included in this PR?

* Adds a new `Apache.Arrow.Flight.IntegrationTest` project that can run in server or client mode for Flight integration tests.
* Includes the integration tests that send then retrieve data defined in JSON files, but doesn't add any of the named scenarios
* Configures archery to include C# in the Flight integration tests, but skip all the named scenarios
* Also skips tests that use dictionary data due to #38045, and the empty data test due to #44363

### Are these changes tested?

These changes are tests.

### Are there any user-facing changes?

No
* GitHub Issue: #44361

Authored-by: Adam Reeve <[email protected]>
Signed-off-by: Curt Hagenlocher <[email protected]>

…ntation (#44424)

### Rationale for this change

See #44360

### What changes are included in this PR?

* Adds a new integration test to allow testing `do_exchange` between C++/Python and .NET.
* Updates the Flight stream reader to handle when a descriptor is sent in the first message without any schema.

### Are these changes tested?

* Yes, using the new integration test.

### Are there any user-facing changes?

No
* GitHub Issue: #44360

Authored-by: Adam Reeve <[email protected]>
Signed-off-by: Curt Hagenlocher <[email protected]>

…source-integration-linux-conda-latest-amd64 (#44395)

### Rationale for this change

Because JPype doesn't support Python 3.13 yet.

See also:
* GH-44389
* https://discuss.python.org/t/api-for-python-3-13-prevents-use-of-3rd-party-gc-allocators/62709/5

### What changes are included in this PR?

Pin Python 3.12.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* GitHub Issue: #44389

Lead-authored-by: Raúl Cumplido <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>

### Rationale for this change

Executable scripts should have shebang.

### What changes are included in this PR?

Add `#!/usr/bin/env bash`.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* GitHub Issue: #44453

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Jacob Wujciak-Jens <[email protected]>

### Rationale for this change

`dev/release/06-java-upload.sh` sometimes failed with the following HTTP error:

```text
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy-file (default-cli) on project standalone-pom: Failed to deploy artifacts: Could not transfer artifact org.apache.arrow:arrow-compression:pom:18.0.0 from/to apache.releases.https (https://repository.apache.org/service/local/staging/deploy/maven2): transfer failed for https://repository.apache.org/service/local/staging/deploy/maven2/org/apache/arrow/arrow-compression/18.0.0/arrow-compression-18.0.0.pom, status: 408 Request Timeout -> [Help 1]
```

### What changes are included in this PR?

Use `-DretryFailedDeploymentCount=10` for retry on HTTP error.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* GitHub Issue: #44451

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Jacob Wujciak-Jens <[email protected]>

…ing binary wheels (#44460)

### Rationale for this change

Currently the folder used to verify the contents of the wheels is not correct when doing a release verification of the wheels and the verification fails. The script is supposed to check wheels individually and we are already verifying the contents on the wheels individually when generating the wheels.

### What changes are included in this PR?

Stop checking wheel content on release verification as the wheel content has already been validated when generating the wheels.

There is also a minor fix on the platform tags for the macOS wheels that was missed when we updated the MACOSX_DEPLOYMENT_TARGET.

### Are these changes tested?

Yes via archery.

### Are there any user-facing changes?

No
* GitHub Issue: #44459

Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>

### Rationale for this change

Artifactory sometimes reports the following 503 HTTP error:

```text
BinaryTask::ArtifactoryClient::Error: failed to request: https://apache.jfrog.io/artifactory/arrow/debian-rc/pool/bookworm/main/a/apache-arrow/libparquet-glib1800_18.0.0-1_arm64.deb: PUT:  503
{
  "errors" : [ {
    "status" : 503,
    "message" : "503 : Failed during addStream to bucket"
  } ]
}
```

It seems that this is a temporary error. So we can retry on this type error for stable upload.

### What changes are included in this PR?

Retry on HTTP error.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* GitHub Issue: #44449

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>

### Rationale for this change

Adopting #40226. 

 The creation and return of a shared_ptr does result in some performance overhead, that makes a difference for a performance-sensitive application.

If someone could use ChunkResolver to learn the indices, they could then instead access the data directly. 

### What changes are included in this PR?

- [X] Updates to documentation (thanks to @ SChakravorti21 )
- [X] Moving `ChunkResolver` to public API, and updating all references to it in the code

### Are these changes tested?

There seemed to be comprehensive tests already: https://github.com/apache/arrow/blob/main/cpp/src/arrow/chunked_array_test.cc#L324 If an edgecase is missing, I'd be happy to add it.

### Are there any user-facing changes?

`ChunkResolver` and `TypedChunkLocation` are now in the public API.
* GitHub Issue: #34535

Lead-authored-by: Anja Kefala <[email protected]>
Co-authored-by: anjakefala <[email protected]>
Co-authored-by: Bryce Mecum <[email protected]>
Co-authored-by: SChakravorti21 <[email protected]>
Co-authored-by: SChakravorti21<[email protected]>
Co-authored-by: Jacob Wujciak-Jens <[email protected]>
Signed-off-by: Felipe Oliveira Carvalho <[email protected]>

…44505)

### Rationale for this change

Since ChunkLocation is a typedef/alias, it needs to use the `doxygentypedef` Sphinx directive rather than `doxygenstruct` in order for it to show up in the generated HTML. Otherwise we get an error.

### What changes are included in this PR?

Just docs:

- Replace `doxygenstruct` directive with `doxygentypedef`
- Remove `:members:` directive as it's not needed

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes, though just docs.
* GitHub Issue: #44506

Authored-by: Bryce Mecum <[email protected]>
Signed-off-by: Bryce Mecum <[email protected]>

…cross-compiling (#44507)

### Rationale for this change

We can't use `find_package(gRPC)` and `gRPC::grpc_cpp_plugin` for cross-compiling because it's for host. We need `grpc_cpp_plugin` for target in cross-compiling.

### What changes are included in this PR?

Add `ARROW_GRPC_CPP_PLUGIN` CMake option that overwrites `gRPC::grpc_cpp_plugin` path found by `find_package(gRPC)`.

### Are these changes tested?

Yes.

conda-forge/arrow-cpp-feedstock#1432

### Are there any user-facing changes?

Yes.
* GitHub Issue: #44448

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>

…y-release-candidate-wheels.bat (#44462)

### Rationale for this change

The Windows wheel verification fails due to missing `PARQUET_TEST_DATA`

### What changes are included in this PR?

Add `PARQUET_TEST_DATA` to `verify-release-candidate-wheels.bat` which is only tested on the binary verification job.

### Are these changes tested?

Via archery

### Are there any user-facing changes?

No
* GitHub Issue: #44461

Lead-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Raúl Cumplido <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>

…on (#44613)

### Rationale for this change
Abseil's recommended usage is to build from trunk, this version does not provide a package version in CMake which breaks `resolve_dependency`.

### What changes are included in this PR?

Add `FindabslAlt.cmake` that adds a big version if it is missing to compare against our minimal version requirements.

### Are these changes tested?

CI

### Are there any user-facing changes?

Abseil build from head should now work without workaround.
* GitHub Issue: #44606

Lead-authored-by: Jacob Wujciak-Jens <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Jacob Wujciak-Jens <[email protected]>