GH-47560: [C++] Fix host handling for default HDFS URI

[dsevilla]

Rationale for this change

In #25324 a fix is introduced for the python HadoopFileSystem, but it does not work if you use from_uri(), as it is passed to the underlying C++ implementation of the options parsing. The "default" case is not handled as in the python case, as the whole "hdfs://default" is passed to the underlying hdfs library, that expect "default" to search in $HADOOP_CONF_DIR/core-site.xml.

What changes are included in this PR?

Handle the HadoopFileSystem.from_uri() (or FileSystem.from_uri() when using hdfs://default:xxx) special HDFS URIs.

Are these changes tested?

There are no specific tests for this feature, but existing HDFS CI jobs pass.

Are there any user-facing changes?

Not exactly, but the documentation is honored for the from_uri() case.

• GitHub Issue: [Python] pyarrow.fs.HadoopFileSystem.from_uri - retrieve options from core-site.xml or hdfs-site.xml if available (variation of #25324) #47560

[github-actions]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[dsevilla]

Any work on this? It is not working the "default" value of HDFS host if used in from_uri().

[dsevilla]

I added a new issue. Now this pull request fixes #47560.

[AlenkaF]

Thank you for proposing a fix!
In principle the change looks ok to me. The comment did confuse me a bit (Special case host = "default" or "hdfs://default" as stated by #25324.). If I understand correctly this change is only dealing with host being default and not hdfs://default also?

I am not sure how to go about testing this. cc @pitrou

[dsevilla]

Hello!

Thank you for proposing a fix! In principle the change looks ok to me. The comment did confuse me a bit (Special case host = "default" or "hdfs://default" as stated by #25324.). If I understand correctly this change is only dealing with host being default and not hdfs://default also?

I think the logic goes like this:

• If the URI is hdfs://default:0/..., then, uri.host() is "default", and uri.scheme() is hdfs.
• If the URI is default:0/..., then again uri.host() is still "default", but uri.scheme() is empty.

Then, in either case, I leave the variable host (that will hold the host plus optional scheme and will be passed down to the library) as just "default". This way libhdfs will see just default:0/....

If the host is not "default" in any case, the old behaviour (scheme://host:port) is maintained.

I am not sure how to go about testing this. cc @pitrou

I am not sure, either. I will investigate, but I remember a test for HadoopFileSystem using "default" as a host (that works as per the other previous patch). It can be added there. I'll search for it ASAP.

I'll look also why the CI is failing, because this change is so small that it is unlikely it is failing because of it.

[pitrou]

I'm not sure we need to test this specifically. OTOH, the hdfs Crossbow-based CI jobs should not regress.

[dsevilla]

@pitrou I've synched the branch just in case that precise moment the tests were failing. We'll see because as I said, the changes are so small that should not affect any CI build.

[pitrou]

@github-actions crossbow submit hdfs

[github-actions]

Revision: 4561d70

Submitted crossbow builds: ursacomputing/crossbow @ actions-2f87af1fda

Task	Status
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1

[dsevilla]

OK, the tests pass all except for a timeout in the azure connection (lasted for 56 minutes).

[github-actions]

⚠️ GitHub issue #47560 has been automatically assigned in GitHub to PR creator.

[dsevilla]

Ouch. Comment length exceeded that expected by the linter :)

[pitrou]

CI failures are unrelated, will merge.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 7129321.

There was 1 benchmark result indicating a performance regression:

• Commit Run on amd64-m5-4xlarge-linux at 2025-09-23 19:12:54Z
  • partitioned-dataset-filter (R) with dataset=dataset-taxi-parquet, language=R, query=vignette

The full Conbench report has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them.