-
Notifications
You must be signed in to change notification settings - Fork 66
Add China mirror support for dependencies #319
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Subham-KRLX
wants to merge
3
commits into
apache:main
Choose a base branch
from
Subham-KRLX:add-china-mirrors
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+195
−16
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,118 @@ | ||
| <!-- | ||
| ~ Licensed to the Apache Software Foundation (ASF) under one | ||
| ~ or more contributor license agreements. See the NOTICE file | ||
| ~ distributed with this work for additional information | ||
| ~ regarding copyright ownership. The ASF licenses this file | ||
| ~ to you under the Apache License, Version 2.0 (the | ||
| ~ "License"); you may not use this file except in compliance | ||
| ~ with the License. You may obtain a copy of the License at | ||
| ~ | ||
| ~ http://www.apache.org/licenses/LICENSE-2.0 | ||
| ~ | ||
| ~ Unless required by applicable law or agreed to in writing, | ||
| ~ software distributed under the License is distributed on an | ||
| ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| ~ KIND, either express or implied. See the License for the | ||
| ~ specific language governing permissions and limitations | ||
| ~ under the License. | ||
| --> | ||
|
|
||
| # Building in China | ||
|
|
||
| This guide helps developers in China who may experience network issues when downloading dependencies from GitHub or international mirrors. | ||
|
|
||
| ## Using Custom Mirror URLs | ||
|
|
||
| If you experience download timeouts, you can override the default dependency URLs using environment variables: | ||
|
|
||
| ```bash | ||
| export ICEBERG_ARROW_URL="<your-mirror-url>/apache-arrow-22.0.0.tar.gz" | ||
| export ICEBERG_NANOARROW_URL="<your-mirror-url>/apache-arrow-nanoarrow-0.7.0.tar.gz" | ||
| export ICEBERG_CROARING_URL="<your-mirror-url>/CRoaring-v4.3.11.tar.gz" | ||
| export ICEBERG_NLOHMANN_JSON_URL="<your-mirror-url>/json-v3.11.3.tar.xz" | ||
| export ICEBERG_SPDLOG_URL="<your-mirror-url>/spdlog-v1.15.3.tar.gz" | ||
| export ICEBERG_CPR_URL="<your-mirror-url>/cpr-1.12.0.tar.gz" | ||
|
|
||
| # For Avro (git repository): | ||
| export ICEBERG_AVRO_GIT_URL="<your-git-mirror>/avro.git" | ||
| # Or if you have a tarball: | ||
| export ICEBERG_AVRO_URL="<your-mirror-url>/avro.tar.gz" | ||
| ``` | ||
|
|
||
| Then build as usual: | ||
|
|
||
| ```bash | ||
| cmake -S . -B build | ||
| cmake --build build | ||
| ``` | ||
|
|
||
| ## Alternative Solutions | ||
|
|
||
| 1. **Use system packages**: Install dependencies via your system package manager | ||
| 2. **Use a proxy**: Set `https_proxy` environment variable | ||
| 3. **Pre-download**: Manually download tarballs to `~/.cmake/Downloads/` | ||
|
|
||
| ## Getting Help | ||
|
|
||
| If you continue experiencing build issues, please open an issue at https://github.com/apache/iceberg-cpp/issues with details about which dependency failed. | ||
|
|
||
| # Building in China | ||
|
|
||
| This guide helps developers in China build iceberg-cpp when network access to GitHub and other international sites is limited. | ||
|
|
||
| ## Mirror Support | ||
|
|
||
| The build system automatically tries alternative download mirrors when the primary URL fails. All third-party dependencies have been configured with China-based mirrors. | ||
|
|
||
| ### Available Mirrors | ||
|
|
||
| Dependencies are automatically downloaded from these mirror sites: | ||
|
|
||
| **Apache Projects (Arrow, Nanoarrow):** | ||
| - Tsinghua University: https://mirrors.tuna.tsinghua.edu.cn/apache/ | ||
| - USTC: https://mirrors.ustc.edu.cn/apache/ | ||
|
|
||
| **GitHub Projects (CRoaring, nlohmann-json, spdlog, cpr):** | ||
| - Gitee: https://gitee.com/mirrors/ | ||
| - FastGit: https://hub.fastgit.xyz/ | ||
|
|
||
| **Note**: Avro requires a git repository (unreleased version). Automatic mirror fallback is not available for git repositories, but you can specify a custom git mirror using the `ICEBERG_AVRO_GIT_URL` environment variable. | ||
|
|
||
| ### Custom Mirror URLs | ||
|
|
||
| To override the default mirrors, set environment variables before running CMake: | ||
|
|
||
| ```bash | ||
| export ICEBERG_ARROW_URL="https://mirrors.tuna.tsinghua.edu.cn/apache/arrow/arrow-22.0.0/apache-arrow-22.0.0.tar.gz" | ||
| export ICEBERG_NANOARROW_URL="https://mirrors.tuna.tsinghua.edu.cn/apache/arrow/apache-arrow-nanoarrow-0.7.0/apache-arrow-nanoarrow-0.7.0.tar.gz" | ||
| export ICEBERG_CROARING_URL="https://gitee.com/mirrors/CRoaring/repository/archive/v4.3.11.tar.gz" | ||
| export ICEBERG_NLOHMANN_JSON_URL="https://gitee.com/mirrors/JSON-for-Modern-CPP/releases/download/v3.11.3/json.tar.xz" | ||
| export ICEBERG_SPDLOG_URL="https://gitee.com/mirrors/spdlog/repository/archive/v1.15.3.tar.gz" | ||
| export ICEBERG_CPR_URL="https://gitee.com/mirrors/cpr/repository/archive/1.12.0.tar.gz" | ||
|
|
||
| # For Avro, you can use either a tarball URL or a git repository URL: | ||
| export ICEBERG_AVRO_URL="https://example.com/avro.tar.gz" # if you have a tarball | ||
| # OR | ||
| export ICEBERG_AVRO_GIT_URL="https://gitee.com/mirrors/avro.git" # for git mirror | ||
| ``` | ||
|
|
||
| Then build as usual: | ||
|
|
||
| ```bash | ||
| cmake -S . -B build | ||
| cmake --build build | ||
| ``` | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| **Download failures:** | ||
| - Try setting a specific mirror using environment variables | ||
| - Use a VPN or proxy: `export https_proxy=http://proxy:port` | ||
| - Pre-download tarballs to `~/.cmake/Downloads/` | ||
|
|
||
| **Slow downloads:** | ||
| - The build will automatically retry with different mirrors | ||
| - Consider using Meson build system as an alternative | ||
|
|
||
| **Still having issues?** | ||
| Open an issue at https://github.com/apache/iceberg-cpp/issues with details about which dependency failed and the error message. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
-1 from me. sorry about that :(
One quick question: do other Apache projects handle this the same way for China?
iceberg-cpp should build against system libraries dependencies, and I think that's probably the right direction.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree with you. We have CMake options such as
ICEBERG_ARROW_URLwhich allow users to customize urls. If other projects use specific urls for China, I would also support following this approach.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the review!
To clarify: this PR doesn’t introduce region specific behavior or change defaults. It only provides optional mirror URLs just like
ICEBERG_ARROW_URLto make the build workable for developers in China who frequently hit GitHub timeouts.If preferred, I can simplify the PR so it only adds optional CMake variables + documentation, matching how other Apache projects handle this.
Please let me know and I’ll update it accordingly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here is my -1 too. I don't think it's good idea to add other mirrors in
ARROW_SOURCE_URLtoo.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After researching Apache best practices I removed all hardcoded mirror URLs and kept only optional environment variables for custom mirrors just like ICEBERG_ARROW_URL. I added documentation showing how users can set these variables if needed. No defaults changed—everything still defaults to the original URLs. This keeps the build flexible respects Apache guidelines and helps developers facing network issues. Please review and let me know if you need any changes.