Skip to content

25.8 Add source code upload job #1079

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Enmk merged 6 commits into from
Oct 16, 2025
Merged

25.8 Add source code upload job #1079

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
71fd9ed
Add source code upload job
MyroTk Oct 15, 2025
29c28d4
add source upload to PR ci
MyroTk Oct 15, 2025
b4847ba
path fix
MyroTk Oct 15, 2025
5641136
temp path fix
MyroTk Oct 15, 2025
f3fb9ad
install aws cli
MyroTk Oct 16, 2025
1c92079
fix aws
MyroTk Oct 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions .github/workflows/master.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4327,3 +4327,38 @@ jobs:
with:
workflow_config: ${{ needs.config_workflow.outputs.data }}
final: true

SourceUpload:
needs: [config_workflow, build_amd_release]
if: ${{ !failure() && !cancelled() }}
runs-on: [self-hosted, altinity-on-demand, altinity-style-checker-aarch64]
env:
COMMIT_SHA: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
PR_NUMBER: ${{ github.event.pull_request.number || 0 }}
VERSION: ${{ fromJson(needs.config_workflow.outputs.data).custom_data.version.string }}
steps:
- name: Check out repository code
uses: Altinity/checkout@19599efdf36c4f3f30eb55d5bb388896faea69f6
with:
clear-repository: true
ref: ${{ fromJson(needs.config_workflow.outputs.data).git_ref }}
submodules: true
fetch-depth: 0
filter: tree:0
- name: Install aws cli
uses: unfor19/install-aws-cli-action@v1
with:
version: 2
arch: arm64
- name: Create source tar
run: |
cd .. && tar czf $RUNNER_TEMP/build_source.src.tar.gz ClickHouse/
- name: Upload source tar
run: |
if [ "$PR_NUMBER" -eq 0 ]; then
S3_PATH="REFs/$GITHUB_REF_NAME/$COMMIT_SHA/build_amd_release"
else
S3_PATH="PRs/$PR_NUMBER/$COMMIT_SHA/build_amd_release"
fi

aws s3 cp $RUNNER_TEMP/build_source.src.tar.gz s3://altinity-build-artifacts/$S3_PATH/clickhouse-$VERSION.src.tar.gz
35 changes: 35 additions & 0 deletions .github/workflows/pull_request.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4261,3 +4261,38 @@ jobs:
with:
workflow_config: ${{ needs.config_workflow.outputs.data }}
final: true

SourceUpload:
needs: [config_workflow, build_amd_release]
if: ${{ !failure() && !cancelled() }}
runs-on: [self-hosted, altinity-on-demand, altinity-style-checker-aarch64]
env:
COMMIT_SHA: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
PR_NUMBER: ${{ github.event.pull_request.number || 0 }}
VERSION: ${{ fromJson(needs.config_workflow.outputs.data).custom_data.version.string }}
steps:
- name: Check out repository code
uses: Altinity/checkout@19599efdf36c4f3f30eb55d5bb388896faea69f6
with:
clear-repository: true
ref: ${{ fromJson(needs.config_workflow.outputs.data).git_ref }}
submodules: true
fetch-depth: 0
filter: tree:0
- name: Install aws cli
uses: unfor19/install-aws-cli-action@v1
with:
version: 2
arch: arm64
- name: Create source tar
run: |
cd .. && tar czf $RUNNER_TEMP/build_source.src.tar.gz ClickHouse/
- name: Upload source tar
run: |
if [ "$PR_NUMBER" -eq 0 ]; then
S3_PATH="REFs/$GITHUB_REF_NAME/$COMMIT_SHA/build_amd_release"
else
S3_PATH="PRs/$PR_NUMBER/$COMMIT_SHA/build_amd_release"
fi

aws s3 cp $RUNNER_TEMP/build_source.src.tar.gz s3://altinity-build-artifacts/$S3_PATH/clickhouse-$VERSION.src.tar.gz
35 changes: 35 additions & 0 deletions .github/workflows/release_builds.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1208,3 +1208,38 @@ jobs:
with:
workflow_config: ${{ needs.config_workflow.outputs.data }}
final: true

SourceUpload:
needs: [config_workflow, build_amd_release]
if: ${{ !failure() && !cancelled() }}
runs-on: [self-hosted, altinity-on-demand, altinity-style-checker-aarch64]
env:
COMMIT_SHA: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
PR_NUMBER: ${{ github.event.pull_request.number || 0 }}
VERSION: ${{ fromJson(needs.config_workflow.outputs.data).custom_data.version.string }}
steps:
- name: Check out repository code
uses: Altinity/checkout@19599efdf36c4f3f30eb55d5bb388896faea69f6
with:
clear-repository: true
ref: ${{ fromJson(needs.config_workflow.outputs.data).git_ref }}
submodules: true
fetch-depth: 0
filter: tree:0
- name: Install aws cli
uses: unfor19/install-aws-cli-action@v1
with:
version: 2
arch: arm64
- name: Create source tar
run: |
cd .. && tar czf $RUNNER_TEMP/build_source.src.tar.gz ClickHouse/
- name: Upload source tar
run: |
if [ "$PR_NUMBER" -eq 0 ]; then
S3_PATH="REFs/$GITHUB_REF_NAME/$COMMIT_SHA/build_amd_release"
else
S3_PATH="PRs/$PR_NUMBER/$COMMIT_SHA/build_amd_release"
fi

aws s3 cp $RUNNER_TEMP/build_source.src.tar.gz s3://altinity-build-artifacts/$S3_PATH/clickhouse-$VERSION.src.tar.gz
36 changes: 36 additions & 0 deletions ci/praktika/yaml_additional_templates.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,42 @@ class AltinityWorkflowTemplates:
with:
workflow_config: ${{ needs.config_workflow.outputs.data }}
final: true
""",
"SourceUpload": r"""
SourceUpload:
needs: [config_workflow, build_amd_release]
if: ${{ !failure() && !cancelled() }}
runs-on: [self-hosted, altinity-on-demand, altinity-style-checker-aarch64]
env:
COMMIT_SHA: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
PR_NUMBER: ${{ github.event.pull_request.number || 0 }}
VERSION: ${{ fromJson(needs.config_workflow.outputs.data).custom_data.version.string }}
steps:
- name: Check out repository code
uses: Altinity/checkout@19599efdf36c4f3f30eb55d5bb388896faea69f6
with:
clear-repository: true
ref: ${{ fromJson(needs.config_workflow.outputs.data).git_ref }}
submodules: true
fetch-depth: 0
filter: tree:0
- name: Install aws cli
uses: unfor19/install-aws-cli-action@v1
with:
version: 2
arch: arm64
- name: Create source tar
run: |
cd .. && tar czf $RUNNER_TEMP/build_source.src.tar.gz ClickHouse/
- name: Upload source tar
run: |
if [ "$PR_NUMBER" -eq 0 ]; then
S3_PATH="REFs/$GITHUB_REF_NAME/$COMMIT_SHA/build_amd_release"
else
S3_PATH="PRs/$PR_NUMBER/$COMMIT_SHA/build_amd_release"
fi

aws s3 cp $RUNNER_TEMP/build_source.src.tar.gz s3://altinity-build-artifacts/$S3_PATH/clickhouse-$VERSION.src.tar.gz
""",
}
ADDITIONAL_JOBS_BANNER = r"""
Expand Down
3 changes: 3 additions & 0 deletions ci/praktika/yaml_generator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -519,6 +519,9 @@ def generate(self):
res += AltinityWorkflowTemplates.ALTINITY_JOBS["CIReport"].replace(
"{ALL_JOBS}", ALL_JOBS
)
if "SourceUpload" in self.workflow_config.additional_jobs:
res += AltinityWorkflowTemplates.ALTINITY_JOBS["SourceUpload"]
ALL_JOBS += "\n - SourceUpload"

return res

Expand Down
2 changes: 1 addition & 1 deletion ci/workflows/master.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
# *JobConfigs.sqlancer_master_jobs,
JobConfigs.sqltest_master_job,
],
additional_jobs=["GrypeScan", "Regression", "SignRelease", "CIReport"],
additional_jobs=["GrypeScan", "Regression", "SignRelease", "CIReport", "SourceUpload"],
artifacts=[
*ArtifactConfigs.unittests_binaries,
*ArtifactConfigs.clickhouse_binaries,
Expand Down
2 changes: 1 addition & 1 deletion ci/workflows/pull_request.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@
# for job in JobConfigs.performance_comparison_with_master_head_jobs
# ], # NOTE (strtgbb): failed previously due to GH secrets not being handled properly, try again later
],
additional_jobs=["GrypeScan", "Regression", "CIReport"],
additional_jobs=["GrypeScan", "Regression", "CIReport", "SourceUpload"],
artifacts=[
*ArtifactConfigs.unittests_binaries,
*ArtifactConfigs.clickhouse_binaries,
Expand Down
2 changes: 1 addition & 1 deletion ci/workflows/release_builds.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
if any(t in job.name for t in ("release", "binary"))
],
],
additional_jobs=["GrypeScan", "SignRelease", "CIReport"],
additional_jobs=["GrypeScan", "SignRelease", "CIReport", "SourceUpload"],
artifacts=[
*ArtifactConfigs.clickhouse_binaries,
*ArtifactConfigs.clickhouse_stripped_binaries,
Expand Down
Loading