Skip to content

24.3 Snapshot runners #439

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Enmk merged 25 commits into from
Nov 21, 2024
+88 −111
Merged

24.3 Snapshot runners #439

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
b29111c
Update common setup runners to use snapshots
MyroTk Aug 15, 2024
218c4a5
Update release_branches.yml
MyroTk Aug 15, 2024
e7c1e97
Update release_branches.yml
MyroTk Aug 15, 2024
95683d6
Update release_branches.yml
MyroTk Aug 15, 2024
73a92d3
check pr_info event for pull request
MyroTk Sep 19, 2024
3c195cc
Update release_branches.yml
MyroTk Oct 1, 2024
a147b7c
Update release_branches.yml
MyroTk Oct 1, 2024
27f0251
Switch to new snapshot and startup config. Enable zram
MyroTk Oct 4, 2024
d06d994
Update release_branches.yml
MyroTk Oct 10, 2024
4de4906
Fix retry path
MyroTk Oct 10, 2024
aec2636
Update ci.py
MyroTk Oct 17, 2024
b49c0bf
update regression tests and use pr number 1 for scheduled runs
MyroTk Oct 17, 2024
ef7cd8d
Update regression.yml
MyroTk Oct 18, 2024
044dafd
Removing location from runners. Minor scheduled run fixes
MyroTk Oct 22, 2024
af3fb93
Update release_branches.yml
MyroTk Oct 29, 2024
1ce8b07
Merge branch 'customizations/24.3.11' into 24.3.5-snapshot_runners
MyroTk Oct 29, 2024
c9d279f
Merge branch 'customizations/24.3.12' into 24.3.5-snapshot_runners
MyroTk Oct 29, 2024
dd1a05f
Update release_branches.yml
MyroTk Oct 30, 2024
4d7a766
Update release_branches.yml
MyroTk Oct 31, 2024
2f7c92c
Merge branch 'customizations/24.3.12' into 24.3.5-snapshot_runners
MyroTk Nov 4, 2024
0ae4c28
Update regression commit
MyroTk Nov 7, 2024
0e43ab1
Update clickhouse-test retry messages to include Memory Limit
MyroTk Nov 7, 2024
27d980c
Update regression
MyroTk Nov 14, 2024
1d76f22
Update release_branches.yml
MyroTk Nov 19, 2024
b4aba3d
Update release_branches.yml
MyroTk Nov 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .github/actions/common_setup/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,16 @@ runs:
run: |
# to remove every leftovers
sudo rm -fr "$TEMP_PATH" && mkdir -p "$TEMP_PATH"
- name: Setup zram
shell: bash
run: |
sudo modprobe zram
MemTotal=$(grep -Po "(?<=MemTotal:)\s+\d+" /proc/meminfo) # KiB
Percent=200
ZRAM_SIZE=$(($MemTotal / 1024 / 1024 * $Percent / 100)) # Convert to GiB
.github/retry.sh 30 2 sudo zramctl --size ${ZRAM_SIZE}GiB --algorithm zstd /dev/zram0
sudo mkswap /dev/zram0 && sudo swapon -p 100 /dev/zram0
sudo sysctl vm.swappiness=200
Copy link
Member

@Enmk Enmk Nov 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why set swappiness to 200?

- name: Tune vm.mmap_rnd_bits for sanitizers
shell: bash
run: |
Expand Down
41 changes: 23 additions & 18 deletions .github/workflows/regression.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,13 +93,17 @@ env:
AWS_DEFAULT_REGION: ${{ secrets.AWS_REPORT_REGION }}
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
CHECKS_DATABASE_HOST: ${{ secrets.CHECKS_DATABASE_HOST }}
CHECKS_DATABASE_USER: ${{ secrets.CHECKS_DATABASE_USER }}
CHECKS_DATABASE_PASSWORD: ${{ secrets.CHECKS_DATABASE_PASSWORD }}
args: --test-to-end
--no-colors
--local
--collect-service-logs
--output classic
--parallel 1
--log raw.log
--with-analyzer
artifacts: builds
artifact_paths: |
./report.html
Expand Down Expand Up @@ -173,8 +177,8 @@ jobs:
run: EXITCODE=0;
python3
-u ${{ env.SUITE }}/regression.py
--clickhouse-binary-path ${{ env.clickhouse_binary_path }}
--attr project="$GITHUB_REPOSITORY" project.id="$GITHUB_REPOSITORY_ID" package="${{ env.clickhouse_binary_path }}" version="${{ env.version }}" user.name="$GITHUB_ACTOR" repository="https://github.com/Altinity/clickhouse-regression" commit.hash="$(git rev-parse HEAD)" job.id="$GITHUB_RUN_ID" job.url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" arch="$(uname -i)"
--clickhouse-binary-path ${{ env.clickhouse_path }}
--attr project="$GITHUB_REPOSITORY" project.id="$GITHUB_REPOSITORY_ID" package="${{ env.clickhouse_path }}" version="${{ env.version }}" user.name="$GITHUB_ACTOR" repository="https://github.com/Altinity/clickhouse-regression" commit.hash="$(git rev-parse HEAD)" job.id="$GITHUB_RUN_ID" job.url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" arch="$(uname -i)"
${{ env.args }} || EXITCODE=$?;
.github/add_link_to_logs.sh;
exit $EXITCODE
Expand Down Expand Up @@ -224,9 +228,9 @@ jobs:
run: EXITCODE=0;
python3
-u alter/regression.py
--clickhouse-binary-path ${{ env.clickhouse_binary_path }}
--clickhouse-binary-path ${{ env.clickhouse_path }}
--only "/alter/${{ matrix.ONLY }} partition/*"
--attr project="$GITHUB_REPOSITORY" project.id="$GITHUB_REPOSITORY_ID" package="${{ env.clickhouse_binary_path }}" version="${{ env.version }}" user.name="$GITHUB_ACTOR" repository="https://github.com/Altinity/clickhouse-regression" commit.hash="$(git rev-parse HEAD)" job.id="$GITHUB_RUN_ID" job.url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" arch="$(uname -i)"
--attr project="$GITHUB_REPOSITORY" project.id="$GITHUB_REPOSITORY_ID" package="${{ env.clickhouse_path }}" version="${{ env.version }}" user.name="$GITHUB_ACTOR" repository="https://github.com/Altinity/clickhouse-regression" commit.hash="$(git rev-parse HEAD)" job.id="$GITHUB_RUN_ID" job.url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" arch="$(uname -i)"
${{ env.args }} || EXITCODE=$?;
.github/add_link_to_logs.sh;
exit $EXITCODE
Expand Down Expand Up @@ -276,7 +280,7 @@ jobs:
run: EXITCODE=0;
python3
-u ${{ env.SUITE }}/benchmark.py
--clickhouse-binary-path ${{ env.clickhouse_binary_path }}
--clickhouse-binary-path ${{ env.clickhouse_path }}
--storage ${{ matrix.STORAGE }}
--gcs-uri ${{ secrets.REGRESSION_GCS_URI }}
--gcs-key-id ${{ secrets.REGRESSION_GCS_KEY_ID }}
Expand All @@ -285,7 +289,7 @@ jobs:
--aws-s3-region ${{ secrets.REGRESSION_AWS_S3_REGION }}
--aws-s3-key-id ${{ secrets.REGRESSION_AWS_S3_KEY_ID }}
--aws-s3-access-key ${{ secrets.REGRESSION_AWS_S3_SECRET_ACCESS_KEY }}
--attr project="$GITHUB_REPOSITORY" project.id="$GITHUB_REPOSITORY_ID" package="${{ env.clickhouse_binary_path }}" version="${{ env.version }}" user.name="$GITHUB_ACTOR" repository="https://github.com/Altinity/clickhouse-regression" commit.hash="$(git rev-parse HEAD)" job.id="$GITHUB_RUN_ID" job.url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" arch="$(uname -i)"
--attr project="$GITHUB_REPOSITORY" project.id="$GITHUB_REPOSITORY_ID" package="${{ env.clickhouse_path }}" version="${{ env.version }}" user.name="$GITHUB_ACTOR" repository="https://github.com/Altinity/clickhouse-regression" commit.hash="$(git rev-parse HEAD)" job.id="$GITHUB_RUN_ID" job.url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" arch="$(uname -i)"
${{ env.args }} || EXITCODE=$?;
.github/add_link_to_logs.sh;
exit $EXITCODE
Expand Down Expand Up @@ -332,8 +336,8 @@ jobs:
python3
-u ${{ env.SUITE }}/regression.py
--ssl
--clickhouse-binary-path ${{ env.clickhouse_binary_path }}
--attr project="$GITHUB_REPOSITORY" project.id="$GITHUB_REPOSITORY_ID" package="${{ env.clickhouse_binary_path }}" version="${{ env.version }}" user.name="$GITHUB_ACTOR" repository="https://github.com/Altinity/clickhouse-regression" commit.hash="$(git rev-parse HEAD)" job.id="$GITHUB_RUN_ID" job.url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" arch="$(uname -i)"
--clickhouse-binary-path ${{ env.clickhouse_path }}
--attr project="$GITHUB_REPOSITORY" project.id="$GITHUB_REPOSITORY_ID" package="${{ env.clickhouse_path }}" version="${{ env.version }}" user.name="$GITHUB_ACTOR" repository="https://github.com/Altinity/clickhouse-regression" commit.hash="$(git rev-parse HEAD)" job.id="$GITHUB_RUN_ID" job.url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" arch="$(uname -i)"
${{ env.args }} || EXITCODE=$?;
.github/add_link_to_logs.sh;
exit $EXITCODE
Expand Down Expand Up @@ -382,8 +386,8 @@ jobs:
run: EXITCODE=0;
python3
-u ${{ env.SUITE }}/regression.py
--clickhouse-binary-path ${{ env.clickhouse_binary_path }}
--attr project="$GITHUB_REPOSITORY" project.id="$GITHUB_REPOSITORY_ID" package="${{ env.clickhouse_binary_path }}" version="${{ env.version }}" user.name="$GITHUB_ACTOR" repository="https://github.com/Altinity/clickhouse-regression" commit.hash="$(git rev-parse HEAD)" job.id="$GITHUB_RUN_ID" job.url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" arch="$(uname -i)"
--clickhouse-binary-path ${{ env.clickhouse_path }}
--attr project="$GITHUB_REPOSITORY" project.id="$GITHUB_REPOSITORY_ID" package="${{ env.clickhouse_path }}" version="${{ env.version }}" user.name="$GITHUB_ACTOR" repository="https://github.com/Altinity/clickhouse-regression" commit.hash="$(git rev-parse HEAD)" job.id="$GITHUB_RUN_ID" job.url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" arch="$(uname -i)"
${{ env.args }} || EXITCODE=$?;
.github/add_link_to_logs.sh;
exit $EXITCODE
Expand Down Expand Up @@ -428,8 +432,8 @@ jobs:
run: EXITCODE=0;
python3
-u ${{ env.SUITE }}/regression.py
--clickhouse-binary-path ${{ env.clickhouse_binary_path }}
--attr project="$GITHUB_REPOSITORY" project.id="$GITHUB_REPOSITORY_ID" package="${{ env.clickhouse_binary_path }}" version="${{ env.version }}" user.name="$GITHUB_ACTOR" repository="https://github.com/Altinity/clickhouse-regression" commit.hash="$(git rev-parse HEAD)" job.id="$GITHUB_RUN_ID" job.url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" arch="$(uname -i)"
--clickhouse-binary-path ${{ env.clickhouse_path }}
--attr project="$GITHUB_REPOSITORY" project.id="$GITHUB_REPOSITORY_ID" package="${{ env.clickhouse_path }}" version="${{ env.version }}" user.name="$GITHUB_ACTOR" repository="https://github.com/Altinity/clickhouse-regression" commit.hash="$(git rev-parse HEAD)" job.id="$GITHUB_RUN_ID" job.url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" arch="$(uname -i)"
${{ env.args }} || EXITCODE=$?;
.github/add_link_to_logs.sh;
exit $EXITCODE
Expand Down Expand Up @@ -479,13 +483,13 @@ jobs:
run: EXITCODE=0;
python3
-u ${{ env.SUITE }}/regression.py
--clickhouse-binary-path ${{ env.clickhouse_binary_path }}
--clickhouse-binary-path ${{ env.clickhouse_path }}
--storage ${{ matrix.STORAGE }}
--aws-s3-bucket ${{ secrets.REGRESSION_AWS_S3_BUCKET }}
--aws-s3-region ${{ secrets.REGRESSION_AWS_S3_REGION }}
--aws-s3-key-id ${{ secrets.REGRESSION_AWS_S3_KEY_ID }}
--aws-s3-access-key ${{ secrets.REGRESSION_AWS_S3_SECRET_ACCESS_KEY }}
--attr project="$GITHUB_REPOSITORY" project.id="$GITHUB_REPOSITORY_ID" package="${{ env.clickhouse_binary_path }}" version="${{ env.version }}" user.name="$GITHUB_ACTOR" repository="https://github.com/Altinity/clickhouse-regression" commit.hash="$(git rev-parse HEAD)" job.id="$GITHUB_RUN_ID" job.url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" arch="$(uname -i)"
--attr project="$GITHUB_REPOSITORY" project.id="$GITHUB_REPOSITORY_ID" package="${{ env.clickhouse_path }}" version="${{ env.version }}" user.name="$GITHUB_ACTOR" repository="https://github.com/Altinity/clickhouse-regression" commit.hash="$(git rev-parse HEAD)" job.id="$GITHUB_RUN_ID" job.url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" arch="$(uname -i)"
${{ env.args }} || EXITCODE=$?;
.github/add_link_to_logs.sh;
exit $EXITCODE
Expand Down Expand Up @@ -535,7 +539,7 @@ jobs:
run: EXITCODE=0;
python3
-u ${{ env.SUITE }}/regression.py
--clickhouse-binary-path ${{ env.clickhouse_binary_path }}
--clickhouse-binary-path ${{ env.clickhouse_path }}
--storage ${{ matrix.STORAGE }}
--gcs-uri ${{ secrets.REGRESSION_GCS_URI }}
--gcs-key-id ${{ secrets.REGRESSION_GCS_KEY_ID }}
Expand All @@ -544,7 +548,7 @@ jobs:
--aws-s3-region ${{ secrets.REGRESSION_AWS_S3_REGION }}
--aws-s3-key-id ${{ secrets.REGRESSION_AWS_S3_KEY_ID }}
--aws-s3-access-key ${{ secrets.REGRESSION_AWS_S3_SECRET_ACCESS_KEY }}
--attr project="$GITHUB_REPOSITORY" project.id="$GITHUB_REPOSITORY_ID" package="${{ env.clickhouse_binary_path }}" version="${{ env.version }}" user.name="$GITHUB_ACTOR" repository="https://github.com/Altinity/clickhouse-regression" commit.hash="$(git rev-parse HEAD)" job.id="$GITHUB_RUN_ID" job.url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" arch="$(uname -i)"
--attr project="$GITHUB_REPOSITORY" project.id="$GITHUB_REPOSITORY_ID" package="${{ env.clickhouse_path }}" version="${{ env.version }}" user.name="$GITHUB_ACTOR" repository="https://github.com/Altinity/clickhouse-regression" commit.hash="$(git rev-parse HEAD)" job.id="$GITHUB_RUN_ID" job.url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" arch="$(uname -i)"
${{ env.args }} || EXITCODE=$?;
.github/add_link_to_logs.sh;
exit $EXITCODE
Expand All @@ -570,6 +574,7 @@ jobs:
uses: actions/checkout@v4
with:
repository: Altinity/clickhouse-regression
ref: ${{ inputs.commit }}
- name: Set envs
run: |
cat >> "$GITHUB_ENV" << 'EOF'
Expand All @@ -593,15 +598,15 @@ jobs:
run: EXITCODE=0;
python3
-u ${{ env.SUITE }}/regression.py
--clickhouse-binary-path ${{ env.clickhouse_binary_path }}
--clickhouse-binary-path ${{ env.clickhouse_path }}
--aws-s3-access-key ${{ secrets.REGRESSION_AWS_S3_SECRET_ACCESS_KEY }}
--aws-s3-key-id ${{ secrets.REGRESSION_AWS_S3_KEY_ID }}
--aws-s3-uri https://s3.${{ secrets.REGRESSION_AWS_S3_REGION}}.amazonaws.com/${{ secrets.REGRESSION_AWS_S3_BUCKET }}/data/
--gcs-key-id ${{ secrets.REGRESSION_GCS_KEY_ID }}
--gcs-key-secret ${{ secrets.REGRESSION_GCS_KEY_SECRET }}
--gcs-uri ${{ secrets.REGRESSION_GCS_URI }}
--with-${{ matrix.STORAGE }}
--attr project="$GITHUB_REPOSITORY" project.id="$GITHUB_REPOSITORY_ID" package="${{ env.clickhouse_binary_path }}" version="${{ env.version }}" user.name="$GITHUB_ACTOR" repository="https://github.com/Altinity/clickhouse-regression" commit.hash="$(git rev-parse HEAD)" job.id="$GITHUB_RUN_ID" job.url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" arch="$(uname -i)"
--attr project="$GITHUB_REPOSITORY" project.id="$GITHUB_REPOSITORY_ID" package="${{ env.clickhouse_path }}" version="${{ env.version }}" user.name="$GITHUB_ACTOR" repository="https://github.com/Altinity/clickhouse-regression" commit.hash="$(git rev-parse HEAD)" job.id="$GITHUB_RUN_ID" job.url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" arch="$(uname -i)"
${{ env.args }} || EXITCODE=$?;
.github/add_link_to_logs.sh;
exit $EXITCODE
Expand Down
Loading
Loading