Skip to content

Commit 8fe1929

Browse files
thc1006friedrichg
thc1006
and
friedrichg
authored
Add ARM64 architecture support to integration tests (#7068)
* Add ARM64 architecture support to integration tests This commit adds ARM64 runner support to the CI pipeline to ensure integration tests run on both amd64 and arm64 architectures, as ARM64 images are widely used in production. Changes: - Add matrix strategy to integration job with separate runners for amd64 (ubuntu-24.04) and arm64 (ubuntu-24.04-arm) - Dynamically set CORTEX_IMAGE based on matrix.arch variable - Add matrix strategy to integration-configs-db job for both architectures - Add appropriate timeouts to accommodate ARM64 test execution times - Set fail-fast: false to ensure all architecture tests complete All existing amd64 tests remain unchanged, and ARM64 tests use the same test suites with architecture-appropriate Docker images. Fixes #6897 Signed-off-by: thc1006 <[email protected]> * Fix install-docker.sh to support ARM64 architecture The script was hardcoded to download x86_64 Docker binaries, causing "Exec format error" on ARM64 runners. This commit adds architecture detection to download the appropriate binaries for both amd64 and arm64. Changes: - Add architecture detection using uname -m - Map system architecture to Docker download paths (x86_64/aarch64) - Map architecture to buildx binary names (amd64/arm64) - Add informative echo to show detected architecture - Add error handling for unsupported architectures This fix is required for ARM64 integration tests to run successfully. Signed-off-by: thc1006 <[email protected]> * Skip backward_compatibility, query_fuzz, and querier tests on ARM64 These tests fail on ARM64 runners and should only execute on AMD64: ## integration_backward_compatibility Old Cortex versions (v1.13.1, v1.13.2, v1.14.0) were released before ARM64 support was added in v1.14.1 and do not have ARM64 Docker images. When Docker attempts to run these amd64-only images on ARM64 runners via QEMU emulation, they crash with a fatal Go runtime error: "runtime: lfstack.push invalid packing ... fatal error: lfstack.push" This is a known issue with Go binaries and QEMU emulation (golang/go#69255). While v1.14.1+ versions do have ARM64 images, skipping the entire test on ARM64 is simpler and sufficient since backward compatibility testing validates protocol compatibility, which is architecture-agnostic. ## integration_query_fuzz This fuzzy testing suite compares query results between Cortex v1.18.1 and the current version. Although v1.18.1 has ARM64 support, the test produces inconsistent results on ARM64 (NaN value mismatches), likely due to floating-point arithmetic differences between architectures. ## integration_querier One specific subtest fails on ARM64: TestQuerierWithBlocksStorageRunningInSingleBinaryMode/ blocks_sharding_enabled,_redis_index_cache,_bucket_index_enabled,thanosEngine=true Error: "unable to find metrics [thanos_store_index_cache_requests_total] with expected values. Last values: [36]" This appears to be a timing-sensitive test where the exact number of cache requests differs between ARM64 and AMD64 runners, likely due to performance characteristics or subtle behavioral differences in the Thanos store gateway. ## Testing Coverage All other ARM64 integration tests (5 test suites) pass successfully: - requires_docker - integration_alertmanager - integration_memberlist - integration_ruler - integration_remote_write_v2 This provides comprehensive validation of core Cortex functionality on ARM64 architecture while avoiding known compatibility and timing issues with historical and edge-case testing scenarios. Fixes #6897 Signed-off-by: thc1006 <[email protected]> * chore: remove deprecated // +build tags Removed deprecated `// +build` build constraint comments from 40 files. These are no longer needed as `//go:build` directives are now used exclusively as per Go 1.17+ requirements. This fixes golangci-lint buildtag errors detected with newer linter versions on ARM64 platform. Files modified: - 37 integration test files - 3 pkg/configs/db/dbtest files Signed-off-by: thc1006 <[email protected]> * Enable all remaining integration tests on ARM64 This commit addresses reviewer feedback to maximize ARM64 test coverage by enabling the remaining integration test suites on ARM64 architecture. ## Changes ### integration_querier - Added runtime.GOARCH skip for Thanos engine subtests on non-amd64 - Allows the test suite to run on ARM64, skipping only timing-sensitive subtests that check exact cache request counts - These assertions vary across architectures due to performance differences ### integration_backward_compatibility - Removed support for Cortex v1.13.x-v1.15.x (11 versions) - Retained only v1.16.0+ (7 versions with ARM64 support) - Per https://cortexmetrics.io/docs/configuration/v1guarantees/, only the last 3 minor versions need backward compatibility testing - All retained versions have ARM64 Docker images available ### integration_query_fuzz - Added to ARM64 matrix to match amd64 test coverage - Known issue #6982 (native histogram precision) affects both architectures - Maintaining test parity across architectures per reviewer feedback ### Workflow updates - Added all three test suites to ARM64 matrix - Updated Docker image preloading to match retained versions - Added v1.19.0 to preload list ## Result ARM64 test coverage increases from 5/8 to 8/8 integration test suites (100%). All integration tests now run on both amd64 and arm64 where technically feasible. Addresses: #7068 (comment) Addresses: #7068 (comment) Addresses: #7068 (comment) Signed-off-by: thc1006 <[email protected]> --------- Signed-off-by: thc1006 <[email protected]> Signed-off-by: Friedrich Gonzalez <[email protected]> Co-authored-by: Friedrich Gonzalez <[email protected]>
1 parent 3b1ddde commit 8fe1929

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+97
-97
lines changed

‎.github/workflows/scripts/install-docker.sh‎

Lines changed: 22 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,30 @@
22

33
set -x
44
VER="28.0.4"
5-
curl -L -o /tmp/docker-$VER.tgz https://download.docker.com/linux/static/stable/x86_64/docker-$VER.tgz
5+
6+
# Detect architecture
7+
ARCH=$(uname -m)
8+
case $ARCH in
9+
x86_64)
10+
DOCKER_ARCH="x86_64"
11+
BUILDX_ARCH="amd64"
12+
;;
13+
aarch64)
14+
DOCKER_ARCH="aarch64"
15+
BUILDX_ARCH="arm64"
16+
;;
17+
*)
18+
echo "Unsupported architecture: $ARCH"
19+
exit 1
20+
;;
21+
esac
22+
23+
echo "Installing Docker $VER for architecture: $ARCH (docker: $DOCKER_ARCH, buildx: $BUILDX_ARCH)"
24+
25+
curl -L -o /tmp/docker-$VER.tgz https://download.docker.com/linux/static/stable/$DOCKER_ARCH/docker-$VER.tgz
626
tar -xz -C /tmp -f /tmp/docker-$VER.tgz
727
mkdir -vp ~/.docker/cli-plugins/
8-
curl --silent -L "https://github.com/docker/buildx/releases/download/v0.3.0/buildx-v0.3.0.linux-amd64" > ~/.docker/cli-plugins/docker-buildx
28+
curl --silent -L "https://github.com/docker/buildx/releases/download/v0.3.0/buildx-v0.3.0.linux-$BUILDX_ARCH" > ~/.docker/cli-plugins/docker-buildx
929
chmod a+x ~/.docker/cli-plugins/docker-buildx
1030
mv /tmp/docker/* /usr/bin
1131
docker run --privileged --rm tonistiigi/binfmt --install all

‎.github/workflows/test-build-deploy.yml‎

Lines changed: 66 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -152,19 +152,60 @@ jobs:
152152

153153
integration:
154154
needs: build
155-
runs-on: ubuntu-24.04
155+
runs-on: ${{ matrix.runner }}
156+
timeout-minutes: 50
156157
strategy:
157158
fail-fast: false
158159
matrix:
159-
tags:
160-
- requires_docker
161-
- integration_alertmanager
162-
- integration_backward_compatibility
163-
- integration_memberlist
164-
- integration_querier
165-
- integration_ruler
166-
- integration_query_fuzz
167-
- integration_remote_write_v2
160+
include:
161+
- runner: ubuntu-24.04
162+
arch: amd64
163+
tags: requires_docker
164+
- runner: ubuntu-24.04
165+
arch: amd64
166+
tags: integration_alertmanager
167+
- runner: ubuntu-24.04
168+
arch: amd64
169+
tags: integration_backward_compatibility
170+
- runner: ubuntu-24.04
171+
arch: amd64
172+
tags: integration_memberlist
173+
- runner: ubuntu-24.04
174+
arch: amd64
175+
tags: integration_querier
176+
- runner: ubuntu-24.04
177+
arch: amd64
178+
tags: integration_ruler
179+
- runner: ubuntu-24.04
180+
arch: amd64
181+
tags: integration_query_fuzz
182+
- runner: ubuntu-24.04
183+
arch: amd64
184+
tags: integration_remote_write_v2
185+
- runner: ubuntu-24.04-arm
186+
arch: arm64
187+
tags: requires_docker
188+
- runner: ubuntu-24.04-arm
189+
arch: arm64
190+
tags: integration_alertmanager
191+
- runner: ubuntu-24.04-arm
192+
arch: arm64
193+
tags: integration_memberlist
194+
- runner: ubuntu-24.04-arm
195+
arch: arm64
196+
tags: integration_ruler
197+
- runner: ubuntu-24.04-arm
198+
arch: arm64
199+
tags: integration_remote_write_v2
200+
- runner: ubuntu-24.04-arm
201+
arch: arm64
202+
tags: integration_query_fuzz
203+
- runner: ubuntu-24.04-arm
204+
arch: arm64
205+
tags: integration_backward_compatibility
206+
- runner: ubuntu-24.04-arm
207+
arch: arm64
208+
tags: integration_querier
168209
steps:
169210
- name: Upgrade golang
170211
uses: actions/setup-go@d35c59abb061a4a6fb18e82ac0862c26744d6ab5 # v5.5.0
@@ -214,19 +255,29 @@ jobs:
214255
env:
215256
TEST_TAGS: ${{ matrix.tags }}
216257
- name: Integration Tests
258+
timeout-minutes: 45
217259
run: |
218260
export CORTEX_IMAGE_PREFIX="${IMAGE_PREFIX:-quay.io/cortexproject/}"
219261
export IMAGE_TAG=$(make image-tag)
220-
export CORTEX_IMAGE="${CORTEX_IMAGE_PREFIX}cortex:$IMAGE_TAG-amd64"
262+
export CORTEX_IMAGE="${CORTEX_IMAGE_PREFIX}cortex:${IMAGE_TAG}-${{ matrix.arch }}"
221263
export CORTEX_CHECKOUT_DIR="/go/src/github.com/cortexproject/cortex"
222-
echo "Running integration tests with image: $CORTEX_IMAGE"
264+
echo "Running integration tests on ${{ matrix.arch }} with image: ${CORTEX_IMAGE}"
223265
go test -tags=slicelabels,integration,${{ matrix.tags }} -timeout 2400s -v -count=1 ./integration/...
224266
env:
225267
IMAGE_PREFIX: ${{ secrets.IMAGE_PREFIX }}
226268

227269
integration-configs-db:
228270
needs: build
229-
runs-on: ubuntu-24.04
271+
runs-on: ${{ matrix.runner }}
272+
timeout-minutes: 20
273+
strategy:
274+
fail-fast: false
275+
matrix:
276+
include:
277+
- runner: ubuntu-24.04
278+
arch: amd64
279+
- runner: ubuntu-24.04-arm
280+
arch: arm64
230281
steps:
231282
- name: Checkout Repo
232283
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
@@ -239,10 +290,12 @@ jobs:
239290
- name: Extract Docker Images Archive
240291
run: tar -xvf images.tar -C /
241292
- name: Run Integration Configs Tests
293+
timeout-minutes: 15
242294
# Github Actions does not support TTY in their default runners yet
243295
run: |
244296
touch build-image/.uptodate
245297
MIGRATIONS_DIR=$(pwd)/cmd/cortex/migrations
298+
echo "Running configs integration tests on ${{ matrix.arch }}"
246299
make BUILD_IMAGE=quay.io/cortexproject/build-image:master-59491e9aae TTY='' configs-integration-test
247300
248301
deploy_website:

‎integration/alertmanager_test.go‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
//go:build integration_alertmanager
2-
// +build integration_alertmanager
32

43
package integration
54

‎integration/api_endpoints_test.go‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
//go:build requires_docker
2-
// +build requires_docker
32

43
package integration
54

‎integration/asserts.go‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
//go:build integration
2-
// +build integration
32

43
package integration
54

‎integration/backward_compatibility_test.go‎

Lines changed: 2 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
//go:build integration_backward_compatibility
2-
// +build integration_backward_compatibility
32

43
package integration
54

@@ -30,49 +29,9 @@ type versionsImagesFlags struct {
3029
var (
3130
// If you change the image tag, remember to update it in the preloading done
3231
// by GitHub Actions too (see .github/workflows/test-build-deploy.yml).
32+
// Per https://cortexmetrics.io/docs/configuration/v1guarantees/#flags-config-and-minor-version-upgrades,
33+
// we only need to support backward compatibility for the last 3 minor versions.
3334
previousVersionImages = map[string]*versionsImagesFlags{
34-
"quay.io/cortexproject/cortex:v1.13.1": {
35-
flagsForOldImage: func(m map[string]string) map[string]string {
36-
m["-ingester.stream-chunks-when-using-blocks"] = "true"
37-
return m
38-
},
39-
flagsForNewImage: func(m map[string]string) map[string]string {
40-
m["-ingester.client.grpc-compression"] = "snappy"
41-
return m
42-
},
43-
},
44-
"quay.io/cortexproject/cortex:v1.13.2": {
45-
flagsForOldImage: func(m map[string]string) map[string]string {
46-
m["-ingester.stream-chunks-when-using-blocks"] = "true"
47-
return m
48-
},
49-
flagsForNewImage: func(m map[string]string) map[string]string {
50-
m["-ingester.client.grpc-compression"] = "snappy"
51-
return m
52-
},
53-
},
54-
"quay.io/cortexproject/cortex:v1.14.0": {
55-
flagsForOldImage: func(m map[string]string) map[string]string {
56-
return m
57-
},
58-
flagsForNewImage: func(m map[string]string) map[string]string {
59-
m["-ingester.client.grpc-compression"] = "snappy"
60-
return m
61-
},
62-
},
63-
"quay.io/cortexproject/cortex:v1.14.1": {
64-
flagsForOldImage: func(m map[string]string) map[string]string {
65-
return m
66-
},
67-
flagsForNewImage: func(m map[string]string) map[string]string {
68-
m["-ingester.client.grpc-compression"] = "snappy"
69-
return m
70-
},
71-
},
72-
"quay.io/cortexproject/cortex:v1.15.0": nil,
73-
"quay.io/cortexproject/cortex:v1.15.1": nil,
74-
"quay.io/cortexproject/cortex:v1.15.2": nil,
75-
"quay.io/cortexproject/cortex:v1.15.3": nil,
7635
"quay.io/cortexproject/cortex:v1.16.0": nil,
7736
"quay.io/cortexproject/cortex:v1.16.1": nil,
7837
"quay.io/cortexproject/cortex:v1.17.0": nil,

‎integration/blocks_storage_backends_test.go‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
//go:build requires_docker
2-
// +build requires_docker
32

43
package integration
54

‎integration/configs.go‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
//go:build integration
2-
// +build integration
32

43
package integration
54

‎integration/distributor_mixed_ha_samples_test.go‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
//go:build requires_docker
2-
// +build requires_docker
32

43
package integration
54

‎integration/e2e/scenario_test.go‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
//go:build requires_docker
2-
// +build requires_docker
32

43
package e2e_test
54

0 commit comments

Comments
 (0)