Skip to content

Commit 85e98b4

Browse files
Copilotwuhuizuo
andauthored
fileserver: migrate tikv/tikv artifact downloads to OCI registry (#4267)
Removes all `fileserver.pingcap.net` references from `tikv/tikv` pipelines, replacing artifact fetches with OCI-based downloads via `hub-zot.pingcap.net/mirrors/hub`. ## Changes - **`pull_unit_test.groovy` (11 files)** — Removed unused `FILE_SERVER_URL` env var across all branches (latest, release-6.1 → release-9.0-beta); the variable was declared but never consumed in these pipelines. - **`pull_integration_test.groovy` (6 files: latest, release-6.1, 6.5, 7.1, 7.5, 8.1)** — Migrated artifact preparation stage from `fetchAndExtractArtifact` to `download_pingcap_oci_artifact.sh`: ```groovy // Before environment { FILE_SERVER_URL = 'http://fileserver.pingcap.net' } // ... component.fetchAndExtractArtifact(FILE_SERVER_URL, 'pd', ...) component.fetchAndExtractArtifact(FILE_SERVER_URL, 'tikv', ...) component.fetchAndExtractArtifact(FILE_SERVER_URL, 'tidb', ...) // After final OCI_TAG_PD = component.computeArtifactOciTagFromPR('pd', REFS.base_ref, REFS.pulls[0].title, 'master') final OCI_TAG_TIKV = component.computeArtifactOciTagFromPR('tikv', REFS.base_ref, REFS.pulls[0].title, 'master') final OCI_TAG_TIDB = component.computeArtifactOciTagFromPR('tidb', REFS.base_ref, REFS.pulls[0].title, 'master') environment { OCI_ARTIFACT_HOST = 'hub-zot.pingcap.net/mirrors/hub' } // ... container('utils') { sh "${WORKSPACE}/scripts/artifacts/download_pingcap_oci_artifact.sh --pd=${OCI_TAG_PD} --tikv=${OCI_TAG_TIKV} --tidb=${OCI_TAG_TIDB}" } ``` - **`pod-pull_integration_test.yaml` (6 files)** — Added `utils` container (`ghcr.io/pingcap-qe/cd/utils/release:v2025.10.12-7-gfdd779c`) to each integration test pod spec, providing the `oras` tooling required by the download script. <!-- START COPILOT ORIGINAL PROMPT --> <details> <summary>Original prompt</summary> > > ---- > > *This section details on the original issue you should resolve* > > <issue_title>fileserver: migrate artifacts for tikv/tikv</issue_title> > <issue_description>Replace fileserver.pingcap.net usage for tikv/tikv with OCI artifacts in a standard registry. > > Acceptance criteria: > - No fileserver.pingcap.net references in jobs/pipelines for tikv/tikv > - Artifacts pulled via OCI registry with documented paths > </issue_description> > > ## Comments on the Issue (you are @copilot in this section) > > <comments> > </comments> > </details> <!-- START COPILOT CODING AGENT SUFFIX --> - Fixes #4212 <!-- START COPILOT CODING AGENT TIPS --> --- 🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. [Learn more about Advanced Security.](https://gh.io/cca-advanced-security) --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: wuhuizuo <2574558+wuhuizuo@users.noreply.github.com> Co-authored-by: wuhuizuo <wuhuizuo@126.com>
1 parent 935f578 commit 85e98b4

24 files changed

+121
-65
lines changed

pipelines/tikv/tikv/latest/pod-pull_integration_test.yaml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,16 @@ spec:
2424
limits:
2525
memory: 32Gi
2626
cpu: "16"
27+
- name: utils
28+
image: ghcr.io/pingcap-qe/cd/utils/release:v2025.10.12-7-gfdd779c
29+
tty: true
30+
resources:
31+
requests:
32+
memory: 256Mi
33+
cpu: 100m
34+
limits:
35+
cpu: "1"
36+
memory: 4Gi
2737
affinity:
2838
nodeAffinity:
2939
requiredDuringSchedulingIgnoredDuringExecution:

pipelines/tikv/tikv/latest/pull_integration_test.groovy

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,9 @@ final GIT_CREDENTIALS_ID = 'github-sre-bot-ssh'
77
final POD_TEMPLATE_FILE = 'pipelines/tikv/tikv/latest/pod-pull_integration_test.yaml'
88
final REFS = readJSON(text: params.JOB_SPEC).refs
99
prow.setPRDescription(REFS)
10+
final OCI_TAG_PD = component.computeArtifactOciTagFromPR('pd', REFS.base_ref, REFS.pulls[0].title, 'master')
11+
final OCI_TAG_TIKV = component.computeArtifactOciTagFromPR('tikv', REFS.base_ref, REFS.pulls[0].title, 'master')
12+
final OCI_TAG_TIDB = component.computeArtifactOciTagFromPR('tidb', REFS.base_ref, REFS.pulls[0].title, 'master')
1013

1114
pipeline {
1215
agent {
@@ -17,7 +20,7 @@ pipeline {
1720
}
1821
}
1922
environment {
20-
FILE_SERVER_URL = 'http://fileserver.pingcap.net'
23+
OCI_ARTIFACT_HOST = 'hub-zot.pingcap.net/mirrors/hub'
2124
}
2225
options {
2326
timeout(time: 50, unit: 'MINUTES')
@@ -45,14 +48,12 @@ pipeline {
4548
sh label: 'Prepare tikv-server', script: 'make release'
4649
}
4750
dir('bin') {
48-
script {
49-
component.fetchAndExtractArtifact(FILE_SERVER_URL, 'pd', REFS.base_ref, REFS.pulls[0].title, 'centos7/pd-server.tar.gz', 'bin')
50-
}
51-
script {
52-
component.fetchAndExtractArtifact(FILE_SERVER_URL, 'tikv', REFS.base_ref, REFS.pulls[0].title, 'centos7/tikv-server.tar.gz', 'bin')
53-
}
54-
script {
55-
component.fetchAndExtractArtifact(FILE_SERVER_URL, 'tidb', REFS.base_ref, REFS.pulls[0].title, 'centos7/tidb-server.tar.gz', 'bin')
51+
container('utils') {
52+
retry(2) {
53+
sh label: 'download components', script: """
54+
${WORKSPACE}/scripts/artifacts/download_pingcap_oci_artifact.sh --pd=${OCI_TAG_PD} --tikv=${OCI_TAG_TIKV} --tidb=${OCI_TAG_TIDB}
55+
"""
56+
}
5657
}
5758
}
5859
}

pipelines/tikv/tikv/latest/pull_unit_test.groovy

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,6 @@ pipeline {
2020
}
2121
}
2222
environment {
23-
FILE_SERVER_URL = 'http://fileserver.pingcap.net'
2423
TIKV_TEST_MEMORY_DISK_MOUNT_POINT = "/home/jenkins/agent/memvolume"
2524
}
2625
options {

pipelines/tikv/tikv/release-6.1/pod-pull_integration_test.yaml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,16 @@ spec:
2424
limits:
2525
memory: 32Gi
2626
cpu: "16"
27+
- name: utils
28+
image: ghcr.io/pingcap-qe/cd/utils/release:v2025.10.12-7-gfdd779c
29+
tty: true
30+
resources:
31+
requests:
32+
memory: 256Mi
33+
cpu: 100m
34+
limits:
35+
cpu: "1"
36+
memory: 4Gi
2737
affinity:
2838
nodeAffinity:
2939
requiredDuringSchedulingIgnoredDuringExecution:

pipelines/tikv/tikv/release-6.1/pull_integration_test.groovy

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,9 @@ final GIT_CREDENTIALS_ID = 'github-sre-bot-ssh'
77
final POD_TEMPLATE_FILE = 'pipelines/tikv/tikv/relese-6.1/pod-pull_integration_test.yaml'
88
final REFS = readJSON(text: params.JOB_SPEC).refs
99
prow.setPRDescription(REFS)
10+
final OCI_TAG_PD = component.computeArtifactOciTagFromPR('pd', REFS.base_ref, REFS.pulls[0].title, 'master')
11+
final OCI_TAG_TIKV = component.computeArtifactOciTagFromPR('tikv', REFS.base_ref, REFS.pulls[0].title, 'master')
12+
final OCI_TAG_TIDB = component.computeArtifactOciTagFromPR('tidb', REFS.base_ref, REFS.pulls[0].title, 'master')
1013

1114
pipeline {
1215
agent {
@@ -17,7 +20,7 @@ pipeline {
1720
}
1821
}
1922
environment {
20-
FILE_SERVER_URL = 'http://fileserver.pingcap.net'
23+
OCI_ARTIFACT_HOST = 'hub-zot.pingcap.net/mirrors/hub'
2124
}
2225
options {
2326
timeout(time: 50, unit: 'MINUTES')
@@ -45,14 +48,12 @@ pipeline {
4548
sh label: 'Prepare tikv-server', script: 'make release'
4649
}
4750
dir('bin') {
48-
script {
49-
component.fetchAndExtractArtifact(FILE_SERVER_URL, 'pd', REFS.base_ref, REFS.pulls[0].title, 'centos7/pd-server.tar.gz', 'bin')
50-
}
51-
script {
52-
component.fetchAndExtractArtifact(FILE_SERVER_URL, 'tikv', REFS.base_ref, REFS.pulls[0].title, 'centos7/tikv-server.tar.gz', 'bin')
53-
}
54-
script {
55-
component.fetchAndExtractArtifact(FILE_SERVER_URL, 'tidb', REFS.base_ref, REFS.pulls[0].title, 'centos7/tidb-server.tar.gz', 'bin')
51+
container('utils') {
52+
retry(2) {
53+
sh label: 'download components', script: """
54+
${WORKSPACE}/scripts/artifacts/download_pingcap_oci_artifact.sh --pd=${OCI_TAG_PD} --tikv=${OCI_TAG_TIKV} --tidb=${OCI_TAG_TIDB}
55+
"""
56+
}
5657
}
5758
}
5859
}

pipelines/tikv/tikv/release-6.1/pull_unit_test.groovy

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,6 @@ pipeline {
2121
}
2222
}
2323
environment {
24-
FILE_SERVER_URL = 'http://fileserver.pingcap.net'
2524
TIKV_TEST_MEMORY_DISK_MOUNT_POINT = "/home/jenkins/agent/memvolume"
2625
}
2726
options {

pipelines/tikv/tikv/release-6.5/pod-pull_integration_test.yaml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,16 @@ spec:
2424
limits:
2525
memory: 32Gi
2626
cpu: "16"
27+
- name: utils
28+
image: ghcr.io/pingcap-qe/cd/utils/release:v2025.10.12-7-gfdd779c
29+
tty: true
30+
resources:
31+
requests:
32+
memory: 256Mi
33+
cpu: 100m
34+
limits:
35+
cpu: "1"
36+
memory: 4Gi
2737
affinity:
2838
nodeAffinity:
2939
requiredDuringSchedulingIgnoredDuringExecution:

pipelines/tikv/tikv/release-6.5/pull_integration_test.groovy

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,9 @@ final GIT_CREDENTIALS_ID = 'github-sre-bot-ssh'
77
final POD_TEMPLATE_FILE = 'pipelines/tikv/tikv/release-6.5/pod-pull_integration_test.yaml'
88
final REFS = readJSON(text: params.JOB_SPEC).refs
99
prow.setPRDescription(REFS)
10+
final OCI_TAG_PD = component.computeArtifactOciTagFromPR('pd', REFS.base_ref, REFS.pulls[0].title, 'master')
11+
final OCI_TAG_TIKV = component.computeArtifactOciTagFromPR('tikv', REFS.base_ref, REFS.pulls[0].title, 'master')
12+
final OCI_TAG_TIDB = component.computeArtifactOciTagFromPR('tidb', REFS.base_ref, REFS.pulls[0].title, 'master')
1013

1114
pipeline {
1215
agent {
@@ -17,7 +20,7 @@ pipeline {
1720
}
1821
}
1922
environment {
20-
FILE_SERVER_URL = 'http://fileserver.pingcap.net'
23+
OCI_ARTIFACT_HOST = 'hub-zot.pingcap.net/mirrors/hub'
2124
}
2225
options {
2326
timeout(time: 50, unit: 'MINUTES')
@@ -45,14 +48,12 @@ pipeline {
4548
sh label: 'Prepare tikv-server', script: 'make release'
4649
}
4750
dir('bin') {
48-
script {
49-
component.fetchAndExtractArtifact(FILE_SERVER_URL, 'pd', REFS.base_ref, REFS.pulls[0].title, 'centos7/pd-server.tar.gz', 'bin')
50-
}
51-
script {
52-
component.fetchAndExtractArtifact(FILE_SERVER_URL, 'tikv', REFS.base_ref, REFS.pulls[0].title, 'centos7/tikv-server.tar.gz', 'bin')
53-
}
54-
script {
55-
component.fetchAndExtractArtifact(FILE_SERVER_URL, 'tidb', REFS.base_ref, REFS.pulls[0].title, 'centos7/tidb-server.tar.gz', 'bin')
51+
container('utils') {
52+
retry(2) {
53+
sh label: 'download components', script: """
54+
${WORKSPACE}/scripts/artifacts/download_pingcap_oci_artifact.sh --pd=${OCI_TAG_PD} --tikv=${OCI_TAG_TIKV} --tidb=${OCI_TAG_TIDB}
55+
"""
56+
}
5657
}
5758
}
5859
}

pipelines/tikv/tikv/release-6.5/pull_unit_test.groovy

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,6 @@ pipeline {
2121
}
2222
}
2323
environment {
24-
FILE_SERVER_URL = 'http://fileserver.pingcap.net'
2524
TIKV_TEST_MEMORY_DISK_MOUNT_POINT = "/home/jenkins/agent/memvolume"
2625
}
2726
options {

pipelines/tikv/tikv/release-7.1/pod-pull_integration_test.yaml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,16 @@ spec:
2424
limits:
2525
memory: 32Gi
2626
cpu: "16"
27+
- name: utils
28+
image: ghcr.io/pingcap-qe/cd/utils/release:v2025.10.12-7-gfdd779c
29+
tty: true
30+
resources:
31+
requests:
32+
memory: 256Mi
33+
cpu: 100m
34+
limits:
35+
cpu: "1"
36+
memory: 4Gi
2737
affinity:
2838
nodeAffinity:
2939
requiredDuringSchedulingIgnoredDuringExecution:

0 commit comments

Comments
 (0)