Skip to content

Commit 909cd31

Browse files
nunnikrinunnikri
authored
Adding new workflow to create deb/rpm native packages (#1528)
## Motivation New workflow to create native packages. The current ROCK builds create tar balls. The new workflow will fetch those tar balls and create debian or rpm linux native packages. ## Technical Details We have a scripts/tools to convert the artifacts to native debian and rpm package. https://github.com/ROCm/TheRock/tree/main/build_tools/packaging/linux This workflow is for using the script and generate packages for each build. This is the initial version. ## Test Plan Use different ROCK build id's and test whether packages are getting created for all the modules and artifacts ## Test Result Script verification done ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.
1 parent 4195856 commit 909cd31

File tree

3 files changed

+359
-0
lines changed

3 files changed

+359
-0
lines changed

‎.github/workflows/build_native_linux_packages.yml‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
name: Build Native Linux Packages
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
amdgpu_family:
7+
description: gfx arch for creating the artifact fetch url
8+
required: true
9+
type: string
10+
artifact_run_id:
11+
description: workflow run id to download the artifacts from
12+
type: string
13+
rocm_version:
14+
description: ROCm version to append to the package( Like 8.0.0 or 8.1.0).
15+
required: true
16+
type: string
17+
package_type:
18+
description: Specify whether debian or rpm packages are needed (deb or rpm).
19+
required: true
20+
type: string
21+
package_suffix:
22+
description: The suffix to be added to package name(build_no or master or rc or combiantion).
23+
required: false
24+
type: string
25+
workflow_dispatch:
26+
inputs:
27+
amdgpu_family:
28+
type: string
29+
default: gfx94X-dcgpu
30+
artifact_run_id:
31+
description: workflow run id to download the artifacts from
32+
type: string
33+
rocm_version:
34+
description: ROCm version to append to the package( Like 8.0.0 or 8.1.0)
35+
type: string
36+
default: "0.0.1"
37+
package_type:
38+
description: Specify whether debian or rpm packages are needed (deb or rpm).
39+
required: true
40+
type: choice
41+
options:
42+
- rpm
43+
- deb
44+
default: "rpm"
45+
package_suffix:
46+
description: The suffix to be added to package name(build_no or master or rc or combiantion).
47+
type: false
48+
default: "test"
49+
50+
permissions:
51+
id-token: write
52+
contents: read
53+
54+
jobs:
55+
build_native_packages:
56+
name: Build Linux native Packages
57+
strategy:
58+
fail-fast: false
59+
runs-on: ${{ github.repository_owner == 'ROCm' && 'azure-linux-scale-rocm' || 'ubuntu-24.04' }}
60+
env:
61+
BUILD_IMAGE: ghcr.io/rocm/therock_build_manylinux_x86_64@sha256:3ac188c17f88f08ce522297b616d8308361b8e9a9b31bcc3c3bbb9429b1efa6c
62+
ARTIFACT_RUN_ID: ${{ inputs.artifact_run_id || github.run_id }}
63+
PACKAGE_SUFFIX: ${{ (inputs.package_suffix || github.event.inputs.package_suffix || 'nightly') }}-${{ inputs.artifact_run_id || github.run_id }}
64+
OUTPUT_DIR: ${{ github.workspace }}/output
65+
ARTIFACTS_DIR: ${{ github.workspace }}/output/artifacts
66+
PACKAGE_DIST_DIR: ${{ github.workspace }}/output/packages
67+
steps:
68+
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
69+
- uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6.0.0
70+
with:
71+
python-version: '3.12'
72+
- name: Install Python requirements
73+
run: |
74+
pip install pyelftools boto3 jinja2
75+
76+
- name: Install System requirements
77+
run: |
78+
# Install the needed tools for creating rpm / deb packages
79+
# Also install tools for creating repo files
80+
sudo apt update
81+
sudo apt install -y llvm
82+
sudo apt install -y rpm debhelper-compat build-essential
83+
sudo apt install -y dpkg-dev createrepo-c
84+
85+
- name: Fetch Artifacts
86+
run: |
87+
echo "Fetching artifacts for build ${{ inputs.artifact_run_id }}"
88+
python ./build_tools/fetch_artifacts.py \
89+
--run-id=${{ env.ARTIFACT_RUN_ID }} \
90+
--run-github-repo="ROCm/TheRock" \
91+
--target=${{ inputs.amdgpu_family }} \
92+
--output-dir=${{ env.ARTIFACTS_DIR }}
93+
94+
- name: Build Packages
95+
id: build-packages
96+
run: |
97+
echo "Building ${{ inputs.package_type }} packages for ${{ inputs.amdgpu_family }} ${{ inputs.artifact_run_id }}"
98+
python ./build_tools/packaging/linux/build_package.py \
99+
--dest-dir ${{ env.PACKAGE_DIST_DIR }} \
100+
--rocm-version ${{ inputs.rocm_version }} \
101+
--target ${{ inputs.amdgpu_family }} \
102+
--artifacts-dir ${{ env.ARTIFACTS_DIR }} \
103+
--pkg-type ${{ inputs.package_type }} \
104+
--version-suffix ${{ env.PACKAGE_SUFFIX }}
105+
106+
- name: Install AWS CLI
107+
run: bash ./dockerfiles/install_awscli.sh
108+
109+
- name: Configure AWS Credentials for non-forked repos
110+
uses: aws-actions/configure-aws-credentials@7474bc4690e29a8392af63c5b98e7449536d5c3a # v4.3.1
111+
with:
112+
aws-region: us-east-2
113+
role-to-assume: arn:aws:iam::692859939525:role/therock-artifacts-external
114+
115+
- name: Upload Package repo to S3
116+
id: upload-packages
117+
run: |
118+
echo "Uploading to s3 bucket"
119+
python ./build_tools/packaging/linux/upload_package_repo.py \
120+
--pkg-type ${{ inputs.package_type }} \
121+
--s3-bucket therock-deb-rpm-test \
122+
--amdgpu-family ${{ inputs.amdgpu_family }} \
123+
--artifact-id ${{ env.ARTIFACT_RUN_ID }}

‎.github/workflows/release_native_linux_packages.yml‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
name: Release native Linux Packages
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
amdgpu_family:
7+
description: gfx arch for creating the s3 bucket url
8+
required: true
9+
type: string
10+
artifact_run_id:
11+
description: workflow run id to download the artifacts from
12+
type: string
13+
rocm_version:
14+
description: ROCm version to append to the package( Like 8.0.0 or 8.1.0).
15+
required: true
16+
type: string
17+
package_type:
18+
description: Specify whether debian or rpm packages are needed (deb or rpm).
19+
required: true
20+
type: string
21+
package_suffix:
22+
description: The suffix to be added to package name(build_no or master or rc or combiantion).
23+
required: true
24+
type: string
25+
workflow_dispatch:
26+
inputs:
27+
amdgpu_family:
28+
type: string
29+
default: gfx94X-dcgpu
30+
artifact_run_id:
31+
description: workflow run id to download the artifacts from
32+
type: string
33+
rocm_version:
34+
description: ROCm version to append to the package( Like 7.0.0 or 7.1.0)
35+
type: string
36+
default: "0.0.1"
37+
package_type:
38+
description: Specify whether debian or rpm packages are needed (deb or rpm).
39+
required: true
40+
type: choice
41+
options:
42+
- rpm
43+
- deb
44+
default: "rpm"
45+
package_suffix:
46+
description: The suffix to be added to package name(build_no or master or rc or combiantion).
47+
type: string
48+
default: "test"
49+
50+
permissions:
51+
id-token: write
52+
contents: read
53+
54+
jobs:
55+
release:
56+
name: Release Native Linux Package
57+
strategy:
58+
fail-fast: false
59+
60+
uses: ./.github/workflows/build_native_linux_packages.yml
61+
with:
62+
amdgpu_family: ${{ inputs.amdgpu_family }}
63+
artifact_run_id: ${{ inputs.artifact_run_id }}
64+
rocm_version: ${{ inputs.rocm_version }}
65+
package_type: ${{ inputs.package_type }}
66+
package_suffix: ${{ inputs.package_suffix }}

‎build_tools/packaging/linux/upload_package_repo.py‎

Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
#!/usr/bin/env python3
2+
3+
# Copyright Advanced Micro Devices, Inc.
4+
# SPDX-License-Identifier: MIT
5+
6+
7+
"""Given ROCm artifacts directories, performs packaging to
8+
create RPM and DEB packages and stores to OUTPUT folder
9+
10+
```
11+
python ./build_tools/packaging/linux/upload_package_repo.py \
12+
--pkg-type deb \
13+
--s3-bucket therock-deb-rpm-test \
14+
--amdgpu-family gfx94X-dcgpu \
15+
--artifact-id 16418185899
16+
```
17+
"""
18+
19+
import os
20+
import argparse
21+
import subprocess
22+
import boto3
23+
import shutil
24+
import datetime
25+
26+
27+
def run_command(cmd, cwd=None):
28+
"""
29+
Function to execute commands in shell.
30+
"""
31+
print(f"Running: {cmd}")
32+
subprocess.run(cmd, shell=True, check=True, cwd=cwd)
33+
34+
35+
def find_package_dir():
36+
"""
37+
Finds the default output dir for packages.
38+
Expects packages in ./output/packages
39+
"""
40+
base_dir = os.path.join(os.getcwd(), "output", "packages")
41+
if not os.path.exists(base_dir):
42+
raise RuntimeError(f"Package directory not found: {base_dir}")
43+
print(f"Using package directory: {base_dir}")
44+
return base_dir
45+
46+
47+
def create_deb_repo(package_dir, origin_name):
48+
"""Function to create rpm repo
49+
It takes all the rpm files in the package_dir parameter
50+
And creates the deb package list using dpkg-scanpackages
51+
Package list is gzipped Packages.gz to pool/main foldre
52+
Also create Release meta package file needed for debian repo
53+
54+
Parameters:
55+
package_dir : Folder to search for deb packages
56+
origin_name : S3 bucket to upload, used in meta data creation
57+
58+
Returns: None
59+
"""
60+
print("Creating APT repository...")
61+
dists_dir = os.path.join(package_dir, "dists", "stable", "main", "binary-amd64")
62+
release_dir = os.path.join(package_dir, "dists", "stable")
63+
pool_dir = os.path.join(package_dir, "pool", "main")
64+
65+
os.makedirs(dists_dir, exist_ok=True)
66+
os.makedirs(pool_dir, exist_ok=True)
67+
for file in os.listdir(package_dir):
68+
if file.endswith(".deb"):
69+
shutil.move(os.path.join(package_dir, file), os.path.join(pool_dir, file))
70+
71+
print(
72+
"Generating Packages file at repository root so 'Filename' paths are 'pool/...'."
73+
)
74+
cmd = "dpkg-scanpackages -m pool/main /dev/null > dists/stable/main/binary-amd64/Packages"
75+
run_command(cmd, cwd=package_dir)
76+
run_command("gzip -9c Packages > Packages.gz", cwd=dists_dir)
77+
78+
print("Creating Release file...")
79+
release_content = f"""\
80+
Origin: {origin_name}
81+
Label: {origin_name}
82+
Suite: stable
83+
Codename: stable
84+
Version: 1.0
85+
Date: {datetime.datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S UTC')}
86+
Architectures: amd64
87+
Components: main
88+
Description: ROCm Repository
89+
"""
90+
os.makedirs(release_dir, exist_ok=True)
91+
release_path = os.path.join(release_dir, "Release")
92+
with open(release_path, "w") as f:
93+
f.write(release_content)
94+
print(f"Wrote Release file to {release_path}")
95+
96+
97+
def create_rpm_repo(package_dir):
98+
"""Function to create rpm repo
99+
It takes all the rpm files in the package_dir parameter
100+
And creates the rpm repo using createrepo_c command inside x86_64 folder
101+
102+
Parameters:
103+
package_dir : Folder to search for rpm packages
104+
105+
Returns: None
106+
"""
107+
print("Creating YUM/DNF repository...")
108+
109+
arch_dir = os.path.join(package_dir, "x86_64")
110+
os.makedirs(arch_dir, exist_ok=True)
111+
for file in os.listdir(package_dir):
112+
if file.endswith(".rpm"):
113+
shutil.move(os.path.join(package_dir, file), os.path.join(arch_dir, file))
114+
run_command("createrepo_c .", cwd=arch_dir)
115+
print(f"Generated repodata/ in {arch_dir}")
116+
117+
118+
def upload_to_s3(source_dir, bucket, prefix):
119+
"""Function to upload the packges and repo files to the s3 bucket
120+
It upload the source_dir contents to s3://{bucket}/{prefix}/
121+
122+
Parameters:
123+
source_dir : Folder with the packages and repo files
124+
bucket : S3 bucket
125+
prefix : S3 prefix
126+
127+
Returns: None
128+
"""
129+
s3 = boto3.client("s3")
130+
print(f"Uploading to s3://{bucket}/{prefix}/")
131+
132+
for root, _, files in os.walk(source_dir):
133+
for filename in files:
134+
local_path = os.path.join(root, filename)
135+
rel_path = os.path.relpath(local_path, source_dir)
136+
s3_key = os.path.join(prefix, rel_path).replace("\\", "/")
137+
print(f"Uploading: {local_path} → s3://{bucket}/{s3_key}")
138+
s3.upload_file(local_path, bucket, s3_key)
139+
140+
141+
def main():
142+
parser = argparse.ArgumentParser()
143+
parser.add_argument(
144+
"--pkg-type",
145+
required=True,
146+
choices=["deb", "rpm"],
147+
help="Type of packages to process",
148+
)
149+
parser.add_argument("--s3-bucket", required=True, help="Target S3 bucket name")
150+
parser.add_argument(
151+
"--amdgpu-family", required=True, help="AMDGPU family identifier (e.g., gfx94X)"
152+
)
153+
parser.add_argument(
154+
"--artifact-id", required=True, help="Unique artifact ID or version tag"
155+
)
156+
args = parser.parse_args()
157+
158+
package_dir = find_package_dir()
159+
s3_prefix = f"{args.amdgpu_family}_{args.artifact_id}/{args.pkg_type}"
160+
161+
if args.pkg_type == "deb":
162+
create_deb_repo(package_dir, args.s3_bucket)
163+
else:
164+
create_rpm_repo(package_dir)
165+
166+
upload_to_s3(package_dir, args.s3_bucket, s3_prefix)
167+
168+
169+
if __name__ == "__main__":
170+
main()

0 commit comments

Comments
 (0)