Skip to content

Commit bbf90f3

Browse files
aktechjaimergp
aktech
and
jaimergp
authored
Automate image build process (#74)
* Automate image build process * run it in a temp directory * add concurrency * cleanup image directory and log image details * upload images to openstack as well * upload image to openstack from server itself * fix image name * remvoe unnecessary secrets * add qemu image options and fix openstack creds * add progress to qemu command * remove unsupported args * reduce repetition * cleanup github workflow and use makefile * Add job summary * use uv instead * install uv and use that * fxi uv path * fix uplaod * update readme * addd image tags * cleanups * Apply suggestions from code review Co-authored-by: jaimergp <[email protected]> --------- Co-authored-by: jaimergp <[email protected]>
1 parent 3d3780e commit bbf90f3

File tree

10 files changed

+1674
-108
lines changed

10 files changed

+1674
-108
lines changed

‎.github/workflows/build-images-ssh.yml‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
name: Build VM Images (SSH)
2+
3+
on:
4+
workflow_dispatch:
5+
6+
concurrency:
7+
group: build-vm-images
8+
cancel-in-progress: false
9+
10+
jobs:
11+
build-images:
12+
name: Build ${{ matrix.image-type }} Image
13+
runs-on: ubuntu-latest
14+
15+
strategy:
16+
matrix:
17+
image-type: [cpu, gpu]
18+
max-parallel: 1
19+
fail-fast: false
20+
21+
steps:
22+
- name: Checkout repository
23+
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
24+
25+
- name: Setup SSH
26+
run: |
27+
mkdir -p ~/.ssh
28+
echo "${{ secrets.SSH_PRIVATE_KEY }}" > ~/.ssh/id_rsa
29+
chmod 600 ~/.ssh/id_rsa
30+
ssh-keyscan -p ${{ secrets.SSH_PORT }} -H ${{ secrets.SSH_HOST }} >> ~/.ssh/known_hosts
31+
cat >> ~/.ssh/config << EOF
32+
Host gpuserver
33+
HostName ${{ secrets.SSH_HOST }}
34+
Port ${{ secrets.SSH_PORT }}
35+
User ${{ secrets.SSH_USER }}
36+
IdentityFile ~/.ssh/id_rsa
37+
StrictHostKeyChecking no
38+
EOF
39+
chmod 600 ~/.ssh/config
40+
41+
- name: Sync repository to server
42+
run: |
43+
ssh gpuserver "mkdir -p /tmp/repo-${{ github.run_id }}"
44+
scp -r ./* gpuserver:/tmp/repo-${{ github.run_id }}/
45+
46+
- name: Install uv on server
47+
run: |
48+
ssh gpuserver "mkdir -p /tmp/uv-${{ github.run_id }} && curl -LsSf https://astral.sh/uv/install.sh | env INSTALLER_NO_MODIFY_PATH=1 sh && mv ~/.local/bin/uv ~/.local/bin/uvx /tmp/uv-${{ github.run_id }}/"
49+
50+
- name: Build ${{ matrix.image-type }} image on remote server
51+
id: build
52+
run: |
53+
ssh -t gpuserver "cd /tmp/repo-${{ github.run_id }} && make build-${{ matrix.image-type }} BUILD_DIR=/tmp/vm-images-build-${{ github.run_id }} UV=/tmp/uv-${{ github.run_id }}/uv"
54+
IMAGE_NAME=$(ssh gpuserver "cd /tmp/vm-images-build-${{ github.run_id }}-${{ matrix.image-type }} && ls -1 *.qcow2 2>/dev/null | head -1 | sed 's/.qcow2$//'")
55+
echo "image_name=$IMAGE_NAME" >> $GITHUB_OUTPUT
56+
57+
- name: Upload image to OpenStack from server
58+
id: upload
59+
if: success() && github.ref == 'refs/heads/main'
60+
run: |
61+
IMAGE_TAGS="--tag name=${{ steps.build.outputs.image_name }} --tag environment=ci --tag project=conda-forge-gpu-ci --tag image-type=${{ matrix.image-type }} --tag build-job-url=${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} --tag commit-hash=${{ github.sha }} --tag branch=${{ github.ref_name }} --tag build-timestamp=$(date -u +%Y%m%d-%H%M%S)"
62+
ssh gpuserver "cd /tmp/repo-${{ github.run_id }} && make upload-${{ matrix.image-type }} BUILD_DIR=/tmp/vm-images-build-${{ github.run_id }} UV=/tmp/uv-${{ github.run_id }}/uv IMAGE_TAGS='$IMAGE_TAGS'"
63+
64+
- name: Cleanup temp directories on server
65+
if: always()
66+
run: |
67+
ssh gpuserver "cd /tmp/repo-${{ github.run_id }} && make clean BUILD_DIR=/tmp/vm-images-build-${{ github.run_id }} && cd / && rm -rf /tmp/repo-${{ github.run_id }} /tmp/uv-${{ github.run_id }}" || true
68+
69+
- name: Add job summary
70+
if: always()
71+
run: |
72+
echo "## 🖼️ ${{ matrix.image-type }} Image Build Summary" >> $GITHUB_STEP_SUMMARY
73+
echo "" >> $GITHUB_STEP_SUMMARY
74+
if [ "${{ steps.build.outcome }}" == "success" ]; then
75+
echo "✅ **Build Status:** Success" >> $GITHUB_STEP_SUMMARY
76+
echo "📦 **Image Name:** \`${{ steps.build.outputs.image_name }}\`" >> $GITHUB_STEP_SUMMARY
77+
echo "" >> $GITHUB_STEP_SUMMARY
78+
if [ "${{ steps.upload.outcome }}" == "success" ]; then
79+
echo "☁️ **OpenStack Upload:** ✅ Success" >> $GITHUB_STEP_SUMMARY
80+
elif [ "${{ steps.upload.outcome }}" == "failure" ]; then
81+
echo "☁️ **OpenStack Upload:** ❌ Failed" >> $GITHUB_STEP_SUMMARY
82+
elif [ "${{ steps.upload.outcome }}" == "skipped" ]; then
83+
echo "☁️ **OpenStack Upload:** ⏭️ Skipped (not main branch)" >> $GITHUB_STEP_SUMMARY
84+
fi
85+
else
86+
echo "❌ **Build Status:** Failed" >> $GITHUB_STEP_SUMMARY
87+
fi
88+
echo "" >> $GITHUB_STEP_SUMMARY

‎.github/workflows/openstack.yml‎

Lines changed: 0 additions & 74 deletions
This file was deleted.

‎.gitignore‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# Build artifacts
2+
vm-images/dib.log
3+
vm-images/build-output.log
4+
vm-images/*.qcow2
5+
vm-images/*.d/
6+
7+
# UV environment
8+
vm-images/.venv/
9+
vm_images.egg-info/

‎Makefile‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
.PHONY: help build-cpu build-gpu build-all upload-cpu upload-gpu upload-all clean show-image
2+
3+
# Default variables
4+
BUILD_DIR ?= /tmp/vm-images-build-$(shell date +%s)
5+
IMAGE_TYPE ?= cpu
6+
TIMESTAMP := $(shell date +%Y%m%d-%H%M%S)
7+
IMAGE_NAME := ubuntu-2404-$(IMAGE_TYPE)-$(TIMESTAMP)
8+
9+
# Path configuration
10+
REPO_DIR := $(shell pwd)
11+
VM_IMAGES_DIR := $(REPO_DIR)/vm-images
12+
OPENSTACK_RC ?= /etc/kolla/admin-openrc.sh
13+
UV ?= uv
14+
IMAGE_TAGS ?=
15+
16+
help:
17+
@echo "VM Image Build Makefile"
18+
@echo ""
19+
@echo "Requirements:"
20+
@echo " - uv (https://docs.astral.sh/uv/)"
21+
@echo ""
22+
@echo "Targets:"
23+
@echo " build-cpu - Build CPU image"
24+
@echo " build-gpu - Build GPU image"
25+
@echo " build-all - Build both CPU and GPU images"
26+
@echo " upload-cpu - Upload CPU image to OpenStack"
27+
@echo " upload-gpu - Upload GPU image to OpenStack"
28+
@echo " upload-all - Upload both images to OpenStack"
29+
@echo " show-image - Show built image information"
30+
@echo " clean - Clean up build directory"
31+
@echo ""
32+
@echo "Variables:"
33+
@echo " BUILD_DIR=$(BUILD_DIR)"
34+
@echo " IMAGE_TYPE=$(IMAGE_TYPE)"
35+
@echo " IMAGE_NAME=$(IMAGE_NAME)"
36+
@echo " UV=$(UV)"
37+
@echo " IMAGE_TAGS=$(IMAGE_TAGS) # OpenStack image tags (e.g., '--tag key=value --tag key2=value2')"
38+
39+
build-cpu:
40+
@$(MAKE) _build IMAGE_TYPE=cpu
41+
42+
build-gpu:
43+
@$(MAKE) _build IMAGE_TYPE=gpu
44+
45+
build-all:
46+
@$(MAKE) build-cpu
47+
@$(MAKE) build-gpu
48+
49+
_build:
50+
@echo "======================================"
51+
@echo "Building $(IMAGE_TYPE) image: $(IMAGE_NAME)"
52+
@echo "======================================"
53+
mkdir -p $(BUILD_DIR)-$(IMAGE_TYPE)
54+
cp -r $(VM_IMAGES_DIR)/* $(BUILD_DIR)-$(IMAGE_TYPE)/
55+
cd $(BUILD_DIR)-$(IMAGE_TYPE) && \
56+
sed -i 's/imagename: $(IMAGE_TYPE)-image.qcow2/imagename: $(IMAGE_NAME).qcow2/' $(IMAGE_TYPE)-image.yaml
57+
cd $(BUILD_DIR)-$(IMAGE_TYPE) && \
58+
$(UV) sync && \
59+
IMAGE_YAML=$(IMAGE_TYPE)-image.yaml UV=$(UV) $(UV) run bash build-image.sh
60+
@echo ""
61+
@$(MAKE) show-image IMAGE_TYPE=$(IMAGE_TYPE) BUILD_DIR=$(BUILD_DIR)
62+
63+
show-image:
64+
@echo "======================================"
65+
@echo "Built Image Information"
66+
@echo "======================================"
67+
@cd $(BUILD_DIR)-$(IMAGE_TYPE) && \
68+
IMAGE_FILE=$$(ls -1 *.qcow2 2>/dev/null | head -1) && \
69+
if [ -n "$$IMAGE_FILE" ]; then \
70+
echo "Image Path: $(BUILD_DIR)-$(IMAGE_TYPE)/$$IMAGE_FILE"; \
71+
echo "Image Size: $$(ls -lh $$IMAGE_FILE | awk '{print $$5}')"; \
72+
echo "Full Details:"; \
73+
ls -lh $$IMAGE_FILE; \
74+
echo ""; \
75+
if command -v qemu-img &> /dev/null; then \
76+
echo "QEMU Image Info:"; \
77+
qemu-img info $$IMAGE_FILE; \
78+
fi; \
79+
else \
80+
echo "No .qcow2 image file found!"; \
81+
fi
82+
@echo "======================================"
83+
84+
upload-cpu:
85+
@$(MAKE) _upload IMAGE_TYPE=cpu
86+
87+
upload-gpu:
88+
@$(MAKE) _upload IMAGE_TYPE=gpu
89+
90+
upload-all:
91+
@$(MAKE) upload-cpu
92+
@$(MAKE) upload-gpu
93+
94+
_upload:
95+
@echo "======================================"
96+
@echo "Uploading $(IMAGE_TYPE) image to OpenStack"
97+
@echo "======================================"
98+
@cd $(BUILD_DIR)-$(IMAGE_TYPE) && \
99+
. $(OPENSTACK_RC) && \
100+
IMAGE_FILE=$$(ls -1 *.qcow2 2>/dev/null | head -1) && \
101+
IMAGE_BASE=$$(basename $$IMAGE_FILE .qcow2) && \
102+
echo "Uploading $$IMAGE_FILE..." && \
103+
$(UV) run openstack image create $$IMAGE_BASE \
104+
--public --disk-format qcow2 \
105+
--container-format bare \
106+
--file $$IMAGE_FILE \
107+
$(IMAGE_TAGS) && \
108+
echo "Image uploaded successfully!" && \
109+
$(UV) run openstack image show $$IMAGE_BASE
110+
111+
clean:
112+
@echo "Cleaning up build directories..."
113+
@if [ -d "$(BUILD_DIR)-cpu" ]; then sudo rm -rf $(BUILD_DIR)-cpu; fi
114+
@if [ -d "$(BUILD_DIR)-gpu" ]; then sudo rm -rf $(BUILD_DIR)-gpu; fi
115+
@echo "Cleanup complete"

‎vm-images/README.md‎

Lines changed: 64 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,78 @@
1-
# VM Images for OpenStack
1+
# VM Images
22

3-
Creates Ubuntu VM Images with following things installed:
3+
Builds Ubuntu 24.04 images for OpenStack with Docker and NVIDIA drivers (GPU only).
44

5-
- Nvidia drivers (GPU images only)
6-
- Docker
5+
## Quick Start
6+
7+
From the repo root, run:
78

8-
The created image is uploaded to a GCS bucket, which is then retrieved by a self-hosted GHA runner running on our OpenStack instance.
9+
```bash
10+
# Build CPU image (can run anywhere)
11+
make build-cpu
912

10-
## GHA workflows
13+
# Build GPU image (MUST run on GPU server)
14+
make build-gpu
1115

12-
These should automate the creation of the images:
16+
# Upload to OpenStack
17+
make upload-cpu
18+
make upload-gpu
19+
```
1320

14-
- `.github/workflows/build-vm-images.yml` - Github Action to build the image
15-
- `.github/workflows/openstack.yml` - Github Action to upload the VM image to OpenStack
21+
## Requirements
1622

17-
## Building manually on Linux
23+
- `uv` package manager ([install here](https://docs.astral.sh/uv/))
24+
- Linux (builds won't work on Mac)
25+
- Root access (diskimage-builder needs it)
26+
- OpenStack credentials from `/etc/kolla/admin-openrc.sh` on GPU server (for uploads only)
1827

19-
1. Install `diskimage-builder` manually via `pip install -r requirements.txt`. A virtual environment is advised. Check if you have all the system dependencies in `ubuntu24-system-requirements.txt`)
20-
2. Run `scripts/build-image.sh` to build the image. This will create a `.qcow2` file.
21-
- Export `$IMAGE_YAML` to choose a different image to build (e.g. `cpu-image.yaml`, `gpu-image.yaml`).
22-
- Export `$OUTPUT_IMAGE` to change the qcow2 output filename.
28+
## Build Directory
2329

24-
## Add Image to OpenStack
30+
By default images build to `/tmp/vm-images-build-{timestamp}`. Override with:
2531

2632
```bash
27-
openstack image create ubuntu-2404-nvidia-docker \
33+
make build-cpu BUILD_DIR=/path/to/build
34+
```
35+
36+
## Custom UV Path
37+
38+
If uv isn't in your PATH:
39+
40+
```bash
41+
make build-cpu UV=/path/to/uv
42+
```
43+
44+
## Automated Builds
45+
46+
GitHub Actions workflow (`.github/workflows/build-images-ssh.yml`) can be triggered manually via workflow_dispatch.
47+
48+
The workflow SSHs into the GPU server to run both CPU and GPU builds. Uploads to OpenStack only happen on `main` branch. GPU images require actual GPU hardware - they'll fail without it.
49+
50+
## Image Contents
51+
52+
**Both images:**
53+
- Ubuntu 24.04 (Noble)
54+
- Docker
55+
- Node.js
56+
- cloud-init
57+
58+
**GPU images only:**
59+
- NVIDIA drivers
60+
- CUDA
61+
62+
## Manual Upload
63+
64+
If you need to upload manually:
65+
66+
```bash
67+
source /etc/kolla/admin-openrc.sh
68+
openstack image create my-image-name \
2869
--public --disk-format qcow2 \
2970
--container-format bare \
30-
--file <created-image>.qcow2
71+
--file path/to/image.qcow2
72+
```
73+
74+
## Cleanup
75+
76+
```bash
77+
make clean BUILD_DIR=/path/to/build
3178
```

0 commit comments

Comments
 (0)