Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 54 additions & 32 deletions qemu_artifact.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,38 +10,58 @@ Given the size of the image, the first VM using it on a node might take a while

## Creating a bare-metal instance

We launch an Ubuntu 22 bare-metal instance; we're using the `c6g.metal` instance type in this case, but any ARM instance type is sufficient for our purposes.

aws ec2 create-security-group --group-name "launch-wizard-1" --description "launch-wizard-1 created 2024-11-26T00:32:56.039Z" --vpc-id "vpc-0fbfcc428751ce76b"
aws ec2 authorize-security-group-ingress --group-id "sg-preview-1" --ip-permissions '{"IpProtocol":"tcp","FromPort":22,"ToPort":22,"IpRanges":[{"CidrIp":"0.0.0.0/0"}]}'
aws ec2 run-instances --image-id "ami-0a87daabd88e93b1f" --instance-type "c6g.metal" --key-name "darora-aps1" --block-device-mappings '{"DeviceName":"/dev/sda1","Ebs":{"Encrypted":false,"DeleteOnTermination":true,"Iops":3000,"SnapshotId":"snap-0fe84a34403e3da8b","VolumeSize":200,"VolumeType":"gp3","Throughput":125}}' --network-interfaces '{"AssociatePublicIpAddress":true,"DeviceIndex":0,"Groups":["sg-preview-1"]}' --tag-specifications '{"ResourceType":"instance","Tags":[{"Key":"Name","Value":"darora-pg-image"}]}' --metadata-options '{"HttpEndpoint":"enabled","HttpPutResponseHopLimit":2,"HttpTokens":"required"}' --private-dns-name-options '{"HostnameType":"ip-name","EnableResourceNameDnsARecord":true,"EnableResourceNameDnsAAAARecord":false}' --count "1"

We launch an Ubuntu 22 bare-metal instance; we're using the `c6g.metal` instance type in this case, but any ARM instance type is sufficient for our purposes. In the example below the region used is: `ap-south-1`.

```bash
# create a security group for your instance
aws ec2 create-security-group --group-name "launch-wizard-1" --description "launch-wizard-1 created 2024-11-26T00:32:56.039Z" --vpc-id "insert-vpc-id"

# using the generated security group ID (insert-sg-group), ensure that it allows for SSH access
aws ec2 authorize-security-group-ingress --group-id "insert-sg-group" --ip-permissions '{"IpProtocol":"tcp","FromPort":22,"ToPort":22,"IpRanges":[{"CidrIp":"0.0.0.0/0"}]}'

# spin up your instance with the generated security group ID (insert-sg-group)
aws ec2 run-instances \
--image-id "ami-0a87daabd88e93b1f" \
--instance-type "c6g.metal" \
--key-name "INSERT_KEY_PAIR_NAME" \ # create a key pair, or use other mechanism of getting on to the box
--block-device-mappings '{"DeviceName":"/dev/sda1","Ebs":{"Encrypted":false,"DeleteOnTermination":true,"Iops":3000,"SnapshotId":"snap-0fe84a34403e3da8b","VolumeSize":200,"VolumeType":"gp3","Throughput":125}}' \
--network-interfaces '{"AssociatePublicIpAddress":true,"DeviceIndex":0,"Groups":["insert-sg-group"]}' \
--tag-specifications '{"ResourceType":"instance","Tags":[{"Key":"Name","Value":"qemu-pg-image"}]}' \
--metadata-options '{"HttpEndpoint":"enabled","HttpPutResponseHopLimit":2,"HttpTokens":"required"}' \
--private-dns-name-options '{"HostnameType":"ip-name","EnableResourceNameDnsARecord":true,"EnableResourceNameDnsAAAARecord":false}' \
--count "1"

```
## Install deps

On the instance, install the dependencies we require for producing QEMU artifacts:

sudo apt-get update
sudo apt-get install -y qemu-system qemu-system-arm qemu-utils qemu-efi-aarch64 libvirt-clients libvirt-daemon libqcow-utils software-properties-common git make libnbd-bin nbdkit fuse2fs cloud-image-utils awscli
sudo usermod -aG kvm ubuntu
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=arm64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install packer=1.11.2-1
sudo apt-get install -y docker.io
On the instance, install the dependencies we require for producing QEMU artifacts. Assuming you are the root user:

```bash
apt-get update
apt-get install -y qemu-system qemu-system-arm qemu-utils qemu-efi-aarch64 libvirt-clients libvirt-daemon libqcow-utils software-properties-common git make libnbd-bin nbdkit fuse2fs cloud-image-utils awscli
usermod -aG kvm ubuntu
curl -fsSL https://apt.releases.hashicorp.com/gpg | apt-key add -
apt-add-repository "deb [arch=arm64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
apt-get update && apt-get install packer=1.11.2-1
apt-get install -y docker.io
```

Some dev deps that might be useful:

sudo apt-get install -y emacs ripgrep vim-tiny byobu

```bash
apt-get install -y emacs ripgrep vim-tiny byobu
```

## Clone repo and build

Logout/login first to pick up new group memberships!

git clone https://github.com/supabase/postgres.git
cd postgres
git checkout da/qemu-rebasing # choose appropriate branch here
make init container-disk-image
``` bash
git clone https://github.com/supabase/postgres.git
cd postgres
git checkout da/qemu-rebasing # choose appropriate branch here
make init container-disk-image
```

### Build process

Expand All @@ -67,15 +87,15 @@ b. packer build (`qemu-arm64-nix.pkr.hcl`)

## Publish image for later use

Publish the built image to a registry of your choosing, and use the published image with KubeVirt.

Following `make init container-disk-image`, the generated image should be found in: `/path/to/postgres/output-cloudimg`. For portability the image is also bundled up as a docker image with the name: `supabase-postgres-test` . Publish the built docker image to a registry of your choosing, and use the published image with KubeVirt.

# Iterating on the QEMU artifact

For a tighter iteration loop on the Postgres artifact, the recommended workflow is to do so on an Ubuntu bare-metal node that's part of the EKS cluster that you're deploying to.

- Use the `host-disk` make target to build the raw image file on disk. (`/path/to/postgres/disk/focal-raw.img`)
- Update the VM spec to use `hostDisk` instead of `containerDisk`. Note that only one VM can use an image at a time, so you can't create multiple VMs backed by the same host disk.
- Instead of running `make init container-disk-image`, use `make init host-disk` instead to build the raw image file on disk. (`/path/to/postgres/disk/focal-raw.img`)
- Update the VM spec to use `hostDisk` instead of `containerDisk`
- Note that only one VM can use an image at a time, so you can't create multiple VMs backed by the same host disk.
- Enable the `HostDisk` feature flag for KubeVirt
- Deploy the VM to the node

Expand All @@ -85,11 +105,13 @@ Additionally, to iterate on the container image part of things, you can build th

Installing `docker.io` on an EKS node might interfere with the k8s setup of the node. You can instead install `nerdctl` and `buildkit`:

curl -L -O https://github.com/containerd/nerdctl/releases/download/v2.0.0/nerdctl-2.0.0-linux-arm64.tar.gz
tar -xzf nerdctl-2.0.0-linux-arm64.tar.gz
sudo mv ./nerdctl /usr/local/bin/
curl -O -L https://github.com/moby/buildkit/releases/download/v0.17.1/buildkit-v0.17.1.linux-arm64.tar.gz
tar -xzf buildkit-v0.17.1.linux-arm64.tar.gz
sudo mv bin/* /usr/local/bin/
```bash
curl -L -O https://github.com/containerd/nerdctl/releases/download/v2.0.0/nerdctl-2.0.0-linux-arm64.tar.gz
tar -xzf nerdctl-2.0.0-linux-arm64.tar.gz
mv ./nerdctl /usr/local/bin/
curl -O -L https://github.com/moby/buildkit/releases/download/v0.17.1/buildkit-v0.17.1.linux-arm64.tar.gz
tar -xzf buildkit-v0.17.1.linux-arm64.tar.gz
mv bin/* /usr/local/bin/
```

You'll need to run buildkit: `sudo buildkitd`
You'll need to run buildkit: `buildkitd`
Loading