Skip to content

Shutdown the runner VM if startup fails #60

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ravwojdyla merged 1 commit into from
Apr 8, 2025
Merged

Shutdown the runner VM if startup fails #60

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 17 additions & 7 deletions action.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -202,12 +202,22 @@ function start_vm {

echo "The new GCE VM will be ${VM_ID}"

shutdown_command="gcloud compute instances delete $VM_ID --zone=$machine_zone --quiet"
startup_prelude="#!/bin/bash
set -e
shutdown() {
echo ❌ Machine setup failed so deleting $VM_ID in ${machine_zone} ...
${shutdown_command}
}
trap shutdown EXIT
Copy link
Collaborator

@ravwojdyla ravwojdyla Apr 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jder actually - wouldn't this run the shutdown as soon as the startup script finish - even if there was no problem?

Copy link
Collaborator

@ravwojdyla ravwojdyla Apr 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jder please see 1ec4146, let me know if that still works for you.

Copy link
Contributor Author

@jder jder Apr 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, sorry, I was too focused on testing the failure case :) This makes more sense. Thanks!

Copy link
Collaborator

@ravwojdyla ravwojdyla Apr 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jder no problem, btw our CI

gce-github-runner/.github/workflows/test.yml

Line 32 in 008d5e1

- run: echo "This runs on the GCE runner VM"
did not catch this because it's too quick (it's just an echo) - it was able to finish before the VM was shutdown.

"

startup_script="
# Create a systemd service in charge of shutting down the machine once the workflow has finished
cat <<-EOF > /etc/systemd/system/shutdown.sh
#!/bin/sh
sleep ${shutdown_timeout}
gcloud compute instances delete $VM_ID --zone=$machine_zone --quiet
${shutdown_command}
EOF

cat <<-EOF > /etc/systemd/system/shutdown.service
Expand Down Expand Up @@ -238,12 +248,12 @@ function start_vm {
./svc.sh start && \\
gcloud compute instances add-labels ${VM_ID} --zone=${machine_zone} --labels=gh_ready=1
# 3 days represents the max workflow runtime. This will shutdown the instance if everything else fails.
nohup sh -c \"sleep 3d && gcloud --quiet compute instances delete ${VM_ID} --zone=${machine_zone}\" > /dev/null &
nohup sh -c \"sleep 3d && ${shutdown_command}\" > /dev/null &
"

if $actions_preinstalled ; then
echo "✅ Startup script won't install GitHub Actions (pre-installed)"
startup_script="#!/bin/bash
startup_script="${startup_prelude}
cd /actions-runner
$startup_script"
else
Expand All @@ -258,20 +268,20 @@ function start_vm {
fi
echo "✅ Startup script will install GitHub Actions v$runner_ver"
if $arm ; then
startup_script="#!/bin/bash
startup_script="${startup_prelude}
mkdir /actions-runner
cd /actions-runner
curl -o actions-runner-linux-arm64-${runner_ver}.tar.gz -L https://github.com/actions/runner/releases/download/v${runner_ver}/actions-runner-linux-arm64-${runner_ver}.tar.gz
tar xzf ./actions-runner-linux-arm64-${runner_ver}.tar.gz
./bin/installdependencies.sh && \\
./bin/installdependencies.sh
$startup_script"
else
startup_script="#!/bin/bash
startup_script="${startup_prelude}
mkdir /actions-runner
cd /actions-runner
curl -o actions-runner-linux-x64-${runner_ver}.tar.gz -L https://github.com/actions/runner/releases/download/v${runner_ver}/actions-runner-linux-x64-${runner_ver}.tar.gz
tar xzf ./actions-runner-linux-x64-${runner_ver}.tar.gz
./bin/installdependencies.sh && \\
./bin/installdependencies.sh
$startup_script"
fi
fi
Expand Down