Skip to content

Gzip compress userdata on Linux #28

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gabriel-samfira merged 3 commits into from
Sep 16, 2025
Merged

Gzip compress userdata on Linux #28

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5f8057a
Gzip compress userdata on Linux
sapslaj Sep 15, 2025
7763a38
Refactor userdata compression to separate function
sapslaj Sep 15, 2025
4809cb9
Fix userdata size limit
sapslaj Sep 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 21 additions & 13 deletions internal/spec/spec.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ package spec
import (
"archive/zip"
"bytes"
"compress/gzip"
"encoding/base64"
"encoding/json"
"fmt"
Expand Down Expand Up @@ -254,37 +255,44 @@ func (r *RunnerSpec) ComposeUserData() (string, error) {
bootstrapParams.UserDataOptions.EnableBootDebug = r.EnableBootDebug

var udata []byte
var b bytes.Buffer
switch bootstrapParams.OSType {
case params.Linux:
cloudCfg, err := cloudconfig.GetCloudConfig(bootstrapParams, r.Tools, bootstrapParams.Name)
if err != nil {
return "", fmt.Errorf("failed to generate userdata: %w", err)
}
udata = []byte(cloudCfg)
gzipped := gzip.NewWriter(&b)
Copy link
Member

@gabriel-samfira gabriel-samfira Sep 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about creating a new function like:

func maybeCompressUserdata(udata []byte, targetOS bootstrapParams.OSType) ([]byte, error) {
	// feel free to define the result of 1<<24 as a constant.
	if len(udata) < 1<<24 {
		return udata, nil
	}
	var b bytes.Buffer
	switch targetOS {
	case params.Windows:
		zipped := zip.NewWriter(&b)
		fd, err := zipped.Create("udata")
		if err != nil {
			return "", err
		}
		if _, err := fd.Write(udata); err != nil {
			return "", fmt.Errorf("failed to compress cloud config: %w", err)
		}
		if err := zipped.Close(); err != nil {
			return "", err
		}
	default:
		gzipped := gzip.NewWriter(&b)
		if _, err := gzipped.Write(udata); err != nil {
			return "", fmt.Errorf("failed to compress cloud config: %w", err)
		}
		if err := gzipped.Close(); err != nil {
			return "", err
		}
	}
	return b.Bytes(), nil
}

For linux, this will most likely not compress the userdata, as it's under 16KB. It will only compress it if you override the userdata in the pool via extra specs with something larger. It will automatically compress if it goes above 16KB. AWS will complain if it's larder than 16 KB anyway because in 2025 AD, it still has the same userdata limit it did in 2007.

We can use a function like this (I have not tested it) immediately after the switch statement in the ComposeUserData() function.

What do you think? This way you should see it in plain text in the instance userdata for linux machines at least.

Copy link
Contributor Author

@sapslaj sapslaj Sep 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. I have to step away for a while and won't be able to test until later today, but went ahead and pushed the change if you happen to get around to testing it before I do.

And don't get me started on AWS's ridiculousness 😂 I have to do deal with it way too much already! haha

Copy link
Contributor Author

@sapslaj sapslaj Sep 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just got around to testing this and it seems to work as intended. I can see the userdata in plaintext by default, so this seems to be good to go unless you want to move that 1<<24 out to a constant.

Copy link
Member

@gabriel-samfira gabriel-samfira Sep 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nah. We can do that later. Thanks for the PR!

Copy link
Member

@gabriel-samfira gabriel-samfira Sep 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you by any chance test with a Windows runner as well? No worries if not, I can test it out before merging.

Copy link
Contributor Author

@sapslaj sapslaj Sep 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ooh thanks for the suggestion. I tried launching a Windows runner and it's complaining about the user data limit 😞

{"time":"2025-09-16T10:50:42.466779291Z","level":"ERROR","source":{"function":"github.com/cloudbase/garm/runner/pool.(*basePoolManager).addPendingInstances.func1","file":"/opt/garm/garm/runner/pool/pool.go","line":1564},"msg":"failed to add instance to provider","error":"error creating instance: provider binary /usr/local/bin/garm-provider-aws returned error: provider binary failed with stdout: ; stderr: failed to run command: failed to create instance in provider: failed to create instance: failed to create instance: operation error EC2: RunInstances, https response error StatusCode: 400, RequestID: b818e72f-edc1-4b9e-95cc-833d41c19932, api error InvalidParameterValue: User data is limited to 16384 bytes\n: exit status 1","runner_name":"garm-uiBdLbGzFKAf","pool_mgr":"sapslaj/garm-test","endpoint":"github.com","pool_type":"repository"}
{"time":"2025-09-16T10:50:42.469819054Z","level":"ERROR","source":{"function":"github.com/cloudbase/garm/runner/pool.(*basePoolManager).addPendingInstances.func1","file":"/opt/garm/garm/runner/pool/pool.go","line":1573},"msg":"failed to create instance in provider","error":"error creating instance: provider binary /usr/local/bin/garm-provider-aws returned error: provider binary failed with stdout: ; stderr: failed to run command: failed to create instance in provider: failed to create instance: failed to create instance: operation error EC2: RunInstances, https response error StatusCode: 400, RequestID: b818e72f-edc1-4b9e-95cc-833d41c19932, api error InvalidParameterValue: User data is limited to 16384 bytes\n: exit status 1","runner_name":"garm-uiBdLbGzFKAf","pool_mgr":"sapslaj/garm-test","endpoint":"github.com","pool_type":"repository"}

let me see if I can figure out what's going wrong if anything...

Copy link
Contributor Author

@sapslaj sapslaj Sep 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found the issue. 1<<24 is 16 MB not 16 KB. Seems to be fixed now. Windows runners are working on my install at least!

Copy link
Member

@gabriel-samfira gabriel-samfira Sep 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's fine. I messed up only by a couple of orders of magnitude. Near miss. 😅

Waiting for the CI and merging. Thanks again for everything!

Copy link
Contributor Author

@sapslaj sapslaj Sep 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haha, that happens to the best of us. No problem, thanks for making a great project!

if _, err := gzipped.Write(udata); err != nil {
return "", fmt.Errorf("failed to compress cloud config: %w", err)
}
if err := gzipped.Close(); err != nil {
return "", err
}

case params.Windows:
cloudCfg, err := cloudconfig.GetCloudConfig(bootstrapParams, r.Tools, bootstrapParams.Name)
if err != nil {
return "", fmt.Errorf("failed to generate userdata: %w", err)
}
wrapped := fmt.Sprintf("<powershell>%s</powershell>", cloudCfg)
udata = []byte(wrapped)
zipped := zip.NewWriter(&b)
fd, err := zipped.Create("udata")
if err != nil {
return "", err
}
if _, err := fd.Write(udata); err != nil {
return "", fmt.Errorf("failed to compress cloud config: %w", err)
}
if err := zipped.Close(); err != nil {
return "", err
}
default:
return "", fmt.Errorf("unsupported OS type for cloud config: %s", bootstrapParams.OSType)
}

var b bytes.Buffer
zipped := zip.NewWriter(&b)
fd, err := zipped.Create("udata")
if err != nil {
return "", err
}
if _, err := fd.Write(udata); err != nil {
return "", fmt.Errorf("failed to compress cloud config: %w", err)
}
if err := zipped.Close(); err != nil {
return "", err
}

asBase64 := base64.StdEncoding.EncodeToString(b.Bytes())
return asBase64, nil
}