sundog error when building an AMI with in-place-updates = false

[rajivr]

I am attempting to create a Bottlerocket variant that has in-place update (migrations) disabled.

Using twoliter version 0.9.0 solves the build issue. However, when the AMI starts, sundog.service fails, which prevents control container from starting. Only way to reach the instance is by using the console. Following is the log.

bash-5.2# journalctl --unit sundog.service
Apr 21 10:45:51 localhost systemd[1]: Starting User-specified setting generators...
Apr 21 10:45:51 localhost settings-committer[1032]: 10:45:51 [INFO] Checking pending settings.
Apr 21 10:45:51 localhost settings-committer[1032]: 10:45:51 [INFO] Committing settings.
Apr 21 10:45:51 localhost sundog[1035]: 10:45:51 [INFO] Sundog started
Apr 21 10:45:51 localhost sundog[1035]: 10:45:51 [INFO] Retrieving setting generators
Apr 21 10:45:51 localhost sundog[1035]: 10:45:51 [INFO] Retrieving settings values
Apr 21 10:45:51 localhost sundog[1035]: 10:45:51 [INFO] Setting generator command 'shibaken' stderr: 10:45:51 [INFO] shibaken started
Apr 21 10:45:51 localhost sundog[1035]: 10:45:51 [INFO] Setting generator command 'shibaken' stderr: 10:45:51 [INFO] Connecting to IMDS
Apr 21 10:45:51 localhost sundog[1035]: 10:45:51 [INFO] Setting generator command 'shibaken' stderr: 10:45:51 [INFO] Fetching list of available public keys from IMDS
Apr 21 10:45:51 localhost sundog[1035]: 10:45:51 [INFO] Setting generator command 'shibaken' stderr: 10:45:51 [INFO] Generating user-data
Apr 21 10:45:51 localhost sundog[1035]: 10:45:51 [INFO] Setting generator command 'shibaken' stderr: 10:45:51 [INFO] Encoding user-data
Apr 21 10:45:51 localhost sundog[1035]: 10:45:51 [INFO] Setting generator command 'shibaken' stderr: 10:45:51 [INFO] Outputting base64-encoded user-data
Apr 21 10:45:51 localhost sundog[1035]: Failed to start generator 'bork seed': No such file or directory (os error 2)
Apr 21 10:45:51 localhost systemd[1]: sundog.service: Main process exited, code=exited, status=1/FAILURE
Apr 21 10:45:51 localhost systemd[1]: sundog.service: Failed with result 'exit-code'.
Apr 21 10:45:51 localhost systemd[1]: Failed to start User-specified setting generators.

You can find the code to reproduce the issue here.

The organization of the repo closely resembles upstream Bottlerocket repo, so with an appropriate Infra.toml, you can do cargo make -e BUILDSYS_VARIANT=aws-dev ami to build and publish AMI.

I would really appreciate any pointers on how I might be able to fix this failure. 🙏

[cbgbt]

Hello.

I believe the issue is that there still exists metadata telling Bottlerocket to attempt to generate a seed/wave position.

The relevant configuration in the repository you linked is here.

sundog runs any "settings generators" that are configured via those settings defaults. In this case, disabling in-place updates removed the bork binary from the system, so the configured setting generator that calls it is failing.

I believe you can resolve this by removing any "updates" related configurations and also removing updates-related settings from your variant by removing this line in the settings plugin.

Let us know if that helps, or if you have any other questions!

[rajivr]

Let us know if that helps, or if you have any other questions!

Thanks a lot for the quick reply @cbgbt .

I implemented the suggested changes. Additionally, I had to remove sources/settings-defaults/aws-dev/defaults.d/15-aws-tuf.toml and sources/settings-defaults/metal-dev/defaults.d/15-public-tuf.toml.

sundog now starts correctly.

bash-5.1# journalctl --unit sundog.service
Apr 25 07:15:35 localhost systemd[1]: Starting User-specified setting generators...
Apr 25 07:15:35 localhost settings-committer[1027]: 07:15:35 [INFO] Checking pending settings.
Apr 25 07:15:35 localhost settings-committer[1027]: 07:15:35 [INFO] Committing settings.
Apr 25 07:15:35 localhost sundog[1030]: 07:15:35 [INFO] Sundog started
Apr 25 07:15:35 localhost sundog[1030]: 07:15:35 [INFO] Retrieving setting generators
Apr 25 07:15:35 localhost sundog[1030]: 07:15:35 [INFO] Retrieving settings values
Apr 25 07:15:35 localhost sundog[1030]: 07:15:35 [INFO] Setting generator command 'shibaken' stderr: 07:15:35 [INFO] shibaken started
Apr 25 07:15:35 localhost sundog[1030]: 07:15:35 [INFO] Setting generator command 'shibaken' stderr: 07:15:35 [INFO] Connecting to IMDS
Apr 25 07:15:35 localhost sundog[1030]: 07:15:35 [INFO] Setting generator command 'shibaken' stderr: 07:15:35 [INFO] Fetching list of available public keys from IMDS
Apr 25 07:15:35 localhost sundog[1030]: 07:15:35 [INFO] Setting generator command 'shibaken' stderr: 07:15:35 [INFO] Generating user-data
Apr 25 07:15:35 localhost sundog[1030]: 07:15:35 [INFO] Setting generator command 'shibaken' stderr: 07:15:35 [INFO] Encoding user-data
Apr 25 07:15:35 localhost sundog[1030]: 07:15:35 [INFO] Setting generator command 'shibaken' stderr: 07:15:35 [INFO] Outputting base64-encoded user-data
Apr 25 07:15:35 localhost sundog[1030]: 07:15:35 [INFO] Sending settings values to the API
Apr 25 07:15:35 localhost systemd[1]: Finished User-specified setting generators.