Dual-homed Bottlerocket instance

[rajivr]

I am trying to set up a dual-homed Bottlerocket EC2 instance, where the primary ENI is in a subnet that has a route to an Internet Gateway, while the other ENI is in a private subnet.

I managed to get a basic setup working by manually introducing /.bottlerocket/net.toml file and rebooting the instance.

# /.bottlerocket/rootfs/.bottlerocket/net.toml
version = 1

[eth0]
dhcp4 = true
primary = true

[eth1]
dhcp4 = true

Is there a more canonical and automate-able approach that I can take?

I tried to see if there was a way to use kernel command line flags and netdog, but it looks like that is not possible.

I also tried to see it there was a cargo metadata option that I could use from buildsys that would let me inject a custom file into the BOTTLEROCKET-PRIVATE partition when building the image. However I could not find any.

I would appreciate any inputs on how this can be done. Thanks! 🙏

[KCSesh]

@rajivr,

Is there a more canonical and automate-able approach that I can take?

Yes! There are actually several ways to do this!

PR #714 added apiclient network configure which lets you write net.toml.

Using bootstrap-commands:

[settings.bootstrap-commands.setup-dual-nic]
commands = [
 ["apiclient", "network", "configure", "base64:dmVyc2lvbiA9IDEKCltldGgwXQpkaGNwNCA9IHRydWUKcHJpbWFyeSA9IHRydWUKCltldGgxXQpkaGNwNCA9IHRydWUK"],
 ["apiclient", "reboot"]
]
mode = "once"
essential = true

The base64 decodes to your net.toml. The instance boots, writes the config, reboots, and comes up with both interfaces configured.

echo "dmVyc2lvbiA9IDEKCltldGgwXQpkaGNwNCA9IHRydWUKcHJpbWFyeSA9IHRydWUKCltldGgxXQpkaGNwNCA9IHRydWUK" | base64 -d
version = 1

[eth0]
dhcp4 = true
primary = true

[eth1]
dhcp4 = true

Other approaches:

• Bootstrap container - Same idea but with a container, if you need more logic or want to provide a file as input:
  apiclient network configure file:// (also stdin is supported too)
• Custom Data partition - Since I know you have worked on and made bottlerocket-compose, you have an option: bake a default net.toml into /usr/share/bottlerocket/net.toml on your data partition (xvdb) - this should avoid the reboot entirely since the config is already present at first boot, as I understand.

Some gotchas to be aware of:

• Both ENIs must be attached at launch - there's no hot-plug support (see #3989)
• No wildcard/auto-discovery for interface names yet (see #416)
• K8s variants usually use CNI plugins to manage secondary ENI's - so probably shouldn't use it there.
• Runtime changes to net.toml require a reboot, which is why /usr/share/bottlerocket/net.toml baked in to data, may be preferable if you want to avoid that
• There is probably more 🤣

[rajivr]

Thanks @KCSesh for the detailed reply 🙏

I tested the bootstrap commands and it worked!

For now I will use this approach over creating another variant with custom /usr/share/bottlerocket/net.toml.