Is my bottlerocket updater & OS working?

[landbaychrisburrell]

Hi

I'm struggling to understand whether my bottlerocket set of nodes are working or not. I saw a similar Github which suggested it might be fine. And another page (https://bottlerocket.dev/en/os/1.16.x/install/quickstart/aws/k8s/) which all looks fine. But it doesn't seem to be a very robust way of checking it's working:

• I've used the fragment to configure/label my nodes, which i can see labelled in EKS

        # extra args added
        [settings.kernel]
        lockdown = "integrity"

        # Configure the node-labels Bottlerocket setting
        [settings.kubernetes.node-labels]
        "bottlerocket.aws/updater-interface-version" = "2.0.0"

• i have 3x api servers, 1x controller-deployment, 1x agent for each server.
• Logs on the controller and api servers are clean

Logs in the agents come with stack traces on the INFO level, so i assume that's ok? I guess, aside from waiting, there's no real way of checking the process works?

INFO agent::agentclient: Checking for Bottlerocket updates.
INFO agent::apiclient::api: The lock for the update API is held by another process ...

These messages happen around 400ms apart, and continually repeat every 2 minutes - I would have thought (liked) to actually see a success message also? So I'm thinking it's not quite working yet...

Status of the BRS is

Status: │

│   crash_count:      0                                                                                                                                                                                         │   current_state:    Idle                                                                                                                                                                                     │   current_version:  1.16.0                                                                                                                                                                                    
│   target_version:   1.16.0

Any further thoughts?

[landbaychrisburrell]

Anyone? Is there a guide somewhere perhaps I could follow to help ascertain the operator/bottlerocket OS are configured correctly?

[yeazelm]

Hello @landbaychrisburrell Thanks for starting this discussion. I believe you are asking about if the Bottlerocket Update Operator is working correctly?

Can you paste the stack traces you referred to at the INFO level?

If there are no updates to fetch, there isn't much to check on. There was a release just yesterday so the waves for that release should be rolling out and you should see your nodes update when the window applies to that node. It typically take a week for all the nodes to get the updates.

[landbaychrisburrell]

Hi

Thanks - one of the issues, is that we are still on our test environments which we turn off overnight, and so my understanding is that the nodes are auto-updated on start-up. So once in production, we should be good to test/check it's all configured, but beforehand, would be good to carry out any other potential checks.

Sorry, I should have posted the stack traces before also:

  2023-11-14T16:18:03.002752Z  INFO agent::agentclient: Checking for Bottlerocket updates.
    at agent/src/agentclient.rs:209
    in agent::agentclient::shadow_status_with_refreshed_system_matadata with shadow_error_info: ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }
    in agent::agentclient::update_status_in_shadow with bottlerocket_shadow: BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2023-11-14T07:22:46Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(6), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some("brupop.bottlerocket.aws/v2"), fields_type: Some("FieldsV1"), fields_v1: Some(FieldsV1(Object {"f:metadata": Object {"f:ownerReferences": Object {".": Object {}, "k:{\"uid\":\"1a2ec357-a06e-4c07-b540-205ecbccfa3a\"}": Object {}}}, "f:spec": Object {".": Object {}, "f:state": Object {}}})), manager: Some("unknown"), operation: Some("Update"), subresource: None, time: Some(Time(2023-11-14T08:02:20Z)) }, ManagedFieldsEntry { api_version: Some("brupop.bottlerocket.aws/v2"), fields_type: Some("FieldsV1"), fields_v1: Some(FieldsV1(Object {"f:status": Object {".": Object {}, "f:crash_count": Object {}, "f:current_state": Object {}, "f:current_version": Object {}, "f:target_version": Object {}}})), manager: Some("unknown"), operation: Some("Update"), subresource: Some("status"), time: Some(Time(2023-11-14T08:02:27Z)) }]), name: Some("brs-ip-10-0-159-57.eu-west-1.compute.internal"), namespace: Some("brupop-bottlerocket-aws"), owner_references: Some([OwnerReference { api_version: "v1", block_owner_deletion: None, controller: None, kind: "Node", name: "ip-10-0-159-57.eu-west-1.compute.internal", uid: "1a2ec357-a06e-4c07-b540-205ecbccfa3a" }]), resource_version: Some("2710631"), self_link: None, uid: Some("5780f965-3c7f-41c4-bd9f-ca25134bb368") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: "1.16.1", target_version: "1.16.1", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, state: Idle, shadow_error_info: ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }
    in agent::agentclient::run

  2023-11-14T16:18:03.402711Z  INFO agent::apiclient::api: The lock for the update API is held by another process ...
    at agent/src/apiclient.rs:298
    in agent::apiclient::api::invoke_apiclient with args: ["raw", "-u", "/updates/status"]
    in agent::apiclient::api::get_update_status
    in agent::agentclient::shadow_status_with_refreshed_system_matadata with shadow_error_info: ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }
    in agent::agentclient::update_status_in_shadow with bottlerocket_shadow: BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2023-11-14T07:22:46Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(6), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some("brupop.bottlerocket.aws/v2"), fields_type: Some("FieldsV1"), fields_v1: Some(FieldsV1(Object {"f:metadata": Object {"f:ownerReferences": Object {".": Object {}, "k:{\"uid\":\"1a2ec357-a06e-4c07-b540-205ecbccfa3a\"}": Object {}}}, "f:spec": Object {".": Object {}, "f:state": Object {}}})), manager: Some("unknown"), operation: Some("Update"), subresource: None, time: Some(Time(2023-11-14T08:02:20Z)) }, ManagedFieldsEntry { api_version: Some("brupop.bottlerocket.aws/v2"), fields_type: Some("FieldsV1"), fields_v1: Some(FieldsV1(Object {"f:status": Object {".": Object {}, "f:crash_count": Object {}, "f:current_state": Object {}, "f:current_version": Object {}, "f:target_version": Object {}}})), manager: Some("unknown"), operation: Some("Update"), subresource: Some("status"), time: Some(Time(2023-11-14T08:02:27Z)) }]), name: Some("brs-ip-10-0-159-57.eu-west-1.compute.internal"), namespace: Some("brupop-bottlerocket-aws"), owner_references: Some([OwnerReference { api_version: "v1", block_owner_deletion: None, controller: None, kind: "Node", name: "ip-10-0-159-57.eu-west-1.compute.internal", uid: "1a2ec357-a06e-4c07-b540-205ecbccfa3a" }]), resource_version: Some("2710631"), self_link: None, uid: Some("5780f965-3c7f-41c4-bd9f-ca25134bb368") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: "1.16.1", target_version: "1.16.1", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, state: Idle, shadow_error_info: ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }
    in agent::agentclient::run

Thanks for getting back to me.

[landbaychrisburrell]

Hi

Thanks - in our test environments we shutdown - and in production the nodes will stay up all the time (we're in the process of migrating from ECS) - so hence the operator requirement and the desire to test ahead - i guess ultimately we will want to either leave our test envs on until the next update or check in prod.

Thanks for confirming on the logs
Cheers

[yeazelm]

You could also specifically find an older version image, launch with it and check that brupop does the right thing if you are trying to validate the configuration of brupop.

Just replace region-code with your desired region (this looks up 1.16.0 which is one version older than latest currently):

aws ssm get-parameter --name /aws/service/bottlerocket/aws-k8s-1.28/x86_64/1.16.0/image_id --region region-code --query "Parameter.Value" --output text

[Glaaj]

My sincere apologies If I am opening this thread by asking this question. I am running into the exact same problem as described in OP's post.

We are testing the setup of brupop before pushing it to our prod workload, our testclusters are rebuilt every day, therefore I made a version pin in our terraform code that deploys an "old AMI" for the nodes.

So far after a couple of days and some tinkering I am unable to have brupop detect a newer version for the (2) nodes and do it's thing afterwards. What am I missing here?

[yeazelm]

Hello @Glaaj

So far after a couple of days and some tinkering I am unable to have brupop detect a newer version for the (2) nodes and do it's thing afterwards. What am I missing here?

I'm not entirely sure what you are seeing but you should first confirm the nodes come up with an old version of Bottlerocket (apiclient get os can confirm that). Then you should check for the BottlerocketShadowStatus entry and confirm if you are seeing the older version or if the node is already on the latest. As of today, 1.41.0 should be the latest but depending on which wave your node is in, it might be seeing 1.40.0.

[Glaaj]

Hi I managed to find the "problem" here. Bottlerocket was configured with the use of waves. So even though updates are available the waves were holding them back from reaching the machine.