Update operation enhancement

[piotrmiskiewicz]

After investigation #2565 we concluded that update operation should be improved in order to defensively approach db potential bottlenecks.

To implement

• If update changes nothing do not do it. Return 200 or error when RuntimeCR is in erorr. Remember that we should not somehow that there was an update. Some counter is a proposition
  • we need to define what we persist, log, metrisc when we return the status and we didn't run operation
  • to keep current behaviour of KEB when users update instance, we need to check what status code should be returned to the user. If input params are the same it doesnt mean we need to always return 200. AS Jarek suggested, maybe we do not need to run new operation in such scenario, but jsty check Runtime Status.
  • New KEB metrics about number operations and number of calls on update per Runtime. We should clearly see instances that have too many operartions, or after this improvement deployment we should see high counters. Create alerts that given Runtime has big number of updates.
• do not store in main db in table operations all update operations. It should be limited to the last 100, the rest can be archived in e.g. operations_archive table for audit or critical failures._ PK comment: we can consider to work on that in the future

Side effects:

• kcp rt should work faster because we will stop getting instance with 6000+ operations. There will be less operations to display.
• table operations will grow linearly in predictable way depending on how many instances there are.

[piotrmiskiewicz]

Update endpoint response is HTTP 200 if updating parameters does not modify the instance and last update operation is succeeded.
If the last update is failed - new operation is created
If the last update is in progress - returns HTTP 202 with that operation id (there is no new operation)

[piotrmiskiewicz]

Docs: PR https://github.com/kyma-project/kyma-environment-broker/pull/2832/changes

[piotrmiskiewicz]

TODO:

Metrics

number_of_updates_total{instance_id}
empty_updates_total{instance_id}

in a dashboard we can see "top of"

in a dashboard - buckets, for example:
3 instances with more than 100 empty updates
13 instances with empty updates between 50 and 100
153 instances with empty updates between 20 and 50
900 instances with empty updates between 5 and 20

Alerts

We have to know when there is a lot of update operation in a short time (for one instance).
We have to know when there is a lot of empty updates in a short time (for one instance).

[piotrmiskiewicz]

number of updates (empty updates) in a given time, for example top 15 instances (number of updates) in last 24 hours

[piotrmiskiewicz]

Metrics proposal

1. number_of_update_operations_total{instance_id} - counts update operations in the DB, consider to not provide all instances but instances modified in last 2 - 10 days to reduce the payload
2. number_of_empty_updates_total{instance_id} - provides value saved in the DB (empty updates)
3. consider istio metrics usage: sum(increase(istio_requests_total{destination_service_name="kcp-kyma-environment-broker", request_method="PATCH", request_class="keb-osb"})[1h]) by (response_code)
4. consider metrics number_of_update_request{instance_id, response_code}