Failed to remove obsolete scheduled downtimes

[drapiti]

I know this issue was already reported previously however in version 2.15.1 the scheduled downtimes configured through Icinga director are not removed automatically when the apply rule is no longer satisfied, nor are new scheduled downtime apply rules applied automatically.

This is quite critical and urgently needs a fix.

Issue 10124

Trouble changing/removing scheduled downtime created from apply rules

The only workaround was to disable the host deploy and then re-enable the object, only then was the old scheduled object cleared and the new one applied.

[jschmidt-icinga]

I can not reproduce this with the steps provided in the original issue (i.e. a manual ScheduledDowntime apply rule with assign where matching a specific host name). Downtimes seem to get removed correctly and the new ones added.

Can you give a minimal (ideally non-director) list of steps to reproduce?

[drapiti]

Yes I will give you my example exactly: We have a list of predefined downtime schedules which are determined from an external source.
We recieve an updated tag which essentially sets the correct tag on the host and therefore should update the schedule. However I noticed in a recent test that for hosts which had an existing schedule, when it was changed while the new tag was updated on the host the scheduled down time did not remove the old schedule nor did it apply the new schedule. We use the same variable host.vars.field_downtime_schedule to associate these schedules.

[yhabteab]

however in version 2.15.1 the scheduled downtimes configured through Icinga director are not removed automatically when the apply rule is no longer satisfied

This is not the same as the original issue! If you update the customvar on the Host object used in the apply conditions for the ScheduledDowtnime object, it won't change anything, as we don't re-evaluate apply rules on changes made via the API. To fix that, you've to reload icinga2, so that the apply rules are re-evaluated and orphaned downtimes are purged. Alternatively, instead of changing the customvars (tags) via the Icinga 2 API, why not change them from within Icinga Director and trigger a deployment? This would also automatically reload Icinga 2 and re-apply the apply rules, otherwise there's currently no way to make this work.

[jschmidt-icinga]

nor did it apply the new schedule.

That is what I would expect in this case.

So in the short-term either do what @yhabteab said and deploy attribute changes via director, or do what the API docs suggest and recreate the relevant ScheduledDowntime objects after you updated the attributes.

Btw: There's already an old feature request here that relatively recently had another comment. Maybe we should look into that at some point. I'll bring it up on our next team meeting.

[drapiti]

ga2, so that the apply rules are re-evaluated and orphaned downtimes are purged. Alternatively, instead of changing the customvars (tags) via the Icinga 2 API, why not change them from within Icinga Director and trigger a deployment? This would also automatically reload Icinga 2 and re-apply the apply rules, otherwise there's curre

Ok got it.. I will test the reload. I would have thought that a reload occurrs in any case on any deploy or when we delpoy the change of the variable. The change of the variable is always performed on the director api not icinga directly. So you suggest to perform an additional reload once the variable has been distruibuted I guess. I will test it and get back to you.

[drapiti]

No as I suspected a successive reload of icinga though another deploy does not resolve the issue. This is quite a handicap, hopefully it can be adressed.

[jschmidt-icinga]

If you roll out changes to the host.vars.field_downtime_schedule variable via /v1/objects, a reload is not going to help you (director or otherwise), because the changed attributes are applied after apply rules have been evaluated.

I've briefly looked into how all these things fit together and it wouldn't be trivial to change this, since there is a bit of a chicken<->egg problem here. Loading the config during restart involves basically two steps, 'Commit' and 'Activation'. Commit needs to be done before the old worker is shut down, to check the config still works, but the shutdown needs to at least be initiated so the old worker can write out a final list of modified attributes. The new worker then loads these modified attributes first thing in the Activation phase. However, apply rules have already been evaluated in Commit and won't take the modified attributes into account anymore.

So this isn't simply a matter of putting things in the correct order, but we would actually need to re-apply the apply rules during runtime, same as when modifying them via the API in the first place. This is something we've avoided doing so far, but as I said, I'll discuss this with my colleagues.

(Alternatively we could add a "pre-shutdown" phase where the old worker dumps the modified attributes and the locks are acquired, but the real shutdown can still be averted if config validation fails. This would come at the cost of complicating our worker process management even further.)

[Al2Klimov]

If you roll out changes to the host.vars.field_downtime_schedule variable via /v1/objects, a reload is not going to help you (director or otherwise), because the changed attributes are applied after apply rules have been evaluated.

I've briefly looked into how all these things fit together and it wouldn't be trivial to change this, since there is a bit of a chicken<->egg problem here.

So this isn't simply a matter of putting things in the correct order,

Isn't it?

• modified-attributes.conf: store attributes to modify in Internal.modified_attributes #8155

[jschmidt-icinga]

So this isn't simply a matter of putting things in the correct order,

Isn't it?

Entirely depends on your definition of simple.

Also, that PR doesn't solve the reload issue, at least on its own. You'd at least need a way to trigger dumping the attributes before shutdown starts and some way to block additional calls to ModifyAttribute() until the Commit step is done.

[jschmidt-icinga]

@drapiti We've discussed this issue at our meeting today and it has become clear that developing this as a feature is not what we want, because reevaluating apply-rules during runtime is just too complex for what it adds.

The biggest issue in my eyes is the aforementioned unexpected behavior of modified_attributes only being evaluated after apply rules have already been applied. I think @Al2Klimov's linked PR is a good step towards making that behavior consistent, but there's still a couple things to do and consider there, so that'll still take some time.

On your end I'd like to know more about the use-case that led you to run into this issue. We want to see if there is maybe a way to cover this use-case with a more streamlined feature, so it doesn't need to rely on updating apply rules, with or without a restart:

Can you tell us based on what criteria the updates of the downtime are dispatched and what kind of automation is involved?
Have you considered manually managing the downtimes via the v1/actions/(schedule|remove)-downtime endpoints and if yes, why does it not work for your use-case?

[drapiti]

@drapiti We've discussed this issue at our meeting today and it has become clear that developing this as a feature is not what we want, because reevaluating apply-rules during runtime is just too complex for what it adds.

The biggest issue in my eyes is the aforementioned unexpected behavior of modified_attributes only being evaluated after apply rules have already been applied. I think @Al2Klimov's linked PR is a good step towards making that behavior consistent, but there's still a couple things to do and consider there, so that'll still take some time.

On your end I'd like to know more about the use-case that led you to run into this issue. We want to see if there is maybe a way to cover this use-case with a more streamlined feature, so it doesn't need to rely on updating apply rules, with or without a restart:

Can you tell us based on what criteria the updates of the downtime are dispatched and what kind of automation is involved? Have you considered manually managing the downtimes via the v1/actions/(schedule|remove)-downtime endpoints and if yes, why does it not work for your use-case?

Yes so we have 2 streams regarding downtimes. We have an immediate synchronous api integration which is triggered from CMDB for a specific server directly on the icinga2 api, and then at a larger scale we have recurring scheduled downtimes which applies asyncronously with a specific tag on the host on the icinga-director api with the aim of updating or removing a specific schedule which has been pre-configured as a scheduled downtime in icinga director. We have elected for this approach because we have a large environment over 30K vm's distributed on multiple icinga clusters, this avoids overwhelming the api. The thing that is a little strange is that at the moment even after a deploy through icinga director the reload is not triggering an update to these apply rules. The current workaround is that we disable the host deploy and then subsequently enable the host and deploy. This will correctly apply the new rules based on the updated attribute/schedule. It works however I would like at least to be able to force a reload without disabling the monitoring completely even if for a short period of time. Users may modify schedules at will for their systems so we need for the configuration to update preferably without disabling monitoring.

[jschmidt-icinga]

It works however I would like at least to be able to force a reload without disabling the monitoring completely even if for a short period of time.

I agree with this. At the very least, this is very inconsistent behavior and will continue to baffle users no matter how well it is documented. I hope we can have this fixed at some point but it might not be soon.

Users may modify schedules at will for their systems so we need for the configuration to update preferably without disabling monitoring.

Have you looked into the Director REST-API instead to trigger these custom var updates and a subsequent director redeploy instead? Do note though that you would have to reset the changed vars on your hosts using the Icinga 2 /v1/objects/Hosts with the restore_attrs attribute.

There's also the Director database import/sync feature, with which you could define all your hosts in an SQL database (maybe even import directly from CMDB?) and let director fully handle the imports/sync between this database and the generated configuration.

I'm not an expert on the Director, so take these suggestions with a grain of salt, but having the entire workflow in the director would seem like the sensible thing to me.

[drapiti]

It works however I would like at least to be able to force a reload without disabling the monitoring completely even if for a short period of time.

I agree with this. At the very least, this is very inconsistent behavior and will continue to baffle users no matter how well it is documented. I hope we can have this fixed at some point but it might not be soon.

Users may modify schedules at will for their systems so we need for the configuration to update preferably without disabling monitoring.

Have you looked into the Director REST-API instead to trigger these custom var updates and a subsequent director redeploy instead? Do note though that you would have to reset the changed vars on your hosts using the Icinga 2 /v1/objects/Hosts with the restore_attrs attribute.

There's also the Director database import/sync feature, with which you could define all your hosts in an SQL database (maybe even import directly from CMDB?) and let director fully handle the imports/sync between this database and the generated configuration.

I'm not an expert on the Director, so take these suggestions with a grain of salt, but having the entire workflow in the director would seem like the sensible thing to me.

Yes we use the director exclusively for scheduled downtimes. No change to the problem however. I am not quite sure of what you mean by resetting the changed vars on your hosts with restore_attrs