salt/salt/minion-check.sls syntax error?

[darth-drew]

We are currently to diagnose errors in salt-master output, seeing regular "Event iteration failed with exception: 'list' object has no attribute 'items'" that may be coming from the title file under splay settings.

We copied this file into local to place a temporary cron.absent on so-salt-minion-check while highstates are taking longer than 2 hours to complete. We may be opening up a second discussion on this.

[dougburks]

Event iteration failed with exception: 'list' object has no attribute 'items

This is a known issue and can be ignored:
https://docs.securityonion.net/en/2.3/salt.html#known-issues

[OrangeDog]

At the Salt project this issue was not known, and the cause given here doesn't make much sense.
Did you file this at https://github.com/saltstack/salt/issues or discuss it with Salt developers somewhere?

[m0duspwnens]

@OrangeDog This was not discussed with the Salt developers, nor was it filed as an issue as I thought it was intended behavior rather than a bug. I'll explain how I came to the conclusion for the root cause below:

In this PR the exception was changed from just passing to logging it.

To figure out what was happening and the value of ret, I modified /usr/lib/python3.6/site-packages/salt/event.py to log it and restarted salt-master. It looks something like this:

log.error(ret)

Then from the salt-master, I tell the minion to run a highstate. Once the state is running, login to the minion and apply a state. Back in the salt-master's log, the following shows up:

2022-10-20 14:40:25,061 [salt.utils.event :887 ][ERROR   ][7714] ['The function "state.highstate" is running as PID 31965 and was started at 2022, Oct 20 14:02:15.910209 with jid 20221020140215910209']
2022-10-20 14:40:25,061 [salt.utils.event :910 ][ERROR   ][7714] Event iteration failed with exception: 'list' object has no attribute 'items'

Looking at the value of ret I concluded that the code it trying to loop through ret.items(), but ret is a list and lists don't have an items() method and therefore the exception is fired.

        try:
            for tag, data in ret.items():

[OrangeDog]

@m0duspwnens That is not intended behaviour. A correctly-working system should not be constantly logging errors.
Please file this as a proper issue so we can work out if there's a bug or if you're doing something wrong.

[m0duspwnens]

@OrangeDog Thanks for the feedback. I have submitted the issue.

[m0duspwnens]

@darth-drew If your highstates are taking 2 hours to complete, then it may be helpful for you to log into one of your non manager nodes and manually run a highstate while logging it to a file. This could be done by running sudo salt-call state.highstate -l info > highstate.log 2>&1. Once you have the log file, you can run egrep 'ID:|Duration:' highstate.log, which will provide a listing of the state IDs and how long they took to compete. If you can paste that output here, I would be interested in seeing which states are taking the most time so that we can possibly assist in reducing your highstate runtime down to a more reasonable level.

[darth-drew]

Unfortunately I can't post directly, and I am not sure what value this will provide, but the top ten results for a forward node that just took 71minutes to complete (with 301 Succeeded jobs, 11 changed, and 0 failed) are:

utilsyncscripts (2204967)
analyzerscripts (1121916)
crsynckeys (124136)
custom_file1 (102904)
nginxhtml (85393)
nginxconf (65349)
tgrafsyncscripts (59617)
salt_minion_service_unit_file ( 54820)
custom_file2 (54738)
custom_file3 (53078)

The custom files are just file.managed states.

These ten jobs make up (if I did this math properly) 3926918 ms. or 65.5 of the 71 minutes.

[dougburks]

For comparison, here are some numbers from a forward node in a healthy deployment:

ID: utilsyncscripts
    Duration: 2429.168 ms
ID: analyzerscripts
    Duration: 2671.596 ms
ID: crsynckeys
    Duration: 122.232 ms
ID: nginxhtml
    Duration: 132.447 ms
ID: nginxhtml
    Duration: 132.447 ms
ID: nginxconf
    Duration: 74.824 ms
ID: tgrafsyncscripts
    Duration: 375.966 ms
ID: salt_minion_service_unit_file
    Duration: 11.876 ms

Are your boxes severely underpowered?

Is the bandwidth between boxes severely limited?

[darth-drew]

Thanks doug. No, the boxes have plenty of resources from what I can tell. I will try to get a highstate runtime on the manager today. We are running just shy of 200 forward nodes and 40 search nodes. Are there any suggestions regarding handling this number of minions? We have a couple ideas but I am hopeful you may have some suggestions. We have tried a few things already, such as increasing ulimit and zeromq settings such as those found here: https://docs.saltproject.io/en/latest/topics/troubleshooting/master.html

*While somewhat slow at times, salt was usable on this grid before the upgrade from SO 2.3.90 directly to 2.3.130. I have experienced this same kind of issue with salt during the upgrade to 2.3.100 though on other grids.

[dougburks]

I will try to get a highstate runtime on the manager today.

We can try to review these metrics and provide basic feedback as time allows.

We are running just shy of 200 forward nodes and 40 search nodes.

Per our other discussion, this may be another opportunity for ammunition for your management chain. Troubleshooting a 200-node deployment at this level of detail goes well beyond the Basic Q&A that Github Discussions is designed for and clearly falls into the Advanced Configuration Support category as shown in the comparison chart here:
https://securityonionsolutions.com/support/

Again, I'm not trying to be a salesman here and I'm not trying to beat you up, just trying to suggest ways to convince your management chain to get you the support that you need. 😄

[EbolaWare]

For comparison, here are some numbers from a forward node in a healthy deployment:

When you say "healthy deployment", what rate of traffic does each node see? How many nodes in it? What kind of resources do forward and manager have?

"Healthy" says nothing without context.