Hung on command during update

[DrOnionMaster]

Hey all,

I am running into an issue where I am getting hung on a command after running soup

The command I keep getting hung on is:

[INFO ] Executing command '/usr/sbin/so-influxdb-manage' in directory '/root'

When I run this command independently from the update script it works fine, but it keeps hanging in the update.

Not sure if there is a fix for this, if not I'd like to edit the soup commands so that it skips this command during the update. The problem is I have no idea where this command is called / where the update script is located.

I'm confused why it even needs to run this command as the usage is:

Supported Operations:
dashboardpath Returns the URL path for a dashboard, requires:
export Exports all templates to stdout
setup Loads all templates and creates all required buckets
userlist Lists users
useradd Adds a new user, requires:
userdel Removes an existing user, requires:
userenable Enables a user, requires:
userdisable Disables a user, requires:
userpass Updates a user's password, requires:
userpromote Promotes a user to admin:
userdemote Demotes a user from admin:

If these are all the functions, then why does it need to run during soup??

[DrOnionMaster]

Another update:

I've tried editing the script within the 'so-influxdb-manage' file to contain only:
#!/bin/bash
exit 0

And then tried running soup again hoping it would exit the command and continue with the script, however this did not work and I am still hung on the command.

[DrOnionMaster]

Update: I was able to get through the update.

The problem I've found is within the call to salt \* state.highstate

ID: influxdb-setup
Function: cmd.run
Name: /usr/sbin/so-influxdb-manage setup &>> /opt/so/log/influxdb/setup.log
Result: False
Comment: Command "/usr/sbin/so-influxdb-manage setup &>> /opt/so/log/influxdb/setup.log" run
Started: 19:38:12.399766
Duration: 302493.461 ms
Changes:
----------
pid:
83801
retcode:
1
stderr:
stdout:
ID: metrics_link_file
Function: cmd.run
Name: so-influxdb-manage dashboardpath "Security Onion Performance" > /opt/so/saltstack/local/salt/influxdb/metrics_link.txt
Result: False
Comment: Command "so-influxdb-manage dashboardpath "Security Onion Performance" > /opt/so/saltstack/local/salt/influxdb/metrics_link.txt" run
Started: 19:43:15.145821
Duration: 301277.019 ms
Changes:
----------
pid:
103999
retcode:
1
stderr:
Thu Apr 11 19:43:15 UTC 2024 | so-influxdb-manage | Server does not appear to be running or fully initialized - will try again in 10 seconds (1 / 30)

[cm-ops]

I have those errors in my influxdb log and I don't see any issues with the service. How long have you let the command sit in that state?

influxdb-setup:
  cmd.run:
    - name: /usr/sbin/so-influxdb-manage setup &>> /opt/so/log/influxdb/setup.log
    - require:
      - file: influxdbbucketsconf
      - file: influxdb_curl_config
      - docker_container: so-influxdb

metrics_link_file:
  cmd.run:
    - name: so-influxdb-manage dashboardpath "Security Onion Performance" > /opt/so/saltstack/local/salt/influxdb/metrics_link.txt
    - require:
      - docker_container: so-influxdb

That seems to be where you are getting stuck, if you get to that state, check to see if the container is running.

[DrOnionMaster]

If we let the command sit long enough, it does eventually skip it. This completely takes down the InfluxDB service and when we run so-status everything is running except influxDB. The container is not running, and we can't get it to start again.

[BJOnionMan]

When sudo so-checkin is run it also triggers the "error" and so-influxdb container is missing yet again. Whenever any of the cheat sheet rules are done to update, the error occurs basically.

[cm-ops]

What files are in /opt/so/conf/influxdb?

[DrOnionMaster]

buckets.json
config.yaml
curl.config
etc
last_bucket_setup
last_template_setup
templates

[Acewiza]

There's gremlins in these things. I have a couple of old obscure, insignificant, forgotten files in a data directory somewhere, that always keep getting re-written on every rsync, even though I know they haven't been changed, touched or even thought of in many years.

Thanks for reminding me! I might go run that again now and see if I can get wrapped around it for a few hours...

[BJOnionMan]

If you run sudo so-checkin does it complete or still hang?

when sudo so-checkin is run it also triggers the "error" and so-influxdb container is missing yet again. Whenever any of the cheat sheet rules are done to update the error occurs basically.

[Koen1999]

I am having a similar issue. I am unable to start influxdb. The issue appears to be initially caused by an automatic update by soup or an issue there. Running so-influxdb-start yields:

[INFO    ] Executing command 'so-influxdb-manage' in directory '/root'
[ERROR   ] Command 'so-influxdb-manage' failed with return code: 1
[ERROR   ] stderr: Wed May 15 13:07:09 UTC 2024 | so-influxdb-manage | Server does not appear to be running or fully initialized - will try again in 10 seconds (1 / 30)

Any advice on how to remediate this issue?

[DrOnionMaster]

Same issue as us. Still have no solve, waiting for someone to figure this one out.

[DrOnionMaster]

Please upvote for more visibility.

[reyesj2]

Can you try

touch -m /opt/so/conf/influxdb/buckets.json
touch -m /opt/so/conf/influxdb/templates/*
so-influxdb-manage setup

After that you might need to run

so-influxdb-start

[Koen1999]

I solved it already by doing a full reinstall. Maybe someone else with the same issue can try these steps?