playbookdbuser issues #9716

deckeryan · 2023-02-03T15:01:40Z

deckeryan
Feb 3, 2023

Hey Guys,

I noticed this a few days ago when trying to run a salt highstate:

[INFO    ] Running state [playbookdbuser] at time 14:36:48.949024
[INFO    ] Executing state mysql_user.present for [playbookdbuser]
[WARNING ] /usr/lib/python3.6/site-packages/salt/modules/mysql.py:673: Warning: (1681, "'PASSWORD' is deprecated and will be removed in a future release.")
  return cur.execute(qry, args)

[INFO    ] User [email protected]/255.255.255.0 is already present with the desired password
[INFO    ] Completed state [playbookdbuser] at time 14:36:48.955927 (duration_in_ms=6.903)
[INFO    ] Running state [query_playbookdbuser_grants] at time 14:36:48.956283
[INFO    ] Executing state mysql_query.run for [query_playbookdbuser_grants]
[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 438, in _connect
    dbc = MySQLdb.connect(**connargs)
  File "/usr/lib64/python3.6/site-packages/MySQLdb/__init__.py", line 85, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib64/python3.6/site-packages/MySQLdb/connections.py", line 247, in __init__
    self.converter[t].append((None, string_decoder))
AttributeError: type object 'str' has no attribute 'append'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/state.py", line 2180, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1249, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/states/mysql_query.py", line 376, in run
    query_result = __salt__["mysql.query"](database, query, **connection_args)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 773, in query
    dbc = _connect(**connection_args)
  File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 444, in _connect
    except MySQLdb.err.InternalError as exc:
AttributeError: module 'MySQLdb' has no attribute 'err'

[INFO    ] Completed state [query_playbookdbuser_grants] at time 14:36:48.973416 (duration_in_ms=17.129)
[INFO    ] Running state [query_updatwebhooks] at time 14:36:48.973899
[INFO    ] Executing state mysql_query.run for [query_updatwebhooks]
[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 438, in _connect
    dbc = MySQLdb.connect(**connargs)
  File "/usr/lib64/python3.6/site-packages/MySQLdb/__init__.py", line 85, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib64/python3.6/site-packages/MySQLdb/connections.py", line 247, in __init__
    self.converter[t].append((None, string_decoder))
AttributeError: type object 'str' has no attribute 'append'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/state.py", line 2180, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1249, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/states/mysql_query.py", line 376, in run
    query_result = __salt__["mysql.query"](database, query, **connection_args)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 773, in query
    dbc = _connect(**connection_args)
  File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 444, in _connect
    except MySQLdb.err.InternalError as exc:
AttributeError: module 'MySQLdb' has no attribute 'err'

[INFO    ] Completed state [query_updatwebhooks] at time 14:36:48.980777 (duration_in_ms=6.877)
[INFO    ] Running state [query_updatename] at time 14:36:48.981103
[INFO    ] Executing state mysql_query.run for [query_updatename]
[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 438, in _connect
    dbc = MySQLdb.connect(**connargs)
  File "/usr/lib64/python3.6/site-packages/MySQLdb/__init__.py", line 85, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib64/python3.6/site-packages/MySQLdb/connections.py", line 247, in __init__
    self.converter[t].append((None, string_decoder))
AttributeError: type object 'str' has no attribute 'append'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/state.py", line 2180, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1249, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/states/mysql_query.py", line 376, in run
    query_result = __salt__["mysql.query"](database, query, **connection_args)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 773, in query
    dbc = _connect(**connection_args)
  File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 444, in _connect
    except MySQLdb.err.InternalError as exc:
AttributeError: module 'MySQLdb' has no attribute 'err'

[INFO    ] Completed state [query_updatename] at time 14:36:48.986930 (duration_in_ms=5.826)
[INFO    ] Running state [query_updatepluginurls] at time 14:36:48.987239
[INFO    ] Executing state mysql_query.run for [query_updatepluginurls]
[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 438, in _connect
    dbc = MySQLdb.connect(**connargs)
  File "/usr/lib64/python3.6/site-packages/MySQLdb/__init__.py", line 85, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib64/python3.6/site-packages/MySQLdb/connections.py", line 247, in __init__
    self.converter[t].append((None, string_decoder))
AttributeError: type object 'str' has no attribute 'append'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/state.py", line 2180, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1249, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/states/mysql_query.py", line 376, in run
    query_result = __salt__["mysql.query"](database, query, **connection_args)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 773, in query
    dbc = _connect(**connection_args)
  File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 444, in _connect
    except MySQLdb.err.InternalError as exc:
AttributeError: module 'MySQLdb' has no attribute 'err'

[INFO    ] Completed state [query_updatepluginurls] at time 14:36:48.993456 (duration_in_ms=6.216)

Has anyone else ran into this? Everything seems to still be working. Even playbook which is interesting.

Answered by deckeryan

Feb 7, 2023

After running soup again my errors have disappeared. Thank you for making such a great product!

View full answer

deckeryan · 2023-02-03T15:10:19Z

deckeryan
Feb 3, 2023
Author

I should probably give a bit more information. I have been running a distributed install for about 6 months without any issues. I upgraded to 2.3.200 yesterday but i noticed the issue before.

0 replies

deckeryan · 2023-02-03T15:42:32Z

deckeryan
Feb 3, 2023
Author

Here is some more info:

Security Onion version as seen in the lower left corner of SOC and in /etc/soversion. 2.3.200

Is this a cloud deployment or on-prem? On-prem with internet access

Did you install from our Security Onion ISO image or did you perform a network installation? Security Onion iso

How many nodes do you have? 7

What are the hardware specs of each of those nodes?'

How are each of those nodes configured? (ex. manager with 2 search nodes and 3 forward nodes)
One manager vm with 4 search nodes and two forward nodes.

Are you experiencing issues monitoring network traffic? If so, are you sniffing from a tap or span port and what is the traffic volume? No

Does so-status show all services running? Yes

Do you get any failures when you run sudo salt-call state.highstate? Yes

Does the SOC Grid page show any failures? No

Explain your issue. Salt highstate fails on master with issues related to playbookdb

I found this in the mysqld log:

2023-02-03T14:26:12.206442Z 17 [Note] Aborted connection 17 to db: 'fleet' user: 'fleetdbuser' host: '172.17.0.1' (Got an error reading communication packets)

That is from right after a restart of the master node.

0 replies

deckeryan · 2023-02-07T15:50:31Z

deckeryan
Feb 7, 2023
Author

After running soup again my errors have disappeared. Thank you for making such a great product!

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

playbookdbuser issues #9716

Uh oh!

{{title}}

Uh oh!

Replies: 3 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

playbookdbuser issues #9716

Uh oh!

deckeryan Feb 3, 2023

Replies: 3 comments

Uh oh!

deckeryan Feb 3, 2023 Author

Uh oh!

Uh oh!

deckeryan Feb 3, 2023 Author

Uh oh!

deckeryan Feb 7, 2023 Author

deckeryan
Feb 3, 2023

deckeryan
Feb 3, 2023
Author

deckeryan
Feb 3, 2023
Author

deckeryan
Feb 7, 2023
Author