TypeError: cannot use a string pattern on a bytes-like object

[heiwu]

Hi,

i tried to get it running on python3-3.6.15-150300.10.27.1 / salt-3004-150400.8.8.1 but get following error:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/pillar/__init__.py", line 896, in render_pstate
    **defaults
  File "/usr/lib/python3.6/site-packages/salt/template.py", line 99, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1203, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1218, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/renderers/jinja.py", line 75, in render
    **kws
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 261, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 548, in render_jinja_tmpl
    "Jinja error: {}{}".format(exc, out), line, tmplstr, trace=tracestr
salt.exceptions.SaltRenderError: Jinja error: cannot use a string pattern on a bytes-like object
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python3.6/site-packages/jinja2/sandbox.py", line 438, in call
    return __context.call(__obj, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1203, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1218, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/passwords/pwgen-salt-module/pwgen.py", line 72, in get_pw
TypeError: cannot use a string pattern on a bytes-like object

; line 1

---
root-pw: {{ salt['pwgen.get_pw'](pw_name='local-root/'+grains['host'], pw_store='/opt/passwords/pass-db', pw_meta_dir='/opt/passwords_metadata') }}    <======================

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
[...]
---
2023-01-20 15:24:45,334 [salt.pillar      :1224][CRITICAL][24779] Pillar render error: Rendering SLS 'root-pw' failed. Please see master log for details.

I think something is wrong with encoding?
System locale is LANG=de_DE.UTF-8

[heiwu]

Before getting to this error i got 2 other error messages. I got rid of them by changing following lines:

line 59:
-         file_times = [ os.path.getmtime(i) for i in pw_file, meta_file ]
+         file_times = [ os.path.getmtime(i) for i in [ pw_file, meta_file ] ]

line 66,67:
-    print pw_meta
-    print pw_file_hash
+    print(pw_meta)
+    print(pw_file_hash)

I really like your idea how to provide system-individual passwords and would really like to implement your solution but my coding skills are not good enough to get further than this.

[clinta]

I no longer run saltstack, so I can't test myself. But please try this branch. And let me know how it works.

https://github.com/clinta/salt-pwgen/blob/python3-fix/pwgen.py

[heiwu]

Running it with the code from your link gives me this error:

2023-01-24 16:46:38,079 [salt.loader.lazy :791 ][ERROR   ][22576] Failed to import module pwgen, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 766, in _load_module
    mod = self.run(spec.loader.load_module)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1203, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1218, in _run_as
    return _func_or_method(*args, **kwargs)
  File "<frozen importlib._bootstrap_external>", line 399, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 823, in load_module
  File "<frozen importlib._bootstrap_external>", line 682, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 684, in _load
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 674, in exec_module
  File "<frozen importlib._bootstrap_external>", line 781, in get_code
  File "<frozen importlib._bootstrap_external>", line 741, in source_to_code
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/passwords/pwgen-salt-module/pwgen.py", line 59
    file_times = [ os.path.getmtime(i) for i in pw_file, meta_file ]
                                                       ^
SyntaxError: invalid syntax

After applying the fixes mentioned above (#8 (comment)) i get this error:

salt.exceptions.SaltRenderError: Jinja error: 'utf-8' codec can't decode byte 0x85 in position 0: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python3.6/site-packages/jinja2/sandbox.py", line 438, in call
    return __context.call(__obj, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1203, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1218, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/passwords/pwgen-salt-module/pwgen.py", line 50, in get_pw
    pw_file_hash = hashlib.sha256(f.read()).hexdigest()
  File "/usr/lib64/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x85 in position 0: invalid start byte

; line 1

---
root-pw: {{ salt['pwgen.get_pw'](pw_name='local-root/'+grains['host'], pw_store='/opt/passwords/pass-db', pw_meta_dir='/opt/passwords_metadata') }}    <======================

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
[...]
---

[clinta]

My mistake, I thought I pushed the fix commit to that branch, but it was unchanged. That branch should have already the fixes you mentioned already incorporated.

Can you try again?

[heiwu]

Hi @clinta, thanks for your help!
Unfortunately there are still bugs. Running the code you pushed yesterday (commit 70b64db) gives me this error:

  File "/opt/passwords/pwgen-salt-module/pwgen.py", line 52, in get_pw
    pw_file_hash = hashlib.sha256(f.read()).hexdigest()
  File "/usr/lib64/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x85 in position 0: invalid start byte

I changed line 51 from with open(pw_file, 'r') as f: to with open(pw_file, 'rb') as f:, that brought me here:

  File "/opt/passwords/pwgen-salt-module/pwgen.py", line 73, in get_pw
    pass_plaintext = ansi_escape.sub('', pass_output).strip().split('\n')[-1]
TypeError: cannot use a string pattern on a bytes-like object

I changed line 73 from pass_plaintext = ansi_escape.sub('', pass_output).strip().split('\n')[-1] to pass_plaintext = ansi_escape.sub('', pass_output.decode('utf-8')).strip().split('\n')[-1], which brought me to:

  File "/opt/passwords/pwgen-salt-module/pwgen.py", line 77, in get_pw
    pw_file_hash = hashlib.sha256(f.read()).hexdigest()
  File "/usr/lib64/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x85 in position 0: invalid start byte

I also tried changing line 41 from ansi_escape = re.compile(r'\x1b[^m]*m') to ansi_escape = re.compile(rb'\x1b[^m]*m'), which gave me:

  File "/opt/passwords/pwgen-salt-module/pwgen.py", line 73, in get_pw
    pass_plaintext = ansi_escape.sub('', pass_output).strip().split('\n')[-1]
TypeError: sequence item 1: expected a bytes-like object, str found

Do you have any more ideas? I have very rudimental python/coding skills (am more sysadmin), so any help is really appreciated!