[cleaner] lower timeout on storeutl when converting certs

[pmoravec]

For some certificates, storeutl command gets stuck when called via Popen. Decrease default 300s timeout to prevent redundant delays.

Closes: #4221

---

Please place an 'X' inside each '[]' to confirm you adhere to our Contributor Guidelines

• Is the commit message split over multiple lines and hard-wrapped at 72 characters?
• Is the subject and message clear and concise?
• Does the subject start with [plugin_name] if submitting a plugin patch or a [section_name] if part of the core sosreport code?
• Does the commit contain a Signed-off-by: First Lastname email@example.com?
• Are any related Issues or existing PRs properly referenced via a Closes (Issue) or Resolved (PR) line?
• Are all passwords or private data gathered by this PR obfuscated?

[packit-as-a-service]

Congratulations! One of the builds has completed. 🍾

You can install the built RPMs by following these steps:

• sudo dnf install -y 'dnf*-command(copr)'
• dnf copr enable packit/sosreport-sos-4221
• And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

[pmoravec]

Trivial reproducer: run cleaner on one (Red Hat?) entitlement certificate PEM file:

mkdir sosreport-ent-cert
cp /etc/pki/entitlement/*pem sosreport-ent-cert
time sos clean --batch sosreport-ent-cert

Execution timeout will be 5m and a few seconds. Most of the time, ps shows timeout 300s openssl storeutl -noout -text -certs command "running".

I tried a few smarter changes like:

• set timeout=None to call the command outside timeout XXXs command
• set sizelimit=0
• set foreground=True

but nothing helped. But if one figures out the root cause and proposes a better solution than my workaround-ish patch, I will appreciate it.

[bmr-cymru]

I'm not familiar with openssl-storeutl at all but this behaviour sounds wonky. Since you can reproduce easily, do you think you might be able to come up with a simple reproducer outside sos? That would be good material for a report to https://github.com/openssl/openssl/issues/new?template=bug_report.md - a single Python script using subprocess.Popen maybe?

Ack to the code change as a workaround - one trivial note - since it is a workaround, perhaps it's worth commenting to that effect:

# Workaround openssl-storeutl delays (#4221)

Then there's less chance of it getting forgotten if/when the OpenSSL problem gets fixed.

[pmoravec]

Commit message reworded.

I forgot to add one another attempt: Popen without our HeadReader (https://github.com/sosreport/sos/blob/main/sos/utilities.py#L354) works fine:

from subprocess import Popen, PIPE

expanded_args=['timeout', '--foreground', '300s', 'openssl', 'storeutl', '-noout', '-text', '-certs', '/etc/pki/entitlement/4498127087553954140.pem']

_output = open('4498127087553954140.pem.text', 'wb')

with Popen(expanded_args, shell=False, stdout=_output,
           stderr=PIPE, bufsize=-1, close_fds=True) as p:
    stdout, stderr = p.communicate()
    print(f"{stdout=}")
#    print(f"{stderr=}")

_output.close()

This terminates swiftly. So some HeadReader/pipe/storeutl mis-communication seems to be the culprit.

It would be great if somebody checks that, either way.

[TurboTurtle]

LGTM.

Has a merge conflict in archive however.

[pmoravec]

OK, here is the minimalistic reproducer outside of sos. It just requires an entitlement certificate granting access to Red Hat repos:

from subprocess import Popen, PIPE

timeout = 300
expanded_args=['openssl', 'storeutl', '-noout', '-text', '-certs', '/etc/pki/entitlement/3405791963348487628.pem']
#expanded_args=['openssl', 'x509', '-in', '/etc/pki/entitlement/3405791963348487628.pem', '-text', '-noout']

with open('3405791963348487628.pem.text', 'w') as _output:  # 'wb' fails the same way

    with Popen(expanded_args, shell=False, stdout=_output,
               stderr=PIPE, bufsize=-1, close_fds=True) as p:
        try:
            p.wait(timeout)
#            stdout, stderr = p.communicate()
#            print(f"{stdout=} {stderr=}")
        except Exception as e:
            print(f"exception {e}, terminating")
            p.terminate()

This script is stuck for 5 minutes (see timeout = 300) in p.wait, despite the openssl storeutl command run in terminal completes in a fraction of a second.

When replacing expanded_args to the commented line, the script completes quickly.

When replacing p.wait(timeout) by the two subsequent commented lines, the script completes quickly as well - even with the original openssl storeutl command triggered.

When replacing stderr=PIPE to some another opened file, or to None or to STDOUT, the script completes quickly as well. The error output is 3071 lines of

004E9EA2467F0000:error:1608010C:STORE routines:ossl_store_handle_load_result:unsupported:crypto/store/store_result.c:162:provider=default

Sadly, we cant use pipesize option of Popen, since that is supported since 3.11 only.

Not sure now where the root cause error is, if Python or openssl. Also not sure if the timeout=10 in my PR is the best workaround, as using stderr=True to redirect the lengthy errors to stdout / to the *.pem.text file sounds better to me, now? (it is fast, just more data to clean and output is bit confusing for users)

What are your thoughts, @TurboTurtle and @pafernanr , if we add stderr ot stdout here? Pros: lowering time from 10s to 0.5s, cons: tons of errors in the *.pem.text file.