New Reactor Upload fails for some host

[joni1993]

A Host of mine outputs a lot of data and the reactor uploaded fails on this

2019-10-30 23:34:55,563 [salt.utils.reactor:106 ][ERROR   ][30823] Failed to render "/srv/salt/foreman_report_upload.sls":
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/renderers/yaml.py", line 65, in render
    data = yamlloader.load(yaml_data, Loader=get_yaml_loader(argline))
  File "/usr/lib/python3/dist-packages/salt/utils/yamlloader.py", line 170, in load
    return yaml.load(stream, Loader=Loader)
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 72, in load
    return loader.get_single_data()
  File "/usr/lib/python3/dist-packages/yaml/constructor.py", line 35, in get_single_data
    node = self.get_single_node()
  File "ext/_yaml.pyx", line 707, in _yaml.CParser.get_single_node (ext/_yaml.c:9612)
  File "ext/_yaml.pyx", line 725, in _yaml.CParser._compose_document (ext/_yaml.c:9922)
  File "ext/_yaml.pyx", line 776, in _yaml.CParser._compose_node (ext/_yaml.c:10814)
  File "ext/_yaml.pyx", line 890, in _yaml.CParser._compose_mapping_node (ext/_yaml.c:12609)
  File "ext/_yaml.pyx", line 776, in _yaml.CParser._compose_node (ext/_yaml.c:10814)
  File "ext/_yaml.pyx", line 890, in _yaml.CParser._compose_mapping_node (ext/_yaml.c:12609)
  File "ext/_yaml.pyx", line 774, in _yaml.CParser._compose_node (ext/_yaml.c:10784)
  File "ext/_yaml.pyx", line 851, in _yaml.CParser._compose_sequence_node (ext/_yaml.c:12011)
  File "ext/_yaml.pyx", line 776, in _yaml.CParser._compose_node (ext/_yaml.c:10814)
  File "ext/_yaml.pyx", line 890, in _yaml.CParser._compose_mapping_node (ext/_yaml.c:12609)
  File "ext/_yaml.pyx", line 774, in _yaml.CParser._compose_node (ext/_yaml.c:10784)
  File "ext/_yaml.pyx", line 851, in _yaml.CParser._compose_sequence_node (ext/_yaml.c:12011)
  File "ext/_yaml.pyx", line 776, in _yaml.CParser._compose_node (ext/_yaml.c:10814)
  File "ext/_yaml.pyx", line 892, in _yaml.CParser._compose_mapping_node (ext/_yaml.c:12639)
  File "ext/_yaml.pyx", line 905, in _yaml.CParser._parse_next_event (ext/_yaml.c:12818)
yaml.parser.ParserError: while parsing a block mapping
  in "<unicode string>", line 5, column 10
did not find expected key
  in "<unicode string>", line 5, column 2131

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/reactor.py", line 97, in render_reaction
    data=data)
  File "/usr/lib/python3/dist-packages/salt/state.py", line 385, in render_template
    **kwargs)
  File "/usr/lib/python3/dist-packages/salt/template.py", line 101, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)
  File "/usr/lib/python3/dist-packages/salt/renderers/yaml.py", line 71, in render
    raise SaltRenderError(exc)
salt.exceptions.SaltRenderError: while parsing a block mapping
  in "<unicode string>", line 5, column 10
did not find expected key
  in "<unicode string>", line 5, column 2131

[sbernhard]

I guess, this might be a salt issue itself @joni1993 ?

[sbernhard]

Do you have an update on this @joni1993 ?

[joni1993]

I'm currently installing a new foreman installation and will check this when I'm ready - installation without puppet is still a little tedious 😒

[joni1993]

Running into another issue this time, but also related to report uploading:

2020-06-22 19:16:40,538 [py.warnings      :99  ][WARNING ][16735] /var/cache/salt/master/extmods/runners/foreman_report_upload.py:102: DeprecationWarning: decodestring() is a deprecated alias since Python 3.1, use decodebytes()
  report = create_report(base64.decodestring(highstate))

2020-06-22 19:16:40,538 [salt.loaded.ext.runners.foreman_report_upload:105 ][ERROR   ][16735] Exception encountered: expected bytes-like object, not str

Edit: I am also required to set Require SSL for smart proxies to False in Foreman, when using /usr/sbin/upload-salt-reports and salt-api without SSL (because of cherrypi problems) - why is this required even tough the foreman-proxy is connected via SSL? - besides this problem /usr/sbin/upload-salt-reports works fine

[joni1993]

#55 fixed above issue

Same uploads worked, but one seems to fail:

Traceback (most recent call last):
  File "/usr/sbin/upload-salt-reports", line 156, in <module>
    upload(jobs_to_upload())
  File "/usr/sbin/upload-salt-reports", line 124, in upload
    for job_id, job in jobs:
  File "/usr/sbin/upload-salt-reports", line 101, in jobs_to_upload
    yield job_id, get_job(job_id)
  File "/usr/sbin/upload-salt-reports", line 44, in get_job
    for key, entry in value.items():
AttributeError: 'list' object has no attribute 'items'

[nadjaheitmann]

Can we close this issue due to inactivity?