Make Utility Modules accessible from the py pillar renderer

[boltronics]

I have a bunch of pillar files using the pure Python renderer (#!py). Some had large sections of code that were similar, so I broke them out into Utility Modules.

After hours of trial and error, I determined that the only options to make this work was to either:

1. Wrap the utility module functions in an execution module. Not ideal, kind of defeats the point of having utility modules.
2. Migrate my pillar file data over to an External Pillar module. Problematic, because it prevents other renderers I'm using from being applied over the top. I have to migrate those renders to operate as external pillar modules as well, and stack those last.
3. Deploy a new custom renderer that does the same thing that the built-in py renderer does, but includes utils support.

I've tried all three options, and I think I like option 3 the best. I eventually came up with this:

salt/_renderers/pyutils.py

# -*-Python-*-
"""Python Renderer for Salt with __utils__ support

Based on salt/renderers/py.py.
"""

import os

import salt.loader
import salt.utils.templates
from salt.exceptions import SaltRenderError


def render(template, saltenv="base", sls="", tmplpath=None, **kws):
    """Render the python module's components with __utils__ support"""
    template = tmplpath
    if not os.path.isfile(template):
        raise SaltRenderError(f"Template {template} is not a file!")

    __utils__ = salt.loader.LazyLoader(
        salt.loader._module_dirs(__opts__, "utils"),
        __opts__,
        tag="utils",
        whitelist=None,
    )

    tmp_data = salt.utils.templates.py(
        template,
        True,
        __salt__=__salt__,
        salt=__salt__,
        __grains__=__grains__,
        grains=__grains__,
        __opts__=__opts__,
        opts=__opts__,
        __pillar__=__pillar__,
        pillar=__pillar__,
        __utils__=__utils__,
        utils=__utils__,
        __env__=saltenv,
        saltenv=saltenv,
        __sls__=sls,
        sls=sls,
        **kws,
    )
    if not tmp_data.get("result", False):
        raise SaltRenderError(
            tmp_data.get("data", "Unknown render error in py renderer")
        )

    return tmp_data["data"]

It appears to do the job, but ideally the built-in "py" renderer would support accessing __utils__ out-of-the-box.

I'm wondering if there is any reason why utility module access isn't included by default, or if its omission was just an oversight?

[whytewolf]

because utils is horrible. it has caused the biggest performance problems of any of the subsystems. as well as caused the most broken loops of all time. it is not recommended to use utils within modules Let alone renderers. this is why it also isn't supported by saltstack extensions. it isn't used in any of the render engines. it isn't a oversight.

the proper thing to do is just include the salt utility modules directly.

I know @s0undt3ch was in the process of rewritting things to the point that it could be decommend and has ALWAYS said don't use it. but because it was wrapped in so many things it wasn't going to be finished before the big decomm of modules. and then broadcom happened. and that project was left half finished.

[boltronics]

Thanks @whytewolf.

I should point out that there seems to be nothing in the documentation that says there is any problem using it, or any official suggestion to use something else. I also cannot find any open bug reports related to the issues you have mentioned. As a long-time user of Salt but someone new to making use of utility modules, I don't think such problems are obvious.

Not that I'm doubting you, of course. Perhaps the real issue here is that the docs could use an update?

the proper thing to do is just include the salt utility modules directly.

This works for me. For some reason it didn't work when I tried it yesterday. I must have had some other problem with my setup, since I wouldn't have went down this rabbit hole otherwise. 😅

It would be great to have a note in the docs pointing out that they can be called via the Python renderer in this way. Then I would have spent more time specifically focused on trying to get that approach working.

[whytewolf]

saltstack/salt-enhancement-proposals#66