[BUG] Salt-formula don't render pillar from imported yaml map file

[schirka]

Your setup

Formula commit hash / release tag

1.8.5

Versions reports (master & minion)

salt-master:

Salt Version:
          Salt: 3002.6
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: 0.6.4
     gitpython: 1.0.1
        Jinja2: 3.0.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.10.1
        pygit2: Not Installed
        Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
  python-gnupg: Not Installed
        PyYAML: 5.4.1
         PyZMQ: 22.1.0
         smmap: 0.9.0
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: centos 7 Core
        locale: UTF-8
       machine: x86_64
       release: 3.10.0-1160.25.1.el7.x86_64
        system: Linux
       version: CentOS Linux 7 Core

Salt-minion

Salt Version:
          Salt: 3002.6
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.0.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.10.1
        pygit2: Not Installed
        Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
  python-gnupg: Not Installed
        PyYAML: 5.4.1
         PyZMQ: 22.1.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: centos 7 Core
        locale: UTF-8
       machine: x86_64
       release: 3.10.0-1160.25.1.el7.x86_64
        system: Linux
       version: CentOS Linux 7 Core

Pillar / config used

/etc/salt/master

gitfs_provider: gitpython
gitfs_remotes:
  - https://github.com/saltstack-formulas/salt-formula

pillar_roots:
  base:
  - /srv/pillar
---

Bug details

Describe the bug

salt['pillar.get'] is not rendered by import_yaml

map.jinja(line 6,7.8)

...
{% import_yaml "salt/osfamilymap.yaml" as osfamilymap %}
{% import_yaml "salt/osmap.yaml" as osmap %}
{% import_yaml "salt/osfingermap.yaml" as osfingermap %}
...

<osfamilymap.yaml(line 5)>
...
{%- set py_ver_repr = salt['pillar.get']('salt:py_ver', '') %}
...


<osmap.yaml(line 5)>
...
{%- set py_ver_repr = salt['pillar.get']('salt:py_ver', '') %}
...

<osfingermap.yaml(line 5)>
...
{%- set py_ver_repr = salt['pillar.get']('salt:py_ver', '') %}
...

Steps to reproduce the bug

 $ salt-call state.apply salt.minion -l debug

> local:
>     Data failed to compile:
> ----------
>     Rendering SLS 'base:salt.minion' failed: Jinja variable 'salt' is undefined
> /var/cache/salt/minion/files/base/salt/osfamilymap.yaml(5):
> ---
> # -*- coding: utf-8 -*-
> # vim: ft=yaml
> ---
> 
> {%- set py_ver_repr = salt['pillar.get']('salt:py_ver', '') %}    <======================
> 
> {%- set osrelease = salt['grains.get']('osrelease', '') %}
> {%- set salt_release = salt['pillar.get']('salt:release', 'latest') %}
> {%- if salt_release.split('.')|length >= 3 %}
> {%-   set salt_release = 'archive/' ~ salt_release %}
> [...]
> ---
> ----------

Expected behaviour

Expect the formula to render

Attempts to fix the bug

For salt['pillar.get'] to be rendered by import_yaml, it needs with context

map.jinja(line 6,7.8)

...
{% import_yaml "salt/osfamilymap.yaml" as osfamilymap with context %}
{% import_yaml "salt/osmap.yaml" as osmap with context %}
{% import_yaml "salt/osfingermap.yaml" as osfingermap with context %}
...

Additional context

salt-formula is now rendering properly.

[baby-gnu]

There is an issue with the use of Jinja2 3.0 saltstack/salt#60188.

[schirka]

Thanks @baby-gnu that helped a lot.

We've ended up downgrading jinja2 after the salt bootstrap in vagrant and terraform provisionner:
pip3 install jinja2==2.11.3

[baby-gnu]

I think we could close this bug report, right?

[kevkha]

I have a similar import_yaml issue on opensuse leap 15.4 minion running the same salt 3002.6 version as @schirka. I tried downgrading jinja2==2.11.3 and jinja2==2.8.1 matching with jinja2 version from its master but could not get passed the error.

master:

# salt --versions-report
Salt Version:
          Salt: 3002.6

Dependency Versions:
          cffi: Not Installed
      cherrypy: unknown
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.8.1
       libgit2: Not Installed
      M2Crypto: 0.33.0
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
  python-gnupg: Not Installed
        PyYAML: 3.11
         PyZMQ: 17.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.1.4

System Versions:
          dist: centos 7 Core
        locale: UTF-8
       machine: x86_64
       release: 3.10.0-1160.42.2.el7.x86_64
        system: Linux
       version: CentOS Linux 7 Core

minion:

# salt --versions-report
Salt Version:
          Salt: 3002.6

Dependency Versions:
          cffi: 1.13.2
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.0.3
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.5
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.17
      pycrypto: Not Installed
  pycryptodome: 3.17
        pygit2: Not Installed
        Python: 3.6.15 (default, Sep 23 2021, 15:41:43) [GCC]
  python-gnupg: Not Installed
        PyYAML: 6.0
         PyZMQ: 25.0.2
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: opensuse 15.4
        locale: UTF-8
       machine: x86_64
       release: 5.14.21-150400.24.33-default
        system: Linux
       version: openSUSE Leap 15.4

map.jinja:

{## Start imports ##}
{% import_yaml "salt/defaults.yaml" as defaults %}
{% import_yaml "salt/osfamilymap.yaml" as osfamilymap %}
{% import_yaml "salt/osmap.yaml" as osmap %}

state.apply error:

----------
    Rendering SLS 'base:salt.master' failed: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/jinja2/debug.py", line 162, in fake_traceback
    exec(code, globals, locals)
  File "/var/cache/salt/minion/files/base/salt/osfamilymap.yaml", line 5, in top-level template code
    {% set ospyver = salt['grains.filter_by'](ospyvermap, salt['grains.get']('os_family')) or {} %}
  File "/usr/lib/python3.6/site-packages/jinja2/sandbox.py", line 303, in getitem
    #: .. versionadded:: 2.6
jinja2.exceptions.UndefinedError: 'salt' is undefined

I have tested downgrading Jinja2 version starting from 3.0.3 down to 2.8.1 without success. Here is a list from opensuse minion

(from versions: 2.0rc1, 2.0, 2.1, 2.1.1, 2.2, 2.2.1, 2.3, 2.3.1, 2.4, 2.4.1, 2.5, 2.5.1, 2.5.2, 2.5.3, 2.5.4, 2.5.5, 2.6, 2.7, 2.7.1, 2.7.2, 2.7.3, 2.8, 2.8.1, 2.9, 2.9.1, 2.9.2, 2.9.3, 2.9.4, 2.9.5, 2.9.6, 2.10, 2.10.1, 2.10.2, 2.10.3, 2.11.0, 2.11.1, 2.11.2, 2.11.3, 3.0.0a1, 3.0.0rc1, 3.0.0rc2, 3.0.0, 3.0.1, 3.0.2, 3.0.3)

I have tried to match PyYAML version 3.11 on the minion too without success.

Besides Jinja2 and PyYAML package what else should I try to overcome this error? Any suggestion is greatly appreciated. Thanks in advance.

[kevkha]

I figured it out. Adding with context to each import_yaml line in map.jinja resolved my issue and it works with Jinja2 3.0.3.

{## Start imports ##}
{% import_yaml "salt/defaults.yaml" as defaults with context %}
{% import_yaml "salt/osfamilymap.yaml" as osfamilymap with context %}
{% import_yaml "salt/osmap.yaml" as osmap with context %}