[BUG] Salt minion is not caching all the formula's files on windows machines

[fgionghi]

Your setup

Formula commit hash / release tag

Release tag: 0.14.0

Versions reports (master & minion)

Salt master:

Salt Version:
          Salt: 3003.3
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 2.0.6
     gitpython: 3.0.7
        Jinja2: 2.10.1
       libgit2: 0.28.3
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.6.1
        pygit2: 1.0.3
        Python: 3.8.10 (default, Jun  2 2021, 10:49:15)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 18.1.1
         smmap: 2.0.5
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2
 
System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.8.0-59-generic
        system: Linux
       version: Ubuntu 20.04 focal

Salt minion:

Salt Version:
          Salt: 3003.3

Dependency Versions:
          cffi: 1.14.5
      cherrypy: 18.6.0
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.5
     gitpython: 3.1.13
        Jinja2: 2.11.3
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: 1.1.4
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
        Python: 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)]
  python-gnupg: 0.4.6
        PyYAML: 5.4.1
         PyZMQ: 18.0.1
         smmap: 3.0.4
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.1

System Versions:
          dist:
        locale: cp1252
       machine: AMD64
       release: 10
        system: Windows
       version: 10 10.0.19041 SP0

Pillar / config used

packages:
  chocolatey:
    wanted:
      prometheus-windows-exporter.install:
        version: '0.16.0'

---

Bug details

Describe the bug

When I run the command salt skyros.pc.itc.it state.test packages.chocolatey, I get the error:

skyros.pc.itc.it:                                                                                         
    Data failed to compile:                                                                               
----------                                                                                                
    Rendering SLS 'base:packages.chocolatey' failed: Jinja error: ./map.jinja                             
Traceback (most recent call last):                                                                        
  File "c:\salt\bin\lib\site-packages\salt-3003.3-py3.7.egg\salt\utils\templates.py", line 500, in render_
jinja_tmpl                                                                                                
    output = template.render(**decoded_context)                                                           
  File "c:\salt\bin\lib\site-packages\jinja2\environment.py", line 1090, in render                        
    self.environment.handle_exception()                                                                   
  File "c:\salt\bin\lib\site-packages\jinja2\environment.py", line 832, in handle_exception               
    reraise(*rewrite_traceback_stack(source=source))                                                      
  File "c:\salt\bin\lib\site-packages\jinja2\_compat.py", line 28, in reraise                             
    raise value.with_traceback(tb)                                                                        
  File "<template>", line 3, in top-level template code                                                   
  File "c:\salt\bin\lib\site-packages\salt-3003.3-py3.7.egg\salt\utils\jinja.py", line 198, in get_source 
    raise TemplateNotFound(template)                                                                      
jinja2.exceptions.TemplateNotFound: ./map.jinja                                                           
                                                                                                          
; line 3                                                                                                  
                                                                                                          
---                                                                                                       
# -*- coding: utf-8 -*-                                                                                   
# vim: ft=sls                                                                                             
{%- from "./map.jinja" import packages with context %}    <======================                         
                                                                                                          
{%- if grains['os'] == 'Windows' %}                                                                       
                                                                                                          
  {%- if packages.chocolatey %}                                                                           
    {%- set req_states = packages.chocolatey.required.states %}                                           
[...]                                                                                                     
---                                                                                                       
ERROR: Minions returned with non-zero exit code

and in the cache folder on the target (C:\salt\var\cache\salt\minion\files\base\packages) the only file is chocolately.sls without any map.jinja

We tried with different version of salt-minion, with different windows version (win10, win server 2016/2019) and with two different salt-master (same version of salt-master).

When I run salt skyros.pc.itc.it state.test packages in the cache folder on the target machine I have more files (for example the map.jinja) but not all files (the default.yaml is missing). In fact I get multiple errors regarding the default.yaml file, like this:

----------                                                                                                
    Rendering SLS 'base:packages.golang.goget' failed: Jinja error: ./defaults.yaml                       
c:\salt\var\cache\salt\minion\files\base\packages/map.jinja(4):                                           
---                                                                                                       
# -*- coding: utf-8 -*-                                                                                   
# vim: ft=jinja                                                                                           
                                                                                                          
{%- import_yaml './defaults.yaml' as defaults %}    <======================                               
{%- import_yaml './osfamilymap.yaml' as osfamilymap %}                                                    
{%- import_yaml './osmap.yaml' as osmap %}                                                                
{%- import_yaml './osfingermap.yaml' as osfingermap %}                                                    
                                                                                                          
{%- set packages = salt['grains.filter_by'](                                                              
[...]                                                                                                     
---                                                                                                       
Traceback (most recent call last):                                                                        
  File "c:\salt\bin\lib\site-packages\salt-3003.3-py3.7.egg\salt\utils\templates.py", line 500, in render_
jinja_tmpl                                                                                                
    output = template.render(**decoded_context)                                                           
  File "c:\salt\bin\lib\site-packages\jinja2\environment.py", line 1090, in render                        
    self.environment.handle_exception()                                                                   
  File "c:\salt\bin\lib\site-packages\jinja2\environment.py", line 832, in handle_exception               
    reraise(*rewrite_traceback_stack(source=source))                                                      
  File "c:\salt\bin\lib\site-packages\jinja2\_compat.py", line 28, in reraise                             
    raise value.with_traceback(tb)                                                                        
  File "<template>", line 6, in top-level template code                                                   
  File "c:\salt\bin\lib\site-packages\jinja2\environment.py", line 1155, in make_module                   
    return TemplateModule(self, self.new_context(vars, shared, locals))                                   
  File "c:\salt\bin\lib\site-packages\jinja2\environment.py", line 1238, in __init__                      
    body_stream = list(template.root_render_func(context))                                                
  File "c:\salt\var\cache\salt\minion\files\base\packages/map.jinja", line 4, in top-level template code  
    {%- import_yaml './defaults.yaml' as defaults %}                                                      
  File "c:\salt\bin\lib\site-packages\salt-3003.3-py3.7.egg\salt\utils\jinja.py", line 198, in get_source 
    raise TemplateNotFound(template)                                                                      
jinja2.exceptions.TemplateNotFound: ./defaults.yaml

Expected behaviour

It should install the package prometheus-windows-exporter using chocolatey.

Attempts to fix the bug

I submitted a PR that seems to fix the problem: #77

[fgionghi]

Additional tests

We tried without chocolatey, so using directly winrepo. My pillar now is:

packages:
  pkgs:
    wanted:
      - prometheus-wmi-exporter

but the problem persist.
Pay attention to delete all the cached file on target machine.

[myii]

@fgionghi This is a tough one to reproduce. I used the pillar you provided, with a slight addition:

 packages:
   chocolatey:
+    required:
+      pkgs: []
     wanted:
       prometheus-windows-exporter.install:
         version: '0.16.0'

Tested this in the Windows CI we have for this formula and all of them are successfully installing prometheus-wmi-exporter. The failures shown are actually because I didn't also update the packages that the tests are looking for:

• https://github.com/myii/packages-formula/actions/runs/1289828884 (Vagrant)
  • Windows 8.1
  • Windows 10
• https://github.com/myii/packages-formula/actions/runs/1289828886 (GitHub's Windows runners)
  • Windows 2016
  • Windows 2019

I also raised this issue in the last Formulas working group meeting but no-one had faced this problem, even though they are working with Windows minions.

I reckon the best way to move forward would be to ask a question in the #windows channel in SaltStack's Slack instance. From there, we can hopefully work out where to look next, or if this is actually a bug with Salt itself. Because it's not really specific to this formula, after all.

Let me know how you intend to proceed.

[myii]

@fgionghi I've gone ahead and added a comment in the #windows channel:

• https://saltstackcommunity.slack.com/archives/C7U9FLK8S/p1632809119091600

Other useful links that might come in handy when attempting to resolve this:

• refactor: Use relative state include #74 -- relative state includes introduced to this formula.
• Support Jinja imports with relative paths. Issue #13889 saltstack/salt#47490 -- PR where this feature was introduced into Salt (Support Jinja imports with relative paths).

[fgionghi]

We did a test formula called "winpackages":
init.sls:

{% from './map.jinja' import packages with context %}
{% for pkg in packages %}
Choco_package_{{ pkg|upper }}_install:
 chocolatey.installed:
     - name: {{ pkg }}
{% endfor %}

map.jinja:

{% set default_packages = salt['grains.filter_by']({
    'Windows': {
        'pkgs': [
           'googlechrome',
      ]
    },
}) %}
{% set packages = salt['pillar.get']('winpackages:pkgs', default=default_packages.pkgs, merge=True) %}

pillar.sls

winpackages:
  pkgs:
    - prometheus-windows-exporter.install

and this is not working, always giving me ./map.jinja not found.

Then we modify the init.sls like this:
init.sls:

{% from 'winpackages/map.jinja' import packages with context %}
{% for pkg in packages %}
Choco_package_{{ pkg|upper }}_install:
 chocolatey.installed:
     - name: {{ pkg }}
{% endfor %}

and this is working.
Our theory is: salt minion is searching ./map.jinja not inside the formula folder, but on the parent folder (C:\salt\var\cache\salt\minion\files\base).

BUT, we did another test:

• I first use the working method (importing using winpackages/map.jinja)
• so I now have all the formula's files chached on the minion (in C:\salt\var\cache\salt\minion\files\base\winpackages we have both map.jinja and init.sls)
• if then we use the first method (importing using ./map.jinja) is now working.
  So our theory is wrong: the salt-minion is not searching in the parent formula (at least when the formula's files are cached).

Could it be that we have some config in the salt-master that broke this relative path thing (only on windows minions)?
The config of the salt-minion is very simple: master: our-salt-master.our-domain.com

Our test will fail only if C:\salt\var\cache\salt\minion\files\base is empty. We use salt skyros.pc.itc.it saltutil.clear_cache.

Sorry if we are not precise, if you need further information let me know.

[mvivaldi]

DISCLAIMER: I am a colleague of @fgionghi
I tried in my installation at home the "winpackages" from the last post and I have the same error. But we are using ubuntu 20.04 in all of our tests, could it be something related to the distro? I tried to downgrade the salt-master to the 3001.8+ds-1 version, but nothing changed.
Maybe we are doing something wrong and we can't see it, someone could try to do the same thing with an ubuntu 20.04 master and verify if the problem is here?
Thank you

[myii]

Thanks for the further info, @mvivaldi. I've now been able to reproduce this bug by connecting the Windows 10 VirtualBox VM that we use for testing this formula to my Salt master (on Ubuntu 20.04):

packages-formula/kitchen.vagrant.yml

Lines 21 to 37 in 7c1ca7c

	- name: windows-10-latest-py3
	driver:
	box: techneg/win10x64-pro-salt
	cache_directory: "/omnibus/cache"
	customize: {}
	ssh: {}
	provisioner:
	# yamllint disable rule:line-length
	init_environment: |
	# Workaround to allow `kitchen converge` to be used multiple times
	# without having to `kitchen destroy` first: remove state files cached by
	# Salt during the previous `converge` (if present)
	rm -recurse `
	C:\Users\vagrant\AppData\Local\Temp\kitchen\var\cache\salt\minion\files\base `
	-ErrorAction SilentlyContinue
	exit 0
	# yamllint enable rule:line-length

Traceback:

DESKTOP-MG2P5NV:
    Data failed to compile:
----------
    Rendering SLS 'base:packages.chocolatey' failed: Jinja error: ./defaults.yaml
c:\salt\var\cache\salt\minion\files\base\packages\map.jinja(4):
---
# -*- coding: utf-8 -*-
# vim: ft=jinja

{%- import_yaml './defaults.yaml' as defaults %}    <======================
{%- import_yaml './osfamilymap.yaml' as osfamilymap %}
{%- import_yaml './osmap.yaml' as osmap %}
{%- import_yaml './osfingermap.yaml' as osfingermap %}

{%- set packages = salt['grains.filter_by'](
[...]
---
Traceback (most recent call last):
  File "c:\salt\bin\lib\site-packages\salt-3003-py3.7.egg\salt\utils\templates.py", line 497, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "c:\salt\bin\lib\site-packages\jinja2\asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "c:\salt\bin\lib\site-packages\jinja2\environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "c:\salt\bin\lib\site-packages\jinja2\environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "c:\salt\bin\lib\site-packages\jinja2\_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 3, in top-level template code
  File "c:\salt\bin\lib\site-packages\jinja2\environment.py", line 1073, in make_module
    return TemplateModule(self, self.new_context(vars, shared, locals))
  File "c:\salt\bin\lib\site-packages\jinja2\environment.py", line 1152, in __init__
    body_stream = list(template.root_render_func(context))
  File "c:\salt\var\cache\salt\minion\files\base\packages\map.jinja", line 4, in top-level template code
    {%- import_yaml './defaults.yaml' as defaults %}
  File "c:\salt\bin\lib\site-packages\salt-3003-py3.7.egg\salt\utils\jinja.py", line 198, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: ./defaults.yaml


ERROR: Minions returned with non-zero exit code

When I get a bit more time, I will test this on Windows 8 as well.

Ultimately, this appears to be a bug in Salt itself, so we'll need to report this to them with all the info we can figure out. It would be useful to test with a Salt master other than Ubuntu 20.04 as well.

[myii]

Steps to reproduce:

1. $ KITCHEN_LOCAL_YAML=kitchen.vagrant.yml bin/kitchen create windows-windows-10-latest-py3
2. Access the instance using VirtualBox.
3. Edit the hosts file to add the relevant entry for salt (the master's IP address).
4. Restart the salt-minion service.
5. $ salt-key -a DESKTOP-MG2P5NV
6. $ salt -Cv 'DESKTOP-MG2P5NV' state.sls packages.chocolatey test=True
7. (Traceback is generated).
8. $ KITCHEN_LOCAL_YAML=kitchen.vagrant.yml bin/kitchen destroy windows-windows-10-latest-py3
9. $ salt-key -d DESKTOP-MG2P5NV

---

@dafyddj If you've got any additional ideas, they'd be welcome.

[myii]

I've added another comment in Slack, if anyone is following it there:

• https://saltstackcommunity.slack.com/archives/C7U9FLK8S/p1633973606132800?thread_ts=1632984505.101200&cid=C7U9FLK8S

[myii]

@fgionghi @mvivaldi I've also reproduced the error on our Windows 8.1 box. I even tried different versions of Salt but didn't make any progress. Perhaps relative includes have never worked on Windows, in a master-minion set up.

It's time for this to be sent through as a bug report to the Salt repo itself. Most of the content you both have supplied here is useful to provide in the report there. I'll be happy to join in the discussion there as well -- please link it back to this issue.

In terms of this formula, the team will have to discuss how to proceed. @saltstack-formulas/wg Since relative includes aren't working on Windows, should we revert to using tplroot? Or is there some other method that we should consider. @ypid-geberit Your feedback will also be useful, since this is going to affect the changes you provided in #74. Could we find a new method, that uses only tpldir instead of tplroot?

[fgionghi]

Here the issue we have opened on the Salt repo: saltstack/salt#61040 .
Thank you.

[myii]

Thanks, @fgionghi -- I will add additional comments there.