jinja_trim_blocks and jinja_lstrip_blocks Seem to Break Formulas

[bnied]

Description of Issue/Question

When using our in-house Salt tree, jinja_trim_blocks and jinja_lstrip_blocks both work without issue. However, if I then enable a GitFS-backed Saltstack formula, things start breaking spectacularly.

For example, when trying to use the users-formula, if you have jinja_trim_blocks enabled:

local:
    - Rendering SLS 'base:users' failed: sequence entries are not allowed here; line 2

      ---
      # vim: sts=2 ts=2 sw=2 et ai
      include:  - users.sudo    <======================

      users_new-user_adm_group:
        group.present:
          - name: new-group
      users_new-user_staff_group:
      [...]
      ---

(please note that the dash in the user and group aren't present in the actual user or group)

Alternatively, when jinja_lstrip_blocks is enabled:

local:
    - Rendering SLS 'base:users' failed: while parsing a block mapping
        in "<unicode string>", line 6, column 1:
          include:
          ^
      expected <block end>, but found '-'
        in "<unicode string>", line 51, column 1:
          - uid: 1111
          ^

Removing these two options allow the Salt Formulas to work normally.

Setup

• Have the following present in /etc/salt/master:

jinja_trim_blocks: True
jinja_lstrip_blocks: True

• Also include a Saltstack Formula via Git (I've seen this issue with users and hostsfile for sure):

gitfs_remotes:
  - https://github.com/saltstack-formulas/hostsfile-formula.git
  - https://github.com/saltstack-formulas/users-formula.git

Steps to Reproduce Issue

Run Salt after configuring as noted above.

Versions Report

Salt Version:
           Salt: 2016.3.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.3.0
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
   python-gnupg: 0.3.6
         PyYAML: 3.10
          PyZMQ: 14.0.1
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-91-generic
         system: Linux
        version: Ubuntu 14.04 trusty

Master and all minions are running the same version.

[Ch3LL]

Just to verify its not the sls files in combination with the jinja settings, can you try cloning that repo to your local file_roots and see if it works then versus the gitfs?

I know you state that your local state files work fine but I want to make sure whether the users-formulas states are the issue or gitfs. Thanks

[bnied]

Can do. I'll try it and report back.

[bnied]

So, I tried the following:

• Manual clone to my file root
• Downloading the ZIP copy (in case the git metadata was somehow involved), and moving that to my file root.

In both cases, I get the same errors as above. So GitFS and python-git do not appear to be at fault.

[bnied]

Is there any other information you need?

[bnied]

ping?

[bnied]

@Ch3LL do you need any other info for this issue? Would this be better-suited to be opened against the saltstack-formulas org?

[Ch3LL]

@bnied okay I believe what is going on is related to how jinja handles - in combination with jinja_trim_blocks and jinja_lstrip_blocks. So in the users formula you can see the user of - often for example:

{%- if used_sudo or used_googleauth or used_user_files %}
include:
{%- if used_sudo %}

Accordign to the jinja docs here:

You can also strip whitespace in templates by hand. If you add a minus sign (-) to the start or end of a block (e.g. a For tag), a comment, or a variable expression, the whitespaces before or after that block will be removed.

This whitespace includes newlines

And in combination with those two jinja settings. according to the docs those do the following:

With both trim_blocks and lstrip_blocks enabled, you can put block tags on their own lines, and the entire block line will be removed when rendered, preserving the whitespace of the contents

so I'm thinking this might be expected behavior, but can you please test one thing for me.

Can you clone that repo down and then remove the - in those jinja blocks and see if that resolves the issue?

[bnied]

Can do! I'll try this either today or this weekend, and have feedback back to you by Monday.

[wwalker]

I'm seeing the same problem. We have in our environment set jinja_trim_blocks and jinja_lstrip_blocks to true. At this point it would be very painful to turn them off. I am hoping that there is a way to turn them off on a per file basis so that I can take advantage of the users formula without having to re do all of our templates, etc. that are dependent on that jinja behavior.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[myii]

I believe this issue needs to be reopened.

Here is another example of it breaking formulas (the issue and the PR that resolved it):

• jinja strip white spaces option in salt master config causes a broken pg_hba.conf saltstack-formulas/postgres-formula#241
• Edit pg_hba template to work with jinja 'trim whitespaces' option on salt master saltstack-formulas/postgres-formula#244

Please also note my comment regarding the fix, since it resulted in other side effects.