[BUG] salt.formulas doesn't result in updated file roots.

[gizahNL]

Your setup

Formula commit hash / release tag

Formula commit 5f62905
salt-master/minion: 3004

Versions reports (master & minion)

Salt Version:
          Salt: 3004
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.5
     gitpython: 3.1.14
        Jinja2: 2.11.3
       libgit2: 1.1.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.9.7
        pygit2: 1.4.0
        Python: 3.9.2 (default, Feb 28 2021, 17:03:44)
  python-gnupg: Not Installed
        PyYAML: 5.3.1
         PyZMQ: 20.0.0
         smmap: 4.0.0
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-11-amd64
        system: Linux
       version: Debian GNU/Linux 11 bullseye

Pillar / config used

pillar:

salt:
  master_config_use_TOFS: true
  master:
    pillar_merge_lists: True
    top_file_merging_strategy: same
    state_output: changes
    state_aggregrate: True
    state_events: True
    file_ignore_regex:
      - '/\.svn($|/)'
      - '/\.git($|/)'
include:
  - salt-master.gitfs
  - salt-master.git_pillar
  - salt-master.formulas

pillar formulas sls:

salt_formulas:
  git_opts:
    default:
      baseurl: git@REPO
      basedir: /srv/formulas
      update: True
      options:
        rev: master
        identity: '/etc/salt/git/git.key'
    basedir_opts:
      makedirs: True
      user: root
      group: root
      mode: 755
    checkout_orig_branch: True
  list:
    base:
      - FORMULAS

sls topfile:

base:
  'MASTER:
     - salt.formulas
     - salt.master

---

Bug details

Describe the bug

Formulas get cloned, but their location does not get written out into the master config.

Steps to reproduce the bug

write simple configuration containing formulas, apply via highstate also containing salt.master, see salt-master file_roots not getting updated.

Expected behaviour

file_roots getting updated

[myii]

@gizahNL A few things are going on here, which would take time to describe in full. Are you able to connect to the Formulas channel via. Slack, Matrix, Telegram or IRC (libera.chat)? You can see the links for the first three at the top of this page:

• https://saltstack-formulas.netlify.app

For libera.chat, the channel is #saltstack-formulas.

---

Briefly, the issue is here:

  master_config_use_TOFS: true
  master:
    pillar_merge_lists: True
    top_file_merging_strategy: same
    state_output: changes
    state_aggregrate: True
    state_events: True
    file_ignore_regex:
      - '/\.svn($|/)'
      - '/\.git($|/)'

If you use master_config_use_TOFS: true, then the whole master section is ignored, since you are managing the master configuration file using TOFS. That also means adding formulas yourself to the TOFS-based override templates.

If you set master_config_use_TOFS: false, you'll find that the template updates the file_roots when running the salt.master state (note, not the salt.formulas state). But if you've been using TOFS, this may change your existing files, so try it out with test=True first.

There's more, but that would be easier to cover in the Formulas chat, if you can get there.