Q for modernized, resilient, self-healing SaltStack for Dummies?

[davidbernat]

tags: third-party package community?, wikipedia for bash commands?

Hello SaltStackers. I am newly testing SaltStack for my LAN system, and am quite excited for its authenticated state checking through SSH, and its persistent session management for executing remote commands through SSH (beyond the scope of this discussion). I am a technical programmer with a few years of Ubuntu experience, and focusing evaluating SaltStack in the modern domain of LLM-enabled server system management for naive users. I also found the SaltStack documentation to be cripplingly complicated, and I sympathize with the massive challenge of upkeeping a system designed for up to tens of thousands of different machines. It is very impressive, and my use case is narrow, and specific, for the purpose of this conversation that I hope we will have routinely.

HomeOS
The SaltStack system is more or less setup on the Ubuntu servers with one Master and multiple Minions. Check.

So I am writing my first state (sls) file:

• check that Ubuntu is the OS
• confirm or create a lengthy list of directories that I want
• confirm or install Miniconda
• create Miniconda environment ML
• confirm pip installed
• install several pip packages

I would love to dive into more complex issues, such as ensuring NVIDIA drivers are adequate, and running diagnostics for performance issues, and sweeping regular logs which may or may not be processed with ML. But for the sake of discussion the ultra-simple series of steps is sufficient to describe what I want to discuss here.

SaltStack has a large number of functions which do certain behaviors, i.e., check the existence of a directory, and this is a core value proposition of SaltStack. SaltStack ensures that, in this instance, file.directory knows all the necessary logic of checking, creating, and confirming the existence of a directory, and that this knowledge is encoded for any of the supported operating systems, not just Ubuntu.

This directly segues into why I am bringing this discussion to you.
I also used an auto-code agent to read Context7 & SaltStack docs to assist me in installation and creation of the .sls to evaluate how well understood the modern LLM technologies are, and how effective the existing docs were. I read the documents extensively myself for several hours before starting and throughout to repair various errors.

To cut to the chase, some of those above operations have SaltStack operations, some need to be written and maintained by me to get SaltStack to work:

So I am writing my first state (sls) file:

• check that Ubuntu is the OS ==> requires custom bash
• confirm or create a lengthy list of directories that I want ==> file.directory
• confirm or install Miniconda ==> requires custom bash (which is an error of the auto-coder, c.f. pkg.installed)
• create Miniconda environment ML ==> requires custom bash
• confirm pip installed ==> requires custom bash
• install several pip packages ==> pkg.installed

Now, I do not want to get into the weeks as to whether "confirm pip installed" is actually handled under the hood by pkg.installed in this discussion, and an auto-coder could not figure out how to confirm pkg.installed should handle the Miniconda installation directly. I did notice that the auto-code agents repeatedly failed to know of specialty module functions on SaltStack, to its own deficit, but that is not necessarily crucial to this discussion right now.

This discussion is about maintenance and placement of those custom bash scripts.

Model
In the LLM age, modularity, explained-ability, confirmability, and rewrite-ability take on extremely high value.

For instance, installing Miniconda is a few lines of bash that easily went into the cmd.run.name field. But clearly, these should each exist outside the .sls file in their own .sh file, so that future readability and modularity is available. Adding descriptive comments to what the bash script is supposed to do, how its intended to operate, and what methods of research were conducted, all written in human language, greatly improve readability for novice engineers, but also allow future LLM revisions to jump in to a shallow pond of transfer knowledge before interpreting the code, including for updating the bash script (as, say Miniconda moves its installation curl'd files).

This is, in the perspective of my company, StarlightHQ, an absolute game changer for systems management, and backbones SaltStack, but I cannot be alone in this perspective among the power users of SaltStack who are LLM enabled, or interested in community standards or AI. It is absolutely clear to me that a dedicated repository should begin to exist which offers the validated, standardized, trusted third-party bash scripts for pseudo-atomic actions, and I simply do not have the platform to do this for myself. I can already see SaltStack doing a tremendous infrastructure hardiness improvement to what my day-to-day is already managing mediocre so far.

An excellent example is ensuring the NVIDIA drivers are appropriately updated on each of the machines, as this can be completely riddled with bugs, from their own installation pages. But setting up a closed-loop process that:

• has previous working example, with version control
• deployment locally within protected droplet environment (i.e. daytona)
• confirmable measures of success (i.e. nvidia-nmi succeeds)
• refreshing of atomic installation instructions updated into version control
• integrates into SaltStack state files in a way I can at a glance and by LLM engage with for inspection

is Fabulous. Many of these steps can be constructed to be automatic and voice/LLM activated, too.

It is just about the community, the authentication of authority, and the containment systems.

I know that to some SaltStack veterans and sys-admin lifers this smacks of another naive community manager, hooked up to another LLM in an over-warm afternoon. But I really do not think that it is, and prompts, for the first time in a long time, a new gateway to the OS. Imagine the power of a library of thousands of confirmed scripts.

But also, a regimented pattern for plugging in to SaltStack, a simple generated DAG, and a conversant UI that you can sit down with and say: I want SaltStack to manage Conda installation of itself ==> and a card shows up with the confirmed script in your .sls ==> and its packages ==> and a card shows up with the confirmed script. I know that with the proper rigidity and limitations on the system this can be very robust and fluid to tens of machines. And it does not require domain knowledge outside of the quality committee boards, for now, for security reasons.

Would love to have conversations about this in my inbox. And thanks to anyone who can help me with specific improvements to my state files. These are obviously very modern design patterns to a very important backbone.

There are a lot of users out there.

Two examples of a bash script that would make Saul Teukolsky (c.f. Recipes in C) blush at these soft kids these days :)

#!/bin/bash
# =============================================================================
# author: starlight.ai
# maintainer: starlight.ai
# date: 2026-02-12
# version: v0.0.3
# =============================================================================
# version history:
# v0.0.1 ==> initial commit inside .sls file
# v0.0.2 ==> move bash command to this file
# v0.0.3 ==> mild housekeeping for public publishing
# =============================================================================

# purpose:
# This bash command initiates a complete installation of Miniconda3 (the minimal
# Anaconda Python distribution) from the official Anaconda repository for
# Linux x86_64 architecture. It downloads the latest Miniconda3 installer,
# executes it in batch mode (-b) to install to the user's home directory,
# and adds the conda binary to the user's PATH via .bashrc.

# generated:
# generated using tacit embedded knowledge of auto-code LLM
# by analyzing the standard Miniconda installation process and condensing
# it into an idempotent bash script suitable for Salt state execution.

# how this works:
# 1. Check if conda is already in PATH using 'command -v conda'
# 2. If NOT found, download the latest Miniconda3 installer from Anaconda
# 3. Execute the installer in batch mode (-b) to auto-accept license
# 4. Install to $HOME/miniconda3 (user's home directory)
# 5. Clean up the installer script after execution
# 6. Append conda binary to PATH in .bashrc for future sessions
# The 'unless' Salt state parameter makes this idempotent - it only runs
# if conda is NOT already installed.

if ! command -v conda &> /dev/null; then
  echo "Installing Miniconda..."
  cd /tmp
  curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
  bash Miniconda3-latest-Linux-x86_64.sh -b -p $HOME/miniconda3
  rm Miniconda3-latest-Linux-x86_64.sh
  echo 'export PATH="$HOME/miniconda3/bin:$PATH"' >> $HOME/.bashrc
fi

#!/bin/bash
# =============================================================================
# author: starlight.ai
# maintainer: starlight.ai
# date: 2026-02-12
# version: v0.0.3
# =============================================================================
# version history:
# v0.0.1 ==> initial commit inside .sls file
# v0.0.2 ==> move bash command to this file 
# v0.0.3 ==> manual paring down to only conda create
# =============================================================================

# purpose:
# Creates a conda environment named "ML" with Python 3.13 

# generated:
# generated using tacit embedded knowledge of auto-code LLM 
# by analyzing conda environment creation syntax. this merges
# the unless and command functionality through the if statement.

# how this works:
# 1. Sources conda initialization script to enable 'conda' command
# 2. Checks if ML environment already exists using 'conda env list'
# 3. Creates new environment with Python 3.13 if it doesn't exist
# 4. The -y flag auto-accepts all prompts (batch mode)
# 6. The 'require' Salt parameter ensures conda is installed first

source $HOME/miniconda3/etc/profile.d/conda.sh
if ! conda env list | grep -q "^ML "; then
  conda create -n ML python=3.13 -y
fi

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here's some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Salt's Contributor Guide
• Join our Community Discord
• Salt Project YouTube channel
• GitHub Discussions

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject.pdl@broadcom.com. We're glad you've joined our community and look forward to doing awesome things with you!