Packaging: Add conf files to deb/rpm packages #6330

cognifloyd · 2025-04-11T00:19:58Z

This PR is working towards doing packaging via pantsbuild. Eventually, I hope to archive and stop using st2-packages.git.

Previous PRs in this series:

The pants nfpm backend was activated in Packaging: Inject st2 version into rpm/deb packages (pants-plugins/release) #6321.
Basic BUILD metadata for deb/rpm packages (nfpm_deb_package and nfpm_rpm_package) was added in Packaging: Add basic BUILD metadata for rpm/deb packages #6325 along with the deb maintainer scripts and rpm scriptlets.
Packaging: Add systemd files to rpm/deb packages #6326 added the systemd files (units and generators) to the deb/rpm packages
Packaging: Add directories and symlinks to deb/rpm packages #6327 added directories and symlinks to the deb/rpm packages
Packaging: Add venv and packs to deb/rpm packages #6328 added venv (pex file) and pack archives to the deb/rpm packages
Packaging: add dep on crudini & regenerate lockfiles/st2.lock #6329 Add crudini pex_binary (needed for this PR)

This PR focuses on adding the conf files installed by this st2-packages.git Makefile:
https://github.com/StackStorm/st2-packages/blob/d4d2d8dfdf1c88412e5d58635adb87da9c671952/packages/st2/Makefile#L66-L104

`shell_command` targets

For the most part, adding the conf files to deb/rpm packages is fairly straightforward. But, in a few cases, the old Makefile made changes to the files on the fly. So, this PR adds shell_command BUILD targets to make those edits.

`/etc/st2/htpasswd`

The /etc/st2/htpasswd file was generated on the fly like this:

install -m640 /dev/null $(DESTDIR)/etc/st2/htpasswd

I used touch to do effectively the same thing with a shell_command.

st2/conf/BUILD

Lines 61 to 67 in ae649f9

    
           shell_command( 
        
               name="htpasswd", 
        
               description="An empty htpasswd file for st2-auth-backend-flat-file", 
        
               command="touch htpasswd", 
        
               tools=["touch"], 
        
               output_files=["htpasswd"], 
        
           )

`/etc/st2/st2.conf`

The /etc/st2/conf file had virtualenv_opts modified on the fly like this:

ifeq ($(REDHAT),1)
	if [[ `$(PYTHON_ALT_BINARY) /root/scripts/platform_major_version.py` = 8 ]]; then sed -i -r 's/virtualenv_opts\s*=.*/virtualenv_opts =/' $(DESTDIR)/etc/st2/st2.conf; fi
endif

I used crudini to do something similar with a shell_command.

st2/conf/BUILD

Lines 69 to 76 in ae649f9

    
           shell_command( 
        
               name="packaged_st2_conf", 
        
               execution_dependencies=[":st2_package_conf"], 
        
               # virtualenv_opts is no longer needed for most OSes. We used to do this only for EL 8. 
        
               command="""crudini --verbose --set st2.package.conf actionrunner virtualenv_opts ''""", 
        
               runnable_dependencies=["//:crudini"], 
        
               output_files=["st2.package.conf"], 
        
           )

To simplify, I make this modification for all OSes. If any of them fail when we test the built packages, we can revisit making this OS-specific. Here are some historical references about the os-specific virtualenv_opts:

`/etc/st2/logging.conf` and `/etc/st2/syslog.conf`

The logging conf files were modified to change the log directory and the default log level like this:

sed -i -r "/args\s*=\s*/s%logs%/var/log/st2%g" $(DESTDIR)/etc/st2/logging.*conf
sed -i "/\[logger_root\]/,/\[.*\]\|\s*$$/ {s/level=DEBUG/level=INFO/}" $(DESTDIR)/etc/st2/logging.*conf
sed -i "/\[logger_root\]/,/\[.*\]\|\s*$$/ {s/level=DEBUG/level=INFO/}" $(DESTDIR)/etc/st2/syslog.*conf

This PR uses a portable sed command to make the log directory change, and uses crudini to change the log level in a reliable and portable way:

st2/pants-plugins/macros.py

Lines 234 to 247 in ae649f9

    
               shell_command(  # noqa: F821 
        
                   name="package_logging_conf", 
        
                   execution_dependencies=deps, 
        
                   # Using "-E" and specifying the ".bak" suffix makes this portable 
        
                   command=""" 
        
                   sed -E -i.bak "/args[[:space:]]*=[[:space:]]*/s:logs/:/var/log/st2/:g" logging.*conf; 
        
                   for conf_file in logging.*conf syslog.*conf; do 
        
                       crudini --verbose --set "${conf_file}" logger_root level INFO; 
        
                   done 
        
                   """, 
        
                   runnable_dependencies=["//:crudini"], 
        
                   tools=["sed"], 
        
                   output_files=["*.conf"], 
        
               )

This shell_command is in a new st2_logging_conf_for_nfpm macro so that it was fairly simple to run it on all the logging conf files.

This was in the st2-packages.git Makefile's post_install target.

To keep this portable so that it runs with all versions of sed, avoid features that are only in gnu sed (like \s). And use crudini instead of sed for more straightforward edits.

When testing the packages, we should discover any non-root processes that rely on access to st2.conf. Hopefully giving them access will be as simple as switching the group to ST2_SVC_USER. Otherwise, we might need to revert this change and make st2.conf world readable. Also note that ST2 now supports passing secrets in env vars. So, people could theoretically include the secrets in systemd conf files that are only accessible by root. If any utils, like st2ctl, need access to those secrets, however, they will need to get them from somewhere else if they are not in st2.conf.

cognifloyd added enhancement st2-packages logging nginx pantsbuild labels Apr 11, 2025

cognifloyd added this to the pants milestone Apr 11, 2025

cognifloyd self-assigned this Apr 11, 2025

pull-request-size bot added the size/L PR that changes 100-499 lines. Requires some effort to review. label Apr 11, 2025

cognifloyd mentioned this pull request Apr 11, 2025

Packaging: add dep on crudini & regenerate lockfiles/st2.lock #6329

Merged

Base automatically changed from crudini to master April 13, 2025 16:58

cognifloyd added 9 commits April 14, 2025 09:09

Packaging: add pants metadata for conf files

11cac0f

Packaging: Add /etc/st2/htpasswd

3e954be

Packaging: Add /usr/share/doc/st2/nginx/st2.conf

363f4b0

Packaging: Add pre-packaging conf manipulation

a98e144

This was in the st2-packages.git Makefile's post_install target.

Packaging: make conf file editing as portable as possible

6fcaa15

To keep this portable so that it runs with all versions of sed, avoid features that are only in gnu sed (like \s). And use crudini instead of sed for more straightforward edits.

Packaging: stub command to modify st2.package.conf

b6b33b7

packaging: use crudini to modify st2.package.conf

a673cb7

update changelog entry

1ad65e4

cognifloyd force-pushed the packaging-nfpm_contents-conf branch from 11e8b1d to ae649f9 Compare April 14, 2025 15:18

cognifloyd marked this pull request as ready for review April 14, 2025 15:20

cognifloyd requested review from a team, amanda11, guzzijones, nzlosh and winem April 14, 2025 15:21

guzzijones approved these changes Apr 25, 2025

View reviewed changes

nzlosh approved these changes Apr 28, 2025

View reviewed changes

cognifloyd merged commit 0ff136d into master Apr 28, 2025
113 of 115 checks passed

cognifloyd deleted the packaging-nfpm_contents-conf branch April 28, 2025 13:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Packaging: Add conf files to deb/rpm packages #6330

Packaging: Add conf files to deb/rpm packages #6330

cognifloyd commented Apr 11, 2025 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	shell_command(
	name="htpasswd",
	description="An empty htpasswd file for st2-auth-backend-flat-file",
	command="touch htpasswd",
	tools=["touch"],
	output_files=["htpasswd"],
	)

	shell_command(
	name="packaged_st2_conf",
	execution_dependencies=[":st2_package_conf"],
	# virtualenv_opts is no longer needed for most OSes. We used to do this only for EL 8.
	command="""crudini --verbose --set st2.package.conf actionrunner virtualenv_opts ''""",
	runnable_dependencies=["//:crudini"],
	output_files=["st2.package.conf"],
	)

	shell_command( # noqa: F821
	name="package_logging_conf",
	execution_dependencies=deps,
	# Using "-E" and specifying the ".bak" suffix makes this portable
	command="""
	sed -E -i.bak "/args[[:space:]]=[[:space:]]/s:logs/:/var/log/st2/:g" logging.*conf;
	for conf_file in logging.conf syslog.conf; do
	crudini --verbose --set "${conf_file}" logger_root level INFO;
	done
	""",
	runnable_dependencies=["//:crudini"],
	tools=["sed"],
	output_files=["*.conf"],
	)

Uh oh!

Packaging: Add conf files to deb/rpm packages #6330

Packaging: Add conf files to deb/rpm packages #6330

Conversation

cognifloyd commented Apr 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

shell_command targets

/etc/st2/htpasswd

/etc/st2/st2.conf

/etc/st2/logging.*conf and /etc/st2/syslog.*conf

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cognifloyd commented Apr 11, 2025 •

edited

Loading

`shell_command` targets

`/etc/st2/htpasswd`

`/etc/st2/st2.conf`

`/etc/st2/logging.conf` and `/etc/st2/syslog.conf`