Pants+Packaging: Add makeself_archive for each pack to facilitate installation

[cognifloyd]

Background

Pack install

There are several packs that get installed with st2 (Installing via the deb/rpm is the only official way to get these):

• chatops
• core
• default
• linux
• packs

Unlike exchange packs, these 5 packs are not git repos, so if anyone makes a change (for shame 😉), there isn't a simple way to revert the pack without reinstalling the st2 rpm/deb.

There are 3 other packs in the st2 repo:

• examples
  • The examples pack gets installed under /usr/share/doc/st2/examples. The official way to install that pack is to copy it manually: https://docs.stackstorm.com/start.html#start-deploy-examples
• hello_st2
  • The hello_st2 pack is only for a docs tutorial on creating packs, so there is no official way to install it: https://docs.stackstorm.com/reference/packs.html#creating-your-first-pack
• debug
  • The debug pack is a much more recent pack added in Add a debug contrib pack #5155, but it is not mentioned in the docs and there is no official install method.

Pantsbuild + nFPM

Once we start building rpm/deb packages with pants + nfpm, we will need to enumerate all of the files that go in the rpm/deb packages. Pants runs nfpm in a sandbox, and pants does not have a way to populate the sandbox with file owner, file group, file permissions (except the execute bit which is also the only thing git can persist), or file modification times. nFPM has a feature that allows it to include a tree of files, but that pulls the file owner, file group, file permissions and file modification times from the tree of files, so pants does not expose that nFPM feature. Instead, each file--with all of its file attributes--must be explicitly defined in BUILD metadata.

For most of the files in our rpm/deb file, that pants+nfpm limitation is a non-issue; we were already defining those explicitly in the deb control files and rpm spec files. The big exception to that is packs. Listing each file in a pack sounds very onerous and error prone. All of the pack files have uniform file attributes (except for the execute bit, but everything has a way to deal with that difference).

Solution: Pants + Makeself

So, I looked for a way to package an archive of each pack's contents, and then include that archive in the rpm/deb. During installation, the post-install (or similar) scriptlet would be responsible for unpacking those archives under /opt/stackstorm/packs.

Someone in the pants community slack pointed me towards a nifty tool that pants has an integration for: makeself. This tool makes it relatively simple to build self-extracting archives with very few runtime deps. These are the relevant pants docs about it:

• https://www.pantsbuild.org/stable/docs/shell/self-extractable-archives
• https://www.pantsbuild.org/stable/reference/targets/makeself_archive

Packaging the 8 in-repo st2 packs via pants makeself_archive targets has several benefits:

• The rpm/deb needs to include only 1 file for each pack: the makeself archive. So, we only need to add nfpm metadata for the archive itself, not each file in its contents. The archive itself handles ownership/permissions of its contents.
• People could also use these archives to re/install packs if something happens. For example, reinstalling a pack after a user inadvertently edits one of the workflows via the workflow composer.

So, this PR adds a makeself_archive target in pants BUILD metadata for each of the packs under contrib/. Since these all need to be built the same, I created the st2_pack_archive macro in pants-plugins/macros.py (see the docs about macros).

The first step in using makeself is enabling the pants makeself backend:

st2/pants.toml

Lines 30 to 31 in 940fb13

	# packaging
	"pants.backend.experimental.makeself",

Macro Name Note: All of our other macros are prefixed with st2_ and targets defined in actual plugins (not macros) do not have that prefix. So, this uses the st2_ prefix as well.

The simpleast usage of the st2_pack_archive macro is for the default pack:

st2/contrib/default/BUILD

Lines 7 to 11 in 940fb13

	st2_pack_archive(
	dependencies=[
	":metadata",
	],
	)

The :metadata target is our pack_metadata target (implemented by pants-plugins/pack_metadata).

Markdown files in packs

So far, pack_metadata did not include markdown files--which should be distributed with the pack--so I added **/*.md to the default sources list:

st2/pants-plugins/pack_metadata/target_types.py

Line 129 in 940fb13

"**/*.md", # including README.md, HISTORY.md

And registered markdown files as the new resource type pack_doc here:

st2/pants-plugins/pack_metadata/target_types.py

Lines 90 to 91 in 940fb13

	if path.suffix == ".md":
	return PackContentResourceTypes.pack_doc

st2_pack_archive usage

The chatops pack has non-metadata files (python actions) and test files that need to be included in the installed pack. So, we register these as deps here:

st2/contrib/chatops/BUILD

Lines 7 to 13 in 940fb13

	st2_pack_archive(
	dependencies=[
	":metadata",
	"./actions",
	"./tests",
	],
	)

This, however caused a visibility error because tests have __dependents_rules__ that block anything outside of the tests directory from depending on the tests. So, I added an exception for the :files target (which is created in the st2_pack_archive macro and is described further below) like this:

st2/contrib/chatops/tests/BUILD

Lines 1 to 2 in 940fb13

	# tests can only be dependencies of other tests in this directory
	__dependents_rules__(("*", "/**", f"//{build_file_dir().parent}:files", "!*"))

In this, build_file_dir() returns a pathlib.Path object that we can use to reference the parent directory. We have to use this instead of ../ because pants does not support relative parent paths in dependencies. So, the // anchors the address at the root of the repo, and then build_file_dir().parent picks the pack's directory (the parent of tests) without hardcoding the absolute path. This should make any future refactoring more seamless as fewer places need to be updated if a pack dir is moved/renamed.

The core pack also needs to include its requirements.txt file (Which is owned by the python_requirements target named :reqs), as well as a shell script in actions (./actions pulls in the python, and ./actions/send_mail:send_mail_resources pulls in the shell script).

st2/contrib/core/BUILD

Lines 21 to 29 in 940fb13

	st2_pack_archive(
	dependencies=[
	":metadata",
	":reqs",
	"./actions",
	"./actions/send_mail:send_mail_resources",
	"./tests",
	],
	)

Unlike the core pack, the hello_st2 and examples packs do not have python_requirements targets. And, we don't want pants to pull requirements from that file as having the same requirement in multiple requirements files (in this case requirements-pants.txt and the pack requirements.txt file) causes problems for pants dep inferrence. So, I had to remove these requirements files from the pants ignore list (in pants.toml) and add a files target (instead of a python_requirements target) to own that file:

st2/contrib/hello_st2/BUILD

Lines 7 to 12 in 940fb13

	# Capture the requirements file for distribution in the pack archive;
	# we do not need to use `python_requirements()` for this sample file.
	files(
	name="reqs",
	sources=["requirements*.txt"],
	)

st2/contrib/examples/BUILD

Lines 22 to 26 in 940fb13

	# Also capture the requirements file for distribution in the pack archive.
	files(
	name="reqs",
	sources=["requirements*.txt"],
	)

The examples pack has the most complex usage of st2_pack_archive because, sadly, transitive deps do not get included, so I had to list a bunch of deps:

st2/contrib/examples/BUILD

Lines 28 to 46 in 940fb13

	st2_pack_archive(
	# we need to list targets of all files because transitive dep targets are NOT included
	dependencies=[
	":metadata",
	":reqs",
	"./actions",
	"./actions:shell",
	"./actions/bash_exit_code",
	"./actions/bash_ping",
	"./actions/bash_random",
	"./actions/pythonactions",
	"./actions/ubuntu_pkg_info",
	"./actions/ubuntu_pkg_info/lib",
	"./actions/windows",
	"./lib",
	"./sensors",
	"./tests",
	],
	)

The other packs are similar.

st2_pack_archive implementation

st2_pack_archive is a macro function:

st2/pants-plugins/macros.py

Lines 120 to 124 in 940fb13

	def st2_pack_archive(**kwargs):
	"""Create a makeself_archive using files from the given dependencies.
	This macro should be used in the same BUILD file as the pack_metadata target.
	"""

First we get the directory of the BUILD file and return without creating any targets if the pack is in st2tests/. This is important because the contrib/core pack is symlinked under st2tests, and we only want to create an archive for the pack once.

st2/pants-plugins/macros.py

Lines 125 to 129 in 940fb13

	build_file_path = build_file_dir() # noqa: F821
	if "st2tests" == build_file_path.parts[0]:
	# avoid creating duplicate archive for the core pack
	# which is also located under st2tests/st2tests/fixtures/packs
	return

There are many packs in st2tests with pack_metadata targets, but they do not need pack archives for distribution. That's why I kept pack_metadata separate from st2_pack_archive. It's a very good thing that pants macros do not have to create targets, allowing us to return early in this case.

Next, the macro ensures that a dep on :metadata is always present, even if we don't explicitly provide it in the BUILD file that uses st2_pack_archive. I did explicitly include :metadata, so this is a safety mechanism for future development where we might add a new pack.

st2/pants-plugins/macros.py

Lines 132 to 134 in 940fb13

	dependencies = kwargs.pop("dependencies", [])
	if ":metadata" not in dependencies:
	dependencies = [":metadata", *dependencies]

:files target

I mentioned the :files target above. One of the quirks of makeself_archive targets is that they only include files targets. So anything that is python_sources, shell_sources, resources, or other targets will be silently excluded from the makeself archive. So, I needed a way to treat all these targets--all of the targets listed in dependencies--as files targets. There are some experimental pants targets that allow for treating targets as if they were python_sources (experimental_wrap_as_python_sources) or resources (experimental_wrap_as_resources). But there is no experimental_wrap_as_files target, so I needed a different way to capture the dependencies as files. I used shell_command to do this:

st2/pants-plugins/macros.py

Lines 136 to 143 in 940fb13

	# This is basically a "wrap_as_files" target (which does not exist yet)
	shell_command( # noqa: F821
	name="files",
	execution_dependencies=dependencies,
	command="true",
	output_directories=["."],
	root_output_directory=".",
	)

The command true serves as a "noop" of sorts. The key pieces of this are execution_dependencies which causes all of the deps to go in this noop command's sandbox, then output_directories=["."] says to capture all output files in the BUILD file's directory in the sandbox, and root_output_directory="." makes pants strip the BUILD file directory so that all of the files in the pack end up at the root of the makeself archive.

Finally, we build the makeself_archive with a convenient label that gets printed when the archive extracts itself:

st2/pants-plugins/macros.py

Lines 147 to 149 in 940fb13

	makeself_archive( # noqa: F821
	name="archive",
	label=f"{pack_name} StackStorm pack",

Next we include all of the pack files by putting :files in files:

st2/pants-plugins/macros.py

Lines 150 to 153 in 940fb13

	files=[
	":files", # archive contents
	"//:license", # LICENSE file included in archive header, excluded from contents
	],

I also included the LICENSE file so that the makeself archive can prompt the user to accept the license before unpacking. I configured that makeself feature here with --license:

st2/pants-plugins/macros.py

Lines 154 to 157 in 940fb13

	args=( # see: https://makeself.io/#usage
	# Makeself expects '--arg value' (space) not '--arg=value' (equals) for cmdline
	"--license",
	"__archive/LICENSE",

Then, we pre-configure the archive so it will extract itself under /opt/stackstorm/packs:

st2/pants-plugins/macros.py

Lines 158 to 159 in 940fb13

	"--target",
	f"/opt/stackstorm/packs/{pack_name}",

And we configure the pack file permissions/attributes using args that get passed through to tar. Note that st2-packages uses owner root and group st2packs for everything in /opt/stackstorm/packs (1, 2, 3 and 4), but this uses root as the group temporarily. a follow-up PR will handle adding the st2packs group. We also set all timestamps to the value of MTIME so that the build is repeatable - no matter when we build an archive from a given commit, it will always generate exactly the same archive byte-for-byte.

st2/pants-plugins/macros.py

Lines 164 to 168 in 1a66cef

	# reproducibility flags:
	"--tar-extra", # extra tar args: '--arg=value' (equals delimited) space separated
	f"--owner=root --group={ST2_PACKS_GROUP} --mtime={MTIME} --exclude=LICENSE",
	"--packaging-date",
	MTIME,

ST2_PACKS_GROUPS is defined here. There was some logic in st2-packages that attempted to pull the user/group from st2.conf (here), but that code was not actually used anywhere (there's even a comment saying "NOT USED!"), so the group name is effectively hardcoded as st2packs (1, 2, 3). Plus, we're already hard-coding the group in the systemd files (here and here), so I went with hard-coding it here:

st2/pants-plugins/macros.py

Lines 119 to 120 in 1a66cef

	# These are used for system packages (rpm/deb)
	ST2_PACKS_GROUP = "st2packs"

MTIME is defined here, using the same value used by pex (thus, this will be the timestamp of /opt/stackstorm/st2 once unpacked using the pex added in #6307):

st2/pants-plugins/macros.py

Lines 115 to 117 in 940fb13

	# Default copied from PEX (which uses zipfile standard MS-DOS epoch).
	# https://github.com/pex-tool/pex/blob/v2.1.137/pex/common.py#L39-L45
	MTIME = "1980-01-01T00:00:00Z"

Using a makeself_archive

These examples use the chatops pack, but could just as easily use any of the other 7 packs.

To package the chatops pack run:

pants package contrib/chatops::

Or to be more precise, you could use the target name like this

pants package contrib/chatops:archive

Then, you can run the pack to install it (/opt/stackstorm/packs should already exist):

sudo dist/packs/chatops.tgz.run

Or to test it with a temp directory:

sudo dist/packs/chatops.tgz.run --target /tmp/chatops-pack

This is how the post-install deb/rpm script will unpack the pack archive (packaging scriptlets run as root):

PAGER=cat /path/to/chatops.tgz.run --quiet --accept

The license gets piped through the PAGER before allowing the user to accept the license, even if auto accept is on the command line, using PAGER=cat ensures this does not pause the automated install.

And, to populate /usr/share/docs/st2/examples using the archive, the post-install script would do:

PAGER=cat /path/to/examples.tgz.run --quiet --accept --target /usr/share/docs/st2/examples

[cognifloyd]

Suggested change

	f"--owner=root --group={ST2_PACKS_GROUP} --mtime={MTIME} --exclude=LICENSE",
	f"--owner=root --group={ST2_PACKS_GROUP} --mtime={MTIME}",

Should the LICENSE file be installed with the pack like this? If we install it, it will end up as /opt/StackStorm/packs/<pack>/LICENSE even though the LICENSE file is not in the pack directory in our git repo.

[nzlosh]

The pack licence is that of st2, so there's no need to install a copy in each pack directory in my opinion.