Refactor SPIRE Integration and Optimize Spire Memory Footprint by gngram · Pull Request #1837 · tiiuae/ghaf

gngram · 2026-03-18T14:39:24Z

Description of Changes

This PR refactors SPIRE (formerly referred as SPIFFE module) server and agent implementation and reduces reduces the memory footprint of the SPIRE binaries.

Includes an option to use custom spire package.
A custom spire package(spire-min) is overlay-ed by stripping of unused cloud-specific modules to reduce memory foot print.
standardized naming conventions by renaming remaining spiffe references to spire.
Server health report published, server health is checked before initiating agent to avoid failure when server is not ready.
implemented a generic agent registration flow based on the specific node attestation method used by each VM.
Updated common configurations to allow the server to dynamically retrieve VM lists, per-VM workloads, and attestation methods.
per-VM workload registration, earlier there was no separation of workloads.
single global config to enable or disable the spiffe identity issuance using spire

Type of Change

New Feature
Bug Fix
Improvement / Refactor

Related Issues / Tickets

Checklist

Clear summary in PR description
Detailed and meaningful commit message(s)
Commits are logically organized and squashed if appropriate
Contribution guidelines followed
Ghaf documentation updated with the commit - https://tiiuae.github.io/ghaf/
Author has run make-checks and it passes
All automatic GitHub Action checks pass - see actions
Author has added reviewers and removed PR draft status

Testing Instructions

Applicable Targets

Installation Method

Requires full re-installation
Can be updated with nixos-rebuild ... switch
Other:

Test Steps To Verify:

check the status of spire-agent service in each vm(except admin), it should be running and without any error.
check the status of spire-server service in admin vm, it should be running and without any error.

milva-unikie · 2026-03-19T07:03:38Z

spire-publish-bundle.service and x509pop-key-setup.service fail in admin-vm on all Orins that we test (native Orin-AGX, cross-compiled Orin-AGX, native Orin-NX, cross-compiled Orin-NX)

spire-publish-bundle.service:

Mar 18 15:37:53 admin-vm spire-publish-bundle[703]: ERROR: SPIRE server socket not found at /tmp/spire-server/private/api.sock
Mar 18 15:37:53 admin-vm systemd[1]: spire-publish-bundle.service: Main process exited, code=exited, status=1/FAILURE
Mar 18 15:37:53 admin-vm systemd[1]: spire-publish-bundle.service: Failed with result 'exit-code'.
Mar 18 15:37:53 admin-vm systemd[1]: Failed to start Publish SPIRE trust bundle (PoC).
Mar 18 15:37:53 admin-vm systemd[1]: spire-publish-bundle.service: Consumed 1.975s CPU time, 5.7M memory peak.

x509pop-key-setup.service has no logs.

These issues should be fixed and the code reviewed before we start manual testing.

kajusnau · 2026-03-19T07:48:01Z

As a general note, is there some overlapping functionality with what NixOS/nixpkgs#481447 introduced recently? Maybe we could reduce the footprint of the module(s) on our side if we can re-use some of the upstream services.spire namespace.

kajusnau

Some basic comments, also suggest using getExe throughout the patch, since it's already used in some places but not all.

modules/common/security/spire/agent.nix

modules/common/security/spire/server.nix

modules/common/common.nix

kajusnau · 2026-03-19T08:31:24Z

Maybe we should put a hold on this until after #1799 and #1822 are in

vunnyso · 2026-03-19T08:41:46Z

overlays/custom-packages/spire-mini/0001-removed-cloud-infra-to-reduce-memory-footprint.patch

+From c00cf4f1dd5a3fedd7dc701828d476efe1f09e4a Mon Sep 17 00:00:00 2001
+From: Ganga Ram <Ganga.Ram@tii.ae>
+Date: Tue, 17 Feb 2026 12:46:42 +0400
+Subject: [PATCH] removed cloud infra to reduce memory footprint


Can we can try to push this patch to upstream?

It's hack basically to remove some builtin plugins. I thought to create a upstream patch but it requires major restructuring. They have used common files in different modules to manage plugins but since Go doesn't provides conditional compilation of code segments within a file, so I will have to split those files for each plugin in order to use Go tags. Which is least likely they will accept also there will be so many tags manage.

Have you compared the memory footprint difference after applying this patch? If it's a hacky patch, then at least an upstream issue could be created to gather ideas and suggestions for improvement.

Could we also consider creating minimal profile files that exclude the required plugins?

gngram · 2026-03-19T08:52:11Z

As a general note, is there some overlapping functionality with what NixOS/nixpkgs#481447 introduced recently? Maybe we could reduce the footprint of the module(s) on our side if we can re-use some of the upstream services.spire namespace.

@kajusnau Thanks, pointing to this, server and agent configurations we can use from this.
I will use their agent and server config as it is and will separately handle token generation, bundle publishing, credential sharing, workload registration etc. When we'll rebase to the nixpkgs, we'll simply remove the file and rename the config.

gngram · 2026-03-21T11:28:42Z

Rebased, and now using spire services from nixpkgs

…s Publish bundle/join-tokens and allow non-root Workload API access via /run/spire tmpfiles. Co-authored-by: shamma-alblooshi1 <shamma.alblooshi@tii.ae> Signed-off-by: Everton de Matos <everton.dematos@tii.ae>

- Binary size reduced by 40% - Also reduces memory footprint Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

Signed-off-by: Everton de Matos <everton.dematos@tii.ae>

- spire server gets following information from the VMs using the common config 1) list of VMs running spire agents 2) workloads per vm 3) node attestation method per vm - also server address and port information is shared thorugh common config. Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

- all cloud related and other unused modules stripped Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

- using spire agent and server services from nixpkgs - per vm workload registration - fine grained spire server and agent configuration - generic agent registartion based on node attestation method - agent and server synchronization - spiffe module renamed to spire Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

everton-dematos · 2026-03-25T13:33:37Z

modules/microvm/sysvms/adminvm-base.nix

    security = {
      fail2ban.enable = globalConfig.development.ssh.daemon.enable or false;
      audit.enable = lib.mkDefault (globalConfig.security.audit.enable or false);
+


In the case that the admin-vm will also have workloads, shouldn't it have an spire agent as well?

gngram requested a review from brianmcgillion March 18, 2026 14:39

gngram temporarily deployed to internal March 18, 2026 14:39 — with GitHub Actions Inactive

gngram requested review from kajusnau and vunnyso March 18, 2026 14:39

gngram temporarily deployed to internal March 18, 2026 14:39 — with GitHub Actions Inactive

gngram requested review from everton-dematos, mbssrc, shamma-alblooshi1 and vadika March 18, 2026 14:39

gngram added the Needs Testing CI Team to pre-verify label Mar 18, 2026

milva-unikie added bug on Orin AGX Issues found on NVIDIA Jetson AGX Orin while checking this PR bug on Orin NX Issues found on NVIDIA Jetson NX Orin while checking this PR and removed Needs Testing CI Team to pre-verify labels Mar 19, 2026

kajusnau reviewed Mar 19, 2026

View reviewed changes

vunnyso reviewed Mar 19, 2026

View reviewed changes

gngram force-pushed the pull/spire_module_update branch from 58b371e to 49eba96 Compare March 21, 2026 11:27

gngram temporarily deployed to internal March 21, 2026 11:27 — with GitHub Actions Inactive

gngram force-pushed the pull/spire_module_update branch from 49eba96 to bbbbc0f Compare March 22, 2026 14:32

gngram temporarily deployed to internal March 22, 2026 14:32 — with GitHub Actions Inactive

gngram temporarily deployed to internal March 22, 2026 15:03 — with GitHub Actions Inactive

gngram force-pushed the pull/spire_module_update branch from 722dd76 to 13bf900 Compare March 23, 2026 12:16

gngram temporarily deployed to internal March 23, 2026 12:16 — with GitHub Actions Inactive

everton-dematos and others added 7 commits March 24, 2026 18:06

security(spiffe): add SPIRE server/agent modules and enable across VM…

ffbed31

…s Publish bundle/join-tokens and allow non-root Workload API access via /run/spire tmpfiles. Co-authored-by: shamma-alblooshi1 <shamma.alblooshi@tii.ae> Signed-off-by: Everton de Matos <everton.dematos@tii.ae>

remove cloud node and workload attestation from spire

6fb22ed

- Binary size reduced by 40% - Also reduces memory footprint Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

Enable spire patch and fix Agent ExecStart

96a231b

Signed-off-by: Everton de Matos <everton.dematos@tii.ae>

stripped spire package to reduce memory foot print

0ee16e4

- all cloud related and other unused modules stripped Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

update common configs for agents and server

c824c63

Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

gngram force-pushed the pull/spire_module_update branch from 13bf900 to c824c63 Compare March 25, 2026 06:25

gngram temporarily deployed to internal March 25, 2026 06:25 — with GitHub Actions Inactive

gngram changed the title ~~Refactor SPIRE Integration, Enable x509pop Attestation, and Optimize Footprint~~ Refactor SPIRE Integration and Optimize Spire Memory Footprint Mar 25, 2026

gngram mentioned this pull request Mar 25, 2026

SPIRE x509pop attestation #1853

Open

19 tasks

everton-dematos reviewed Mar 25, 2026

View reviewed changes

Conversation

gngram commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of Changes

Type of Change

Related Issues / Tickets

Checklist

Testing Instructions

Applicable Targets

Installation Method

Test Steps To Verify:

Uh oh!

milva-unikie commented Mar 19, 2026

Uh oh!

kajusnau commented Mar 19, 2026

Uh oh!

kajusnau left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kajusnau commented Mar 19, 2026

Uh oh!

vunnyso Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

gngram Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

vunnyso Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

gngram commented Mar 19, 2026

Uh oh!

gngram commented Mar 21, 2026

Uh oh!

everton-dematos Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gngram commented Mar 18, 2026 •

edited

Loading

everton-dematos Mar 25, 2026 •

edited

Loading