Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
6faec5c
Item 02: Add Shibboleth SP (mod_shib) configuration to Keystone Helm …
bbobrov Mar 17, 2026
91611de
Item 04: Add SAML SP key/certificate Kubernetes secret template
bbobrov Mar 17, 2026
d4cf5f7
Item 12: Add Prometheus alert for SAML assertion validation failures
bbobrov Mar 17, 2026
8126a2e
Bump Keystone chart version to 0.13.0 for SAML federation support
bbobrov Mar 17, 2026
2ea6ef7
Fix: Add missing metadataConfigMap default to federation.saml.idp values
bbobrov Mar 17, 2026
4f2ea92
Add noop init container for SAML SP key decryption (SF.Sem.2 pattern)
bbobrov Mar 20, 2026
32279e1
Implement per-tenant SAML SP key pairs for cryptographic tenant isola…
bbobrov Mar 20, 2026
5b1ddbc
Fix: Use 'saml2' as federation protocol name instead of 'mapped'
bbobrov Mar 20, 2026
c21f5b5
Fix: Use 'saml2' auth method in keystone.conf instead of 'mapped'
bbobrov Mar 20, 2026
e4a51a3
PLAN_C: Runtime SAML config generation from federation chart ConfigMaps
bbobrov Mar 21, 2026
9f57d4f
Fix: Use Docker Hub mirror for SAML init container busybox image
bbobrov Mar 21, 2026
706ccaf
Fix: Remove redundant subPath mount for generate-saml-config.py
bbobrov Mar 21, 2026
04842c3
Fix: Resolve Shibboleth startup warnings
bbobrov Mar 21, 2026
9ff691a
Fix: Move SAML IncludeOptional inside VirtualHost block
bbobrov Mar 21, 2026
4ebe156
Fix: Move Shibboleth handler and LoadModule directly into VirtualHost…
bbobrov Mar 21, 2026
4dd4bee
Fix: Use WSGIScriptAliasMatch to exclude /Shibboleth.sso from WSGI ro…
bbobrov Mar 21, 2026
290aa56
Fix: Follow upstream Keystone pattern for mod_shib + WSGIScriptAlias …
bbobrov Mar 21, 2026
8ab7f96
Fix: Add UseCanonicalName On for mod_shib hostname matching
bbobrov Mar 21, 2026
6ef6e80
Fix: Set handlerSSL=false in shibboleth2.xml for TLS-terminated ingress
bbobrov Mar 21, 2026
8e12e63
Clean up: Remove experimental Apache config, keep only essential SAML…
bbobrov Mar 21, 2026
6f047d9
Clean up: Fix stale comment about LoadModule in generated federation-…
bbobrov Mar 21, 2026
515d671
Fix: Add MetadataProviders and CredentialResolver to ApplicationDefaults
bbobrov Mar 21, 2026
77a2d33
Per-tenant Shibboleth handlerURL for correct ACS applicationId routing
bbobrov Mar 21, 2026
6031b63
Fix: Add RequestMapper for per-tenant Shibboleth handler URL routing
bbobrov Mar 21, 2026
9e55077
Fix: Include all handlers in per-tenant ApplicationOverride Sessions …
bbobrov Mar 21, 2026
4a14d95
Fix: Extract SAML NameID as attribute for REMOTE_USER population
bbobrov Mar 21, 2026
81fd81f
Fix: Add unspecified NameID format decoder and SAP IAS plain-name att…
bbobrov Mar 21, 2026
b83f9dc
Add Chaining CredentialResolver for SP key rotation
bbobrov Mar 21, 2026
58267e7
Fix: Use nested Path elements in RequestMapper for multi-tenant support
bbobrov Mar 25, 2026
640337e
Replace busybox init container with distroless Go binary (keystone-sa…
bbobrov Mar 29, 2026
8463aa1
Fix: Set SP key Secret volume defaultMode to 0444 for nonroot init co…
bbobrov Mar 29, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion openstack/keystone/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ maintainers:
name: keystone
sources:
- https://github.com/sapcc/keystone
version: 0.12.0
version: 0.13.0
dependencies:
- condition: mariadb.enabled
name: mariadb
Expand Down
14 changes: 14 additions & 0 deletions openstack/keystone/alerts/openstack/keystone.alerts
Original file line number Diff line number Diff line change
Expand Up @@ -90,3 +90,17 @@ groups:
annotations:
description: Api Requests to Keystone is {{ $value }}ops/s
summary: Api Requests to Keystone is {{ $value }}ops/s

- alert: OpenstackKeystoneSAMLAssertionValidationFailed
expr: increase(apache_error_log_saml_failures_total{service="keystone"}[5m]) > 0
for: 1m
labels:
context: security
dashboard: keystone
service: keystone
severity: warning
tier: os
support_group: identity
annotations:
description: mod_shib rejected {{ $value }} SAML assertion(s) in the last 5 minutes. Check Apache error logs for signature, timing, or issuer validation failures.
summary: SAML assertion validation failures detected in Keystone
253 changes: 253 additions & 0 deletions openstack/keystone/templates/bin/_generate_saml_config.py.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,253 @@
#!/usr/bin/env python3
"""Generate shibboleth2.xml and federation-saml.conf at runtime.

Reads the tenant list from a JSON file (mounted from the
keystone-saml-tenant-list ConfigMap generated by the federation chart)
and generates:
- /etc/shibboleth/shibboleth2.xml (Shibboleth SP configuration)
- /etc/apache2/conf-enabled/federation-saml.conf (Apache Location blocks)

Static SP configuration (entityId, session settings, memcached) is baked
into this script at Helm template time. Per-tenant configuration (name,
entityId, ApplicationOverride, Location blocks) is read from the tenant
list at runtime.

When the tenant list changes (new customer onboarded), Stakater Reloader
triggers a pod restart, this script runs again, and the new config is
generated.
"""

import json
import os
import sys

TENANT_LIST_PATH = "/etc/shibboleth/tenant-list/tenants.json"
SHIBBOLETH_CONF_PATH = "/etc/shibboleth/shibboleth2.xml"
APACHE_CONF_PATH = "/etc/apache2/conf-enabled/federation-saml.conf"

# Static values from Helm (baked at template time)
SP_ENTITY_ID = "{{ .Values.federation.saml.sp.entityId }}"
SESSION_LIFETIME = "{{ .Values.federation.saml.shibboleth.sessionLifetime | default 28800 }}"
SIGN_AUTHN = "{{ if .Values.federation.saml.shibboleth.signAuthnRequests }}true{{ else }}false{{ end }}"
ENCRYPTION = "{{ if .Values.federation.saml.shibboleth.encryptionEnabled }}true{{ else }}false{{ end }}"

{{- $memcachedHost := "" }}
{{- if .Values.memcached.host }}
{{- $memcachedHost = printf "%s:%s" .Values.memcached.host (.Values.memcached.port | default "11211" | toString) }}
{{- else }}
{{- $memcachedHost = printf "%s-memcached.%s.svc:%s" .Release.Name .Release.Namespace (.Values.memcached.port | default "11211" | toString) }}
{{- end }}
MEMCACHED_HOST = "{{ $memcachedHost }}"


def load_tenants():
"""Load tenant list from the ConfigMap-mounted JSON file."""
if not os.path.exists(TENANT_LIST_PATH):
print("No tenant list found at %s, generating empty config" % TENANT_LIST_PATH)
return []
with open(TENANT_LIST_PATH) as f:
tenants = json.load(f)
print("Loaded %d tenant(s) from %s" % (len(tenants), TENANT_LIST_PATH))
return tenants


def generate_shibboleth_xml(tenants):
"""Generate shibboleth2.xml with per-tenant ApplicationOverride blocks.

Each tenant gets its own handlerURL (/Shibboleth.sso/<tenant>) so that
the ACS endpoint is unique per tenant. This ensures Shibboleth resolves
the correct applicationId when the IdP posts the assertion back, without
relying on relay state to carry the applicationId.
"""

# Per-tenant ApplicationOverride blocks with per-tenant handlerURL.
# The <Sessions> block must be fully specified (not inherited) because
# Shibboleth SP 3.4 does not support handlerURL as an attribute of
# <ApplicationOverride>. The Sessions block must include all handlers
# (Status, Session, MetadataGenerator) or they won't be available
# under the per-tenant handler path.
app_overrides = ""
for t in tenants:
# During key rotation (rotation: true in the region flag file),
# use a Chaining CredentialResolver with both current and next keys.
# Shibboleth tries each resolver in order for decryption/signing.
if t.get("rotation"):
cred_resolver = """
<CredentialResolver type="Chaining">
<CredentialResolver type="File"
key="/etc/shibboleth/sp-keys/{name}-sp-key.pem"
certificate="/etc/shibboleth/sp-keys/{name}-sp-cert.pem"/>
<CredentialResolver type="File"
key="/etc/shibboleth/sp-keys/{name}-sp-next-key.pem"
certificate="/etc/shibboleth/sp-keys/{name}-sp-next-cert.pem"/>
</CredentialResolver>""".format(name=t["name"])
else:
cred_resolver = """
<CredentialResolver type="File"
key="/etc/shibboleth/sp-keys/{name}-sp-key.pem"
certificate="/etc/shibboleth/sp-keys/{name}-sp-cert.pem"/>""".format(name=t["name"])

app_overrides += """
<ApplicationOverride id="{name}" entityID="{sp_entity_id}">
<Sessions handlerURL="/Shibboleth.sso/{name}"
lifetime="{session_lifetime}"
timeout="3600"
relayState="ss:mc"
checkAddress="false"
handlerSSL="false"
cookieProps="https"
redirectLimit="exact">
<SSO>SAML2</SSO>
<Logout>Local</Logout>
<Handler type="MetadataGenerator" Location="/Metadata" signing="false"/>
<Handler type="Status" Location="/Status" acl="127.0.0.1 ::1"/>
<Handler type="Session" Location="/Session" showAttributeValues="false"/>
</Sessions>
<MetadataProvider type="XML"
path="/etc/shibboleth/metadata/{name}-metadata.xml"
reloadChanges="true"/>{cred_resolver}
</ApplicationOverride>""".format(
name=t["name"],
sp_entity_id=SP_ENTITY_ID,
session_lifetime=SESSION_LIFETIME,
cred_resolver=cred_resolver)

# RequestMapper: maps per-tenant handler URLs to the correct applicationId.
# When a request arrives at /Shibboleth.sso/<tenant>/SAML2/POST (the ACS),
# the RequestMapper tells Shibboleth to use the <tenant> ApplicationOverride.
# Paths must be NESTED (not flat with slashes) because Shibboleth parses
# path segments hierarchically. Flat "Shibboleth.sso/ajax" only matches
# the first tenant; nested <Path name="Shibboleth.sso"><Path name="ajax"/>
# correctly matches all tenants.
tenant_paths = ""
for t in tenants:
tenant_paths += """
<Path name="{name}" applicationId="{name}"/>""".format(name=t["name"])
request_mapper_paths = """
<Path name="Shibboleth.sso">{tenant_paths}
</Path>""".format(tenant_paths=tenant_paths)

xml = """<SPConfig xmlns="urn:mace:shibboleth:3.0:native:sp:config"
xmlns:conf="urn:mace:shibboleth:3.0:native:sp:config"
clockSkew="180">

<OutOfProcess tranLogFormat="%u|%s|%IDP|%i|%ac|%t|%attr|%n|%b|%E|%S|%SS|%L|%UA|%a">
<Extensions>
<Library path="memcache-store.so" fatal="false"/>
</Extensions>
</OutOfProcess>

<InProcess>
<ISAPI normalizeRequest="true" safeHeaderNames="true"/>
</InProcess>

<StorageService type="MEMCACHE" id="mc" prefix="shib_" buildMap="1">
<Hosts>{memcached_host}</Hosts>
</StorageService>

<SessionCache type="StorageService" StorageService="mc" cacheAllowance="900"/>
<ReplayCache StorageService="mc"/>
<ArtifactMap StorageService="mc" artifactTTL="180"/>

<RequestMapper type="Native">
<RequestMap>
<Host name="{hostname}">{request_mapper_paths}
</Host>
</RequestMap>
</RequestMapper>

<ApplicationDefaults entityID="{sp_entity_id}"
REMOTE_USER="unspecified-nameid email-nameid persistent-id transient-id eppn subject-id pairwise-id mail"
signing="{sign_authn}"
encryption="{encryption}">

<Sessions lifetime="{session_lifetime}"
timeout="3600"
relayState="ss:mc"
checkAddress="false"
handlerSSL="false"
cookieProps="https"
redirectLimit="exact">

<SSO>SAML2</SSO>
<Logout>Local</Logout>

<Handler type="Status" Location="/Status" acl="127.0.0.1 ::1"/>
<Handler type="Session" Location="/Session" showAttributeValues="false"/>
</Sessions>

<Errors supportContact="root@localhost" helpLocation="/about.html" styleSheet="/shibboleth-sp/main.css"/>

<AttributeExtractor type="XML" validate="true" reloadChanges="true"
path="/etc/shibboleth/attribute-map.xml"/>

<AttributeFilter type="XML" validate="true"
path="/etc/shibboleth/attribute-policy.xml"/>
{app_overrides}

</ApplicationDefaults>

<SecurityPolicyProvider type="XML" validate="true"
path="/etc/shibboleth/security-policy.xml"/>

<ProtocolProvider type="XML" validate="true" reloadChanges="false"
path="protocols.xml"/>

</SPConfig>""".format(
memcached_host=MEMCACHED_HOST,
sp_entity_id=SP_ENTITY_ID,
sign_authn=SIGN_AUTHN,
encryption=ENCRYPTION,
session_lifetime=SESSION_LIFETIME,
hostname=SP_ENTITY_ID.replace("https://", "").replace("http://", "").split("/")[0],
request_mapper_paths=request_mapper_paths,
app_overrides=app_overrides,
)

with open(SHIBBOLETH_CONF_PATH, "w") as f:
f.write(xml)
print("Generated %s with %d tenant(s)" % (SHIBBOLETH_CONF_PATH, len(tenants)))


def generate_apache_conf(tenants):
"""Generate federation-saml.conf with per-tenant Location blocks."""

conf = """#
# Apache mod_shib configuration for SAML 2.0 federation
# Generated at runtime from keystone-saml-tenant-list ConfigMap
# Note: The /Shibboleth.sso handler is in the VirtualHost config
# (wsgi-keystone.conf), not here.
#
"""

for t in tenants:
conf += """
# Tenant: {name} (IdP: {entity_id})
<Location /v3/OS-FEDERATION/identity_providers/{name}/protocols/saml2/auth>
AuthType shibboleth
ShibRequestSetting requireSession 1
ShibRequestSetting applicationId {name}
ShibRequestSetting entityID {entity_id}
ShibUseHeaders Off
Require valid-user
</Location>

<Location /v3/auth/OS-FEDERATION/identity_providers/{name}/protocols/saml2/websso>
AuthType shibboleth
ShibRequestSetting requireSession 1
ShibRequestSetting applicationId {name}
ShibRequestSetting entityID {entity_id}
ShibUseHeaders Off
Require valid-user
</Location>
""".format(name=t["name"], entity_id=t["entityId"])

with open(APACHE_CONF_PATH, "w") as f:
f.write(conf)
print("Generated %s with %d tenant(s)" % (APACHE_CONF_PATH, len(tenants)))


if __name__ == "__main__":
tenants = load_tenants()
generate_shibboleth_xml(tenants)
generate_apache_conf(tenants)
14 changes: 14 additions & 0 deletions openstack/keystone/templates/bin/_keystone_api.sh.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,20 @@ function start () {
rm -f "$APACHE_PID_FILE"
fi

{{- if .Values.federation.saml.enabled }}
# Generate shibboleth2.xml and federation-saml.conf at runtime from the
# tenant-list ConfigMap (keystone-saml-tenant-list). This avoids duplicating
# tenant data in the Keystone chart values — all tenant data lives in the
# federation repo and is passed through via ConfigMaps.
mkdir -p /etc/shibboleth/metadata /etc/shibboleth/sp-keys /var/run/shibboleth /var/cache/shibboleth
python3 /scripts/generate-saml-config.py
# If shibd is available and we are running out-of-process, start it
if command -v shibd &> /dev/null; then
echo "Starting shibd daemon for SAML federation..."
shibd -t 2>/dev/null && shibd -f || echo "WARN: shibd not started, running mod_shib in-process mode"
fi
{{- end }}

# Start Apache2
exec apache2 -DFOREGROUND
}
Expand Down
4 changes: 4 additions & 0 deletions openstack/keystone/templates/configmap-bin.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,7 @@ data:
{{ include (print .Template.BasePath "/bin/_keystone_api.sh.tpl") . | indent 4 }}
region-check.py: |
{{ .Files.Get "files/region-check.py" | indent 4 }}
{{- if .Values.federation.saml.enabled }}
generate-saml-config.py: |
{{ include (print .Template.BasePath "/bin/_generate_saml_config.py.tpl") . | indent 4 }}
{{- end }}
Loading