AppCred controller support #567

Deydra71 · 2025-04-16T10:12:41Z

This PR introduces a new ApplicationCredential (AC) controller in the keystone-operator. It watches ApplicationCredential custom resources and performs these actions:

Creates Keystone ApplicationCredentials for each CR (authenticating as that user due to Keystone’s default policy)
Stores the AC’s ID and Secret in a k8s secret
Implements rotation logic based on expirationDays and gracePeriodDays:
- Reconciles at least once a day, rotating any AC that’s within or past its grace window
- If an AC is already in the grace period at the next reconcile, it rotates immediately
- The old ApplicationCredential in Keystone is not revoked on rotation (it naturally expires)

Additionally:

The controller waits for a KeystoneAPI resource to be Ready before proceeding with AC operations

Notes:

CRD & RBAC for the ApplicationCredential resource are not automatically installed yet. These must be applied manually until openstack-operator integration is complete

To apply rbac permissions run oc edit clusterrole keystone-operator-manager-role and add:

- apiGroups:
  - keystone.openstack.org
  resources:
  - applicationcredentials
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch

- apiGroups:
  - keystone.openstack.org
  resources:
  - applicationcredentials/finalizers
  verbs:
  - patch
  - update

- apiGroups:
  - keystone.openstack.org
  resources:
  - applicationcredentials/status
  verbs:
  - get
  - patch
  - update

Example AC CR for barbican service user:

apiVersion: keystone.openstack.org/v1beta1
kind: ApplicationCredential
metadata:
  name: ac-barbican
  namespace: openstack
spec:
  expirationDays: 365
  gracePeriodDays: 182
  passwordSelector: BarbicanPassword
  roles:
  - service
  secret: osp-secret
  userName: barbican

openshift-ci · 2025-04-16T10:12:46Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

vakwetu · 2025-05-12T21:55:27Z

api/v1beta1/keystoneapi.go

+) (string, error) {
+
+	// The name of the Secret containing the service passwords
+	const ospSecretName = "osp-secret"


We've discussed this - but basically we provide the ability for the operators to specify the secret that contains the admin user password. This is osp-secret by default - but it need not be. See https://github.com/openstack-k8s-operators/barbican-operator/blob/main/api/v1beta1/common_types.go#L44 for instance.

Agreed, we should not hardcode any value like this. It could be part of the deployment YAML spec.

vakwetu · 2025-05-12T21:59:14Z

api/v1beta1/keystoneapi.go

+		return "", fmt.Errorf("failed to get Secret/%s: %w", ospSecretName, err)
+	}
+
+	key := capitalizeFirst(userName) + "Password"


This may be a default, but its not what is specified. In barbican, for instance we have a PasswordSelectors field - https://github.com/openstack-k8s-operators/barbican-operator/blob/main/api/v1beta1/common_types.go#L49 which identifies the correct key. But, this needn't be the case.

Ultimately, I think you are going to need to have the AC specification include the name of the user, the name of the user secret and the relevant field. You can set these appropriately in openstack-operator.

api/v1beta1/keystoneapplicationcredential_types.go

vakwetu · 2025-05-12T22:52:30Z

controllers/keystoneapplicationcredential_controller.go

+		// Always assign these roles:
+		Roles: []applicationcredentials.Role{
+			{Name: "admin"},
+			{Name: "service"},
+		},


This makes this controller less generic and therefore potentially less useful. Perhaps this is another parameter - like the access rules that should be passed in as part of the AC spec.

The same can also be said for the Unrestricted field.

I see there is support for all of Roles, Unrestricted and AccessRules in the gophercloud call - https://pkg.go.dev/github.com/gophercloud/gophercloud/openstack/identity/v3/applicationcredentials#CreateOpts

vakwetu · 2025-05-12T23:11:45Z

internal/controller/keystoneapplicationcredential_controller.go

+	}
+
+	// Otherwise check again in 24 hours
+	return defaultRequeue


Shouldn't this be return rotateAt ?

rotateAt is a timestamp, we need return a duration

OK that makes sense. I (mis)understood this function when I first read it.

So if I understand this correctly now, we should be returning a recheck duration of 24 hours, unless we are already in the grace period - in which case we would be returning a 0 to immediately recheck.

Yes, that's exactly right

vakwetu · 2025-05-12T23:12:22Z

internal/controller/keystoneapplicationcredential_controller.go

+	// default requeue is 24h as minimal grace period is 1 day
+	defaultRequeue := 24 * time.Hour
+	if ac.Status.ExpiresAt == nil || ac.Status.ExpiresAt.IsZero() {
+		return defaultRequeue


I'm curious - why would we want to requeue if the application credential does not expire?

I was thinking about this as an ultimate fallback to wake the controller at least once a day. If because of some error the status of the CR is not updated.

vakwetu · 2025-05-12T23:13:49Z

controllers/keystoneapplicationcredential_controller.go

+		return "fallback" // placeholder for generating failure
+	}
+	s := hex.EncodeToString(b)
+	return s[:n]


Its unlikely, but should we check for collisions?

We could check if the AC with same suffix already exists, but unless we are creating millions of AC in short period the chance is basically zero. Or we can increase n.

should this function be a lib-common utility in the long term?

vakwetu · 2025-05-13T05:14:37Z

controllers/keystoneapplicationcredential_controller.go

+
+	logger := r.GetLogger(ctx)
+
+	// Only if user explicitly does "oc delete" do we revoke the AC


Its actually quite useful, I think, to have the delete revoke the application credential. This gives us a nice way to do a revocation.

The problem is, of course, that there will be some time between when the app cred is revoked and the new one is issued.

I wonder if there is a way to trigger a rotation without doing a delete. Could we implement a reconcileUpdate to do this instead? Then, the procedure when trying to revoke the cert would be to -

patch the AC so that we are within the grace period. This triggers the creation of a new AC.

Wait for the new AC to be propagated.

Revoke the old AC by deleting it.

I'm just concerned about step 2. - Wait for the new AC to be propagated. Because AC controller would need a feedback that the service deployment successfully rolled out with new credential, which would add another logic to watch the deployment status in AC controller. I’d prefer to leave revocation as a manual step for now.

stuggi

I have not finished my initial review, but probably won't have time today to do so. will continue tomorrow, and just wanted to add what I have so far.

stuggi · 2025-05-13T10:19:42Z

controllers/keystoneapplicationcredential_controller.go

+	// Decide if we need to create (or rotate)
+	needsRotation := false
+	if instance.Status.ACID == "" {
+		needsRotation = true
+		logger.Info("AC does not exist, creating")
+	} else if r.shouldRotateNow(instance) {
+		needsRotation = true
+		logger.Info("AC is within grace period, rotating")
+	}


lets split this out into a func

stuggi · 2025-05-13T10:21:19Z

controllers/keystoneapplicationcredential_controller.go

+	// Check if KeystoneAPI is ready
+	keystoneAPI, err := keystonev1.GetKeystoneAPI(ctx, helperObj, instance.Namespace, nil)
+	if err != nil {
+		logger.Info("KeystoneAPI not found, requeue", "error", err)


should set KeystoneAPIReadyCondition

stuggi · 2025-05-13T10:24:12Z

controllers/keystoneapplicationcredential_controller.go

+		return ctrl.Result{RequeueAfter: 10 * time.Second}, nil
+	}
+	if !keystoneAPI.IsReady() {
+		logger.Info("KeystoneAPI not ready, requeue")


set a condition as in https://github.com/openstack-k8s-operators/keystone-operator/blob/main/controllers/keystoneendpoint_controller.go#L186

stuggi · 2025-05-13T10:37:46Z

controllers/keystoneapplicationcredential_controller.go

+
+	// Ensure we have an initial ReadyCondition
+	condList := condition.CreateList(
+		condition.UnknownCondition(condition.ReadyCondition, condition.InitReason, condition.ReadyInitMessage),


the ReadyCondition gets auto added in Init.

should init KeystoneAPIReadyCondition and set it when checking for the keystoneapi bellow

condition.UnknownCondition(keystonev1.KeystoneAPIReadyCondition, condition.InitReason, keystonev1.KeystoneAPIReadyInitMessage),

I think we could just use the same condition init as in https://github.com/openstack-k8s-operators/keystone-operator/blob/main/controllers/keystoneendpoint_controller.go#L111-L127 and just init it with what is used in this controller?

stuggi · 2025-05-13T10:38:51Z

controllers/keystoneapplicationcredential_controller.go

+	}
+
+	// Check if KeystoneAPI is ready
+	keystoneAPI, err := keystonev1.GetKeystoneAPI(ctx, helperObj, instance.Namespace, nil)


https://github.com/openstack-k8s-operators/keystone-operator/blob/main/controllers/keystoneendpoint_controller.go#L134-L165 is an example which sets the conditions

stuggi · 2025-05-13T10:42:17Z

internal/controller/keystoneapplicationcredential_controller.go

+		logger.Info("KeystoneAPI not ready, requeue")
+		return ctrl.Result{RequeueAfter: 10 * time.Second}, nil
+	}
+


mark the KeystoneAPIReadyCondition when we get here, https://github.com/openstack-k8s-operators/keystone-operator/blob/main/controllers/keystoneendpoint_controller.go#L195

also I'd move all the above tasks into the main reconcile func as these are also the steps done for handling deletion. then you won't need login to check for the keystoneapi in the cleanup method. like https://github.com/openstack-k8s-operators/keystone-operator/blob/main/controllers/keystoneendpoint_controller.go#L225 where the admin client got checked before

stuggi · 2025-05-13T10:45:26Z

controllers/keystoneapplicationcredential_controller.go

+	logger := r.GetLogger(ctx)
+	adminOS, ctrlResult, err := keystonev1.GetAdminServiceClient(ctx, helperObj, keystoneAPI)
+	if err != nil {
+		return "", err


should set condition as in https://github.com/openstack-k8s-operators/keystone-operator/blob/main/controllers/keystoneendpoint_controller.go#L197-L222

stuggi · 2025-05-13T10:46:29Z

controllers/keystoneapplicationcredential_controller.go

+		if err != nil {
+			logger.Error(err, "Failed to find user ID")
+			instance.Status.Conditions.Set(condition.FalseCondition(
+				condition.ReadyCondition,


this seems to be not the correct condition. the ReadyCondition is only set in the defer function based on the sub condition status during the reconcile

stuggi · 2025-05-13T10:48:18Z

controllers/keystoneapplicationcredential_controller.go

+	}
+	savedConditions := instance.Status.Conditions.DeepCopy()
+
+	// Defer patch logic (skips if we are deleting)


I think we should keep the defer function, like we use in all the controllers https://github.com/openstack-k8s-operators/keystone-operator/blob/main/controllers/keystoneendpoint_controller.go#L91-L109

stuggi · 2025-05-13T10:58:37Z

controllers/keystoneapplicationcredential_controller.go

+		keystoneAPI, err := keystonev1.GetKeystoneAPI(ctx, helperObj, instance.Namespace, nil)
+		if err == nil && keystoneAPI.IsReady() {
+			userID, userErr := r.findUserIDAsAdmin(ctx, helperObj, keystoneAPI, instance.Spec.UserName)
+			if userErr == nil {


I think we should try to reduce those amount of nested ifs.

what if there is an userErr, which is not "user not found"? is it ok continue and remove the finalizer? shouldn't we return the actual error if it is not "user not found" ?

can't we just do

userID, userErr := r.findUserIDAsAdmin(ctx, helperObj, keystoneAPI, instance.Spec.UserName) if userErr != nil { return ctrl.Result{}, err } userOS, userRes, userErr2 := keystonev1.GetUserServiceClient(ctx, helperObj, keystoneAPI, instance.Spec.UserName) ...

mauricioharley · 2025-05-13T14:45:00Z

api/v1beta1/keystoneapi.go

+) (string, error) {
+
+	// The name of the Secret containing the service passwords
+	const ospSecretName = "osp-secret"


Agreed, we should not hardcode any value like this. It could be part of the deployment YAML spec.

mauricioharley · 2025-05-13T15:02:20Z

api/v1beta1/keystoneapi.go

+		return "", fmt.Errorf("failed to get Secret/%s: %w", ospSecretName, err)
+	}
+
+	key := capitalizeFirst(userName) + "Password"


Why do you need to capitalize the username's first letter? Also, why are you harcoding "Password"?

This was just to filter user password from osp-secret. Now, passwordSelector will be passed to AC CR, so that will be used for filtering

I don't get it. Can you elaborate?

The previous code assumed this password scheme in osp-secret:

BarbicanPassword: <secret_data> CinderPassword: <secret_data> GlancePassword: <secret_data> …

So to pick the right field we had to:

Take the userName ("barbican")

Capitalize the first letter → "Barbican"

Append "Password" → "BarbicanPassword"

and then look up that key in the secret. Of course that'd force everyone to use that convention. So now openstack-operator extracts and passes into AC CR these as well:

spec: secret: # the Secret name (by default it's osp-secret) passwordSelector: # how we extract this key, e.g. BarbicanPassword userName: # e.g. barbican, user can customize this in control plane spec …

api/v1beta1/keystoneapplicationcredential_types.go

Deydra71 · 2025-05-14T11:54:06Z

The latest update makes the controller take into consideration the custom password secret, custom service user name, and password selectors. Continuing to address other reviews.

Deydra71 · 2025-05-16T13:59:00Z

Latest update includes corrections based on some reviews (not all, will still continue), and also adds support for the Unrestricted, Roles and AccessRoles fields.

Currently AccessRoles field is not correctly passed to the AC CR by openstack-operator, so it's missing in the AC CR, and for services this field is specified in the openstackcontrolplane spec, ACs are not generated. We are continuing on Slack what should be in AccessRules fields in the first place.

And also automatic revocation is disabled for now for testing, based on what we will agree it will be enabled again.

vakwetu · 2025-05-27T19:55:41Z

api/v1beta1/keystoneapi.go

+	scope := &gophercloud.AuthScope{
+		ProjectName: "service",
+		DomainName:  "Default",
+	}


I'm very wary about hard-coding stuff here. I think that;

there may be cases where either the service project or DomainName may be different

we might be interested in obtaining an app cred for a non-service user or for a different project. For example, I know that there is work ongoing upstream to do manila share encryption where the user would use app creds to retrieve a barbican secret.

In any case, I think it makes sense to make this function more generic - maybe call it GetUserClient() and pass in the domain and project name. You could then add these parameters to the app cred spec, and default to "service" and "Default". This will future-proof the interface a bit.

I understand, but right now we don't have any spec.ServiceProject or spec.ServiceDomain. So, that's why this serves as a mean to only get scoped token for our service.

Since FR3 is explicitly about wiring up service-account auth, I suggest we leave this helper as is for now, and add a // TODO pointing at the future work to extend the CRD with project/domain fields and refactor into a generic GetUserClient(…, project, domain…)

The domain name is actually hard coded for admin as well - https://github.com/openstack-k8s-operators/keystone-operator/blob/main/api/v1beta1/keystoneapi.go#L157

there may be cases where either the service project or DomainName may be different

Is this still true if we take into account only service users?

I agree that hard coding is not good and we could do it like we did in the GetScopedAdminServiceClient, but can not comment on the need for this service client. there is just really little difference to the func GetScopedAdminServiceClient , maybe we could just make a generic GetScopedClient with parameters:
scope, TenantName, DomainName and Region and could deprecate the GetScopedAdminServiceClient

I agree about having a generic function, the old one can be deprecated and it might produce a warning, so we would know in the service operators that we need to update to the new function call (e.g. [1]).

[1] https://github.com/openstack-k8s-operators/glance-operator/blob/main/internal/controller/glance_controller.go#L1126

vakwetu · 2025-05-27T19:56:02Z

api/v1beta1/keystoneapi.go

+			AuthURL:    authURL,
+			Username:   userName,
+			Password:   password,
+			TenantName: "service",


ditto comment as above.

vakwetu · 2025-05-27T20:48:53Z

internal/controller/keystoneapplicationcredential_controller.go

+		secretName := fmt.Sprintf("%s-secret", instance.Name)
+		if err := r.storeACSecret(ctx, helperObj, instance, secretName, newID, newSecret); err != nil {
+			return ctrl.Result{}, err
+		}


The keystone-operator is somewhat unique in the operators in that there isn't a lot of documentation of what it does. Just looking at the code , for instance, it wasn't completely clear to me what the name of the ac would be, and the name of the secret etc. I'm sure I could figure it out - but I shouldn't have to work so hard.

It would be great if there were a small doc (maybe in a docs directory like the other operators) which contains a description of what we'd expect this operator to produce. It doesn't have to be fancy, but it will help future reviewers and coders who want to enhance this feature.

I added the doc dir and a new file describing basic functionality of the AC controller - will add more details soon.

fmount · 2025-12-10T08:38:50Z

api/v1beta1/keystoneapplicationcredential.go

+	secret := &corev1.Secret{}
+	key := types.NamespacedName{Namespace: namespace, Name: GetACSecretName(serviceName)}
+	if err := c.Get(ctx, key, secret); err != nil {
+		if k8s_errors.IsNotFound(err) {


@Deydra71 just wondering if VerifyApplicationCredentialsForService should be embedded here since we pass pretty much the same parameters and it would hide more details from the service operators that right now call both functions.
It would be nice to have a single entrypoint, and delegate keystone to do the verification, that of course can return an error we can handle.

Not sure it would be that much beneficial for service operators, because VerifyApplicationCredentialsForService just tracks the AC secret and add the hash to configVars, it doesn't read the actual credentials. The GetApplicationCredentialFromSecret then serves only to get the actual data from the secret during generating config at later stage. Separating the change tracking from credential extraction seems logical to me and makes the reconcile flow clearer to understand in my opinion. @stuggi What do you think?

see my comment here https://github.com/openstack-k8s-operators/glance-operator/pull/812/files#r2609517115 I guess it depends on if we can do it in a single place for the service or not. I think we do not need a dedicated hash since the AC is part of the service config which we already hash. so there is probably no need to get a dedicated hash for it.

Deydra71 · 2026-01-07T18:11:21Z

@stuggi @fmount hi! Could you please take a look again at the updated PR with the suggested changes? We’d like to merge it so we can proceed with kuttl and other testing in the service operators. I also updated the service operators that are supposed to use AppCred (telemetry still in progres) to use field indexers instead of custom watchers.

stuggi · 2026-01-14T07:08:04Z

api/v1beta1/keystoneapi.go

+	scope := &gophercloud.AuthScope{
+		ProjectName: "service",
+		DomainName:  "Default",
+	}


I agree that hard coding is not good and we could do it like we did in the GetScopedAdminServiceClient, but can not comment on the need for this service client. there is just really little difference to the func GetScopedAdminServiceClient , maybe we could just make a generic GetScopedClient with parameters:
scope, TenantName, DomainName and Region and could deprecate the GetScopedAdminServiceClient

stuggi · 2026-01-14T07:12:07Z

api/v1beta1/keystoneapi.go

+		passwordSelector,
+	)
+	if err != nil {
+		return "", ctrl.Result{}, fmt.Errorf("failed to get %q from Secret/%s: %w", passwordSelector, ospSecretName, err)


do we need this message, the lib-common func already returns an error string depending on the condition it checks ( generic error, secret not there, key not found)

stuggi · 2026-01-14T07:13:03Z

api/v1beta1/keystoneapi.go

+	if res != (ctrl.Result{}) {
+		return "", res, nil
+	}
+	if data == "" {


do we need this? is an empty string a valid pwd in keystone? creating the authclient should fail.

stuggi · 2026-01-14T07:25:22Z

docs/applicationcredentials.md

+| description  | Created by keystone-operator for AC CR openstack/ac-barbican (user... |
+| expires_at   | 2026-05-29T09:02:28.705012                                             |
+| id           | 7b23dbac20bc4f048f937415c84bb329                                       |
+| name         | ac-barbican-29090                                                      |
+| project_id   | 24f05745bc2145c6a625b528ce21d7a3               |
+| roles        | service                                        |
+| system       | None                                           |
+| unrestricted | False                                          |
+| user_id      | 2ecd25b38f0d432388ad8b838e46f36d               |


ni, table formating

stuggi · 2026-01-14T07:28:37Z

docs/applicationcredentials.md

+- Create New AC
+  - Generates a new Keystone AC with a fresh 5-char suffix
+  - Uses the same roles, unrestricted flag, access rules, and expirationDays
+  - Does _not_ revoke the old AC, the old credential naturally expires


what happens when it expired. gets it removed from keystone db automatic, or do we need a cron which cleans them, like we have for the tokens?

that's a goo dpoint... I don't think they are deleted from the database. The expiration only makes the auth to fail. @vakwetu can you please confirm?

stuggi · 2026-01-14T08:25:29Z

internal/controller/keystoneapplicationcredential_controller.go

+				keystonev1.KeystoneApplicationCredentialReadyCondition,
+				condition.RequestedReason,
+				condition.SeverityInfo,
+				"Waiting for secret %q to be available",


lets add a KeystoneApplicationCredentialWaitingMessage

stuggi · 2026-01-14T08:32:48Z

internal/controller/keystoneapplicationcredential_controller.go

+	data := map[string]string{
+		keystonev1.ACIDSecretKey:     newID,
+		keystonev1.ACSecretSecretKey: newSecret,
+	}
+
+	// Add application-credentials label
+	labels := map[string]string{
+		"application-credentials": "true",
+	}
+
+	tmpl := []util.Template{{
+		Name:       secretName,
+		Namespace:  ac.Namespace,
+		Type:       util.TemplateTypeNone,
+		CustomData: data,
+		Labels:     labels,
+	}}
+	if err := oko_secret.EnsureSecrets(ctx, helperObj, ac, tmpl, nil); err != nil {
+		return err
+	}
+
+	// Ensure protection finalizer is present on the Secret
+	secret := &corev1.Secret{}
+	key := types.NamespacedName{Namespace: ac.Namespace, Name: secretName}
+	if err := helperObj.GetClient().Get(ctx, key, secret); err != nil {
+		return err
+	}
+	base := secret.DeepCopy()
+	if controllerutil.AddFinalizer(secret, acSecretFinalizer) {
+		if err := helperObj.GetClient().Patch(ctx, secret, client.MergeFrom(base)); err != nil {
+			if k8s_errors.IsConflict(err) {
+				return fmt.Errorf("%w: requeue: patch conflict on %s", err, secretName)
+			}
+			return err
+		}
+	}


since EnsureSecrets does not support adding a finalizer, its more efficient if you just use controllerutil.CreateOrPatch() to create/update the secret in a single request. could also enhance CreateOrPatchSecret for adding a finalizer if present in the secret passed in.

stuggi · 2026-01-14T08:38:35Z

internal/controller/keystoneapplicationcredential_controller.go

+				instance,
+				corev1.EventTypeNormal,
+				"ApplicationCredentialRotated",
+				fmt.Sprintf("ApplicationCredential '%s' (user: %s) rotated - EDPM nodes may need credential updates. Previous expiration: %s, New expiration: %s",


nice! would be also good to set a warning condition https://github.com/openstack-k8s-operators/dev-docs/blob/main/conditions.md#severity on the ctlpane, that the controlplane flips to not ready?

@stuggi I'm not sure I udnerstand, could you please elaborate where specifically we should set the warning condition? On KeystoneApplicationCredential or else?

yes that was my idea this morning, but thinking of it again, we probably can not, since we do not know when the edpm side got updated to flip it back to ready.

@slagle is there a way to check if a nodedataset (all nodes) got updated with the latest config?

stuggi · 2026-01-14T08:42:01Z

internal/controller/keystoneapplicationcredential_controller.go

+	if err := r.Patch(ctx, instance, client.MergeFrom(base)); err != nil {
+		if k8s_errors.IsConflict(err) {
+			return ctrl.Result{Requeue: true}, nil
+		}
+		return ctrl.Result{}, err
+	}


since it is on the instance itself and return afterwards, the patch is not required since we will do it in the defer func

stuggi · 2026-01-14T08:48:14Z

internal/controller/keystoneapplicationcredential_controller.go

+
+// GetLogger returns a logger configured for this controller.
+func (r *ApplicationCredentialReconciler) GetLogger(ctx context.Context) logr.Logger {
+	return ctrlLog.FromContext(ctx).WithName("ApplicationCredentialReconciler")


we usually use the format:

return ctrlLog.FromContext(ctx).WithName("Controllers").WithName("KeystoneApplicationCredential")

auniyal61 · 2026-01-14T10:02:04Z

internal/controller/keystoneapplicationcredential_controller.go

+		// Store it in a Secret (create or update)
+		secretName := fmt.Sprintf("%s-secret", instance.Name)
+		if err := r.storeACSecret(ctx, helperObj, instance, secretName, newID, newSecret); err != nil {
+			if strings.HasPrefix(err.Error(), "requeue: ") {


the error msg created at L482 in storeACSecret do not start with "requeue"

return fmt.Errorf("%w: requeue: patch conflict on %s", err, secretName)

may use Contains instead or put %w at the end of msg

auniyal61 · 2026-01-14T10:30:43Z

internal/controller/keystoneapplicationcredential_controller.go

+	// If we're not deleting this and the service object doesn't have our finalizer, add it.
+	if instance.DeletionTimestamp.IsZero() && controllerutil.AddFinalizer(instance, finalizer) || isNewInstance {
+		return ctrl.Result{}, nil
+	}


true and true or true
should it be true and (true or true)

so for first reconcide
not-deleted-yet and (able-to-add-finalizer-just-now or its-new-instance)

because AND has more precedence, for later reconciles it can be read as
not-deleted-yet and able-to-add-finalizer-just-now
true and false || false
which is never true as we cant add same finalizer again (as assumed already here on condition)

So I meant say seems fine for first reconcile may create issue in second and later reconciles because of condition precedent - need to test though.
true and (false or false)

wow, thanks for this , I realized with this comemnt that there are two cases where we could return too early -->

if an instance is created and immediately deleted before first reconcile

and if for some reason the instance looks new, because of the status wan't persisted yet, because of eg failed first reconcile

I think it will reconcile delete on the next event, but we can't assume that of course... I will add the parentess.

if instance.DeletionTimestamp.IsZero() && (controllerutil.AddFinalizer(instance, finalizer) || isNewInstance) { return ctrl.Result{}, nil }

Deydra71 · 2026-01-14T13:53:58Z

/retest

Deydra71 · 2026-01-14T14:05:41Z

ATM not sure why the functional test fails, locally the suite passes, and also the failure is connected to the MariaDB part, not our app cred changes https://github.com/openstack-k8s-operators/keystone-operator/blob/main/test/functional/keystoneapi_controller_test.go#L2246

stuggi · 2026-01-14T14:50:29Z

internal/controller/keystoneapplicationcredential_controller.go

+	if !instance.DeletionTimestamp.IsZero() {
+		return r.reconcileDelete(ctx, instance, helperObj)
+	}


this is not the same as above L166 ? if we don't do any cleanup on the keystone side, I guess we can just remove it here?

oh, yes I forgot to remove it, it's redundant now

fmount · 2026-01-14T14:00:51Z

api/v1beta1/keystoneapi.go

+	scope := &gophercloud.AuthScope{
+		ProjectName: "service",
+		DomainName:  "Default",
+	}


I agree about having a generic function, the old one can be deprecated and it might produce a warning, so we would know in the service operators that we need to update to the new function call (e.g. [1]).

[1] https://github.com/openstack-k8s-operators/glance-operator/blob/main/internal/controller/glance_controller.go#L1126

fmount · 2026-01-14T14:05:25Z

api/v1beta1/keystoneapplicationcredential.go

+
+// ComputeSecurityHash computes a hash of security-critical spec fields
+// (roles, accessRules, unrestricted). Used to detect changes that require immediate rotation.
+func ComputeSecurityHash(spec KeystoneApplicationCredentialSpec) (string, error) {


is this function used somewhere, perhaps in service operators?

Not in service operators, but in internal/controller/keystoneapplicationcredential_controller.go[1] to detect permission changes and trigge app cred rotation.

[1] https://github.com/openstack-k8s-operators/keystone-operator/pull/567/files#diff-bbfa4b5edb0132743cb1ad35d9483082d2cab3d6a16540023740f107ed3e1075R309

fmount · 2026-01-14T14:17:30Z

internal/controller/keystoneapplicationcredential_controller.go

+
+	// ApplicationCredential already exists and is valid
+	// Update RotationEligibleAt in case GracePeriodDays changed
+	if instance.Status.ExpiresAt != nil {


@Deydra71 @stuggi quick question here: should we add ObservedGeneration check before reaching the Ready state (we assign it on L124 iirc)?

what check you are thinking about?

I was wondering if it makes sense to check that .Status.ObservedGenerations matches w/ instance.ObservedGeneration in case of multiple updated to the same CR but we probably don't need it as there aren't multiple controllers involved.
Let's put this aside for now as I'm not entirely sure it would change anything in the reconciliation (I didn't test it).

stuggi · 2026-01-14T16:24:14Z

/test keystone-operator-build-deploy-kuttl

…r functions for service operators. Signed-off-by: Veronika Fisarova <[email protected]>

Deydra71 · 2026-01-15T04:44:13Z

@stuggi @fmount I refactored generic GetScopedClient + deprecate GetScopedAdminServiceClient based on your comments, can you please recheck it?

auniyal61 · 2026-01-16T05:55:39Z

internal/controller/keystoneapplicationcredential_controller.go

+				instance,
+				corev1.EventTypeNormal,
+				"ApplicationCredentialRotated",
+				fmt.Sprintf("ApplicationCredential '%s' (user: %s) rotated - EDPM nodes may need credential updates. Previous expiration: %s, New expiration: %s",


+1, this helps in events

Deydra71 added the do-not-merge/work-in-progress label Apr 16, 2025

Deydra71 requested review from mauricioharley, stuggi, vakwetu and xek April 16, 2025 10:12

Deydra71 force-pushed the underlined-ac-support branch 2 times, most recently from 46e453b to 040976c Compare April 28, 2025 10:00

Deydra71 force-pushed the underlined-ac-support branch from 040976c to 38cb512 Compare May 7, 2025 07:37

This was referenced May 7, 2025

AppCred support openstack-k8s-operators/openstack-operator#1430

Open

AppCred service operator support openstack-k8s-operators/barbican-operator#248

Open

vakwetu reviewed May 12, 2025

View reviewed changes

api/v1beta1/keystoneapplicationcredential_types.go Show resolved Hide resolved

vakwetu reviewed May 12, 2025

View reviewed changes

api/v1beta1/keystoneapplicationcredential_types.go Show resolved Hide resolved

vakwetu reviewed May 12, 2025

View reviewed changes

vakwetu reviewed May 13, 2025

View reviewed changes

stuggi reviewed May 13, 2025

View reviewed changes

mauricioharley reviewed May 13, 2025

View reviewed changes

Deydra71 force-pushed the underlined-ac-support branch from 38cb512 to 15fcdb9 Compare May 14, 2025 11:50

Deydra71 force-pushed the underlined-ac-support branch from 15fcdb9 to 2d8a489 Compare May 16, 2025 13:50

Deydra71 force-pushed the underlined-ac-support branch from 2d8a489 to 4962009 Compare May 19, 2025 12:35

vakwetu reviewed May 27, 2025

View reviewed changes

fmount reviewed Dec 10, 2025

View reviewed changes

Deydra71 force-pushed the underlined-ac-support branch 2 times, most recently from 070277e to 3e1a3e0 Compare December 11, 2025 08:56

Deydra71 mentioned this pull request Dec 15, 2025

Application Credential Support openstack-k8s-operators/watcher-operator#317

Open

Deydra71 mentioned this pull request Jan 7, 2026

Application Credential support openstack-k8s-operators/nova-operator#1055

Open

Deydra71 requested review from fmount and stuggi January 7, 2026 18:07

Deydra71 mentioned this pull request Jan 8, 2026

Add Application Credential support openstack-k8s-operators/telemetry-operator#829

Open

amoralej mentioned this pull request Jan 14, 2026

DNM test only openstack-k8s-operators/watcher-operator#330

Open

stuggi reviewed Jan 14, 2026

View reviewed changes

afaranha mentioned this pull request Jan 14, 2026

Application Credential support openstack-k8s-operators/heat-operator#610

Open

auniyal61 requested changes Jan 14, 2026

View reviewed changes

openshift-ci bot assigned auniyal61 Jan 14, 2026

Deydra71 force-pushed the underlined-ac-support branch from 3e1a3e0 to ec8a09f Compare January 14, 2026 13:39

Deydra71 requested review from auniyal61 and stuggi January 14, 2026 13:40

stuggi reviewed Jan 14, 2026

View reviewed changes

Deydra71 force-pushed the underlined-ac-support branch from ec8a09f to 3c7806d Compare January 14, 2026 15:29

fmount reviewed Jan 14, 2026

View reviewed changes

Add Application Credential API types, AC-controller support and helpe…

12e6048

…r functions for service operators. Signed-off-by: Veronika Fisarova <[email protected]>

Deydra71 force-pushed the underlined-ac-support branch from 3c7806d to 12e6048 Compare January 15, 2026 04:39

auniyal61 reviewed Jan 16, 2026

View reviewed changes

Deydra71 requested review from auniyal61 and fmount January 16, 2026 06:46


		logger := r.GetLogger(ctx)

		// Only if user explicitly does "oc delete" do we revoke the AC

AppCred controller support #567

Are you sure you want to change the base?

AppCred controller support #567

Conversation

Deydra71 commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Apr 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stuggi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Deydra71 commented Apr 16, 2025 •

edited

Loading

Deydra71 commented May 16, 2025 •

edited

Loading

Deydra71 May 28, 2025 •

edited

Loading

Deydra71 May 28, 2025 •

edited

Loading