SSO RBAC Namespace Delegation Question (OpenShift 4.10)

[fhochleitner]

Hi there,

I have been trying to get SSO and Argo Workflows running for days now, and I think I am almost where I want to be however, there is still an Issue with namespace delegation that I can't seem to fix.

First things first, what's our setup:

• Keycloak as OAuth Identity Provider, Version 19.0.1
• Argo Workflows Version 3.3.9
• OpenShift 4.10.28

Controller-Configmap SSO relevant parts:

sso:
  issuer: https://sso.apps.play.gepaplexx.com/realms/internal
  clientId: 
    name: argo-workflows-sso
    key: client-id
  clientSecret: 
    name: argo-workflows-sso
    key: client-secret
  redirectUrl: https://workflows.apps.play.gepaplexx.com/oauth2/callback
  rbac:
    enabled: true

Argo Workflows Server Env SSO_DELEGATE_RBAC_TO_NAMESPACE="true"

I have multiple Service Accounts in Namespace of Argo Workflows Server for RBAC Authentication.
Login with different users that have different Groups works in Argo Workflows Namespace
The correct Service account gets utilized, however when it comes to namespace delegation I am facing an issue.

Theres a read-only user for default login like this:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    workflows.argoproj.io/rbac-rule: '''system:authenticated:oauth'' in groups'
    workflows.argoproj.io/rbac-rule-precedence: "0"
  name: workflows-read-only
  namespace: gepaplexx-cicd-tools

I have a Serviceaccount in the Namespace that I want to delegate RBAC to:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: workflows-developer
  namespace: gepardenblick-cicd
  annotations:
    workflows.argoproj.io/rbac-rule: "'Gepardec' in groups"
    workflows.argoproj.io/rbac-rule-precedence: "200"

The server seems to correctly identify the Serviceaccount in the Namespace, however it then tries to access workflow resources via system:anonymous and I have no idea how or why this happens. Any help would be greatly appreciated.

User is listed following Groups in /userinfo endpoint:
Groups:
Gepardec, system:authenticated, system:authenticated:oauth

and ServiceAccount: workflows-read-only

Then I go to workflowtemplates without a namespace selected and everything seems fine.
Log entries:

time="2022-09-02T15:58:13.695Z" level=info msg="selected SSO RBAC service account for user" email=felix.hochleitner@gepardec.com loginServiceAccount=workflows-read-only serviceAccount=workflows-read-only ssoDelegated=false ssoDelegationAllowed=false subject=677d0b8d-087b-4c50-a07c-69231884a2ed
time="2022-09-02T15:58:13.741Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ListWorkflowTemplates grpc.service=workflowtemplate.WorkflowTemplateService grpc.start_time="2022-09-02T15:58:13Z" grpc.time_ms=49.257 span.kind=server system=grpc

Then I add Namespace to filter, for every Namespace that I don't want to delegate RBAC - same result.
For RBAC delegated Namespace I get following UI error:

Forbidden: workflowtemplates.argoproj.io is forbidden: User "system:anonymous" cannot list resource "workflowtemplates" in API group "argoproj.io" in the namespace "gepardenblick-cicd"

Corresponding logs in argo-server pod:

time="2022-09-02T16:01:05.729Z" level=info msg="selected SSO RBAC service account for user" email=felix.hochleitner@gepardec.com loginServiceAccount=workflows-read-only serviceAccount=workflows-developer ssoDelegated=true ssoDelegationAllowed=true subject=677d0b8d-087b-4c50-a07c-69231884a2ed
time="2022-09-02T16:01:05.743Z" level=warning msg="finished unary call with code PermissionDenied" error="rpc error: code = PermissionDenied desc = workflowtemplates.argoproj.io is forbidden: User \"system:anonymous\" cannot list resource \"workflowtemplates\" in API group \"argoproj.io\" in the namespace \"gepardenblick-cicd\"" grpc.code=PermissionDenied grpc.method=ListWorkflowTemplates grpc.service=workflowtemplate.WorkflowTemplateService grpc.start_time="2022-09-02T16:01:05Z" grpc.time_ms=16.248 span.kind=server system=grpc

and if I check whether the identified service account can actually get workflowtemplates in that namespaces I get:

kubectl auth can-i get workflowtemplates -n gepardenblick-cicd --as system:serviceaccount:gepardenblick-cicd:workflows-developer
yes

[ElhamAhmadlou]

hello @fhochleitner

did you find any solution?

[fhochleitner]

unfortunately not until now. but I have been busy doing some other things lately. was hoping for some input of someone from the argo team/community as I seem to be at a loss of what else there is I could try to fix this behaviour

[fhochleitner]

@alexec do you maybe have any ideas on how to fix our problem or what kind of permissions might be missing?

[fhochleitner]

I have finally managed to find out what the issue was in our setup.
Basically we had several things that in combination made it a bit tricky to find the issue:

• we have started with using namespace delegation RBAC after the change where serviceaccounts don't automatically get their api token referenced as secret in their defintion.
• I have forgotten that I need an additional clusterrolebinding to allow the serviceaccounts in the namespaces to view clusterworkflowtemplates.

[michael-datareply]

Hi Felix,

could you maybe expand a bit on this point:

we have started with using namespace delegation RBAC after the change where serviceaccounts don't automatically get their api token referenced as secret in their defintion.

Unfortunately I don't get what you mean here.

Thank you!

[fhochleitner]

sure.
so basically in the past (previous kubernetes/openshift versions) when you created a ServiceAccount, the apiserver automatically generated an api-token (and a imagepullsecret in case of openshift). Now it doesn't.

Argo Workflows when using SSO and mapping to serviceaccounts uses the serviceaccount to execute whatever you are trying to do via the UI. But without the secret being referenced by the ServiceAccount it appears to not have the token and therefore map to the default unauthenticated user "system:anonymous".

Our solution (besides the additional clusterrolebindings) was the following:

apiVersion: v1
kind: Secret
metadata:
  name: workflow-developer-token
  namespace: hogarama-cicd
  annotations:
    kubernetes.io/service-account.name: workflow-developer
type: kubernetes.io/service-account-token
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: workflow-developer
  namespace: hogarama-cicd
automountServiceAccountToken: true
secrets:
  - name: workflow-developer-token

[ElhamAhmadlou]

Hi @fhochleitner

Your explanation was really helpful for us.
Although our k8s clusters are < 1.24 , on one namespace argo works properly without this error and in other namespace with same configuration doesn't .
We checked the SSO SAs . Their auto generated api-token secrets and imagepullsecret are existed there. The problem was in the order of the secrets in SA's yaml file, if the secret which contains "token" in it's name comes first and one which has "dockercfg" comes next , then the user can get the correct token and the error will disapper from argo ui .

We also tested it many times , token-secret and docker-cfg secrets order is random.

Here you can find the related issue : #8320
@alexec mentioned that it will be fixed in v3.5.

[remicres]

@fhochleitner thanks, this really have helped us a lot! This should deserve a proper note in the documentation

[zillani]

I faced this issue on Openshift.
@fhochleitner Thanks for that amazing example, that saved a lot of time & well explained here #8320

I changed the order of the secrets as shown below, mxp-workflow-token first & then docker-cfg next

apiVersion: v1
automountServiceAccountToken: true
imagePullSecrets:
- name: mxp-dockercfg-qcl48
kind: ServiceAccount
metadata:
  annotations:
    workflows.argoproj.io/rbac-rule: '''mxpadmin'' in groups'
    workflows.argoproj.io/rbac-rule-precedence: "1"
  labels:
    app.kubernetes.io/instance: mxp-core
  name: mxp
  namespace: mxp-platform
secrets:
- name: mxp-workflow-token
- name: mxp-dockercfg-qcl48

[fhochleitner]

glad it helped you :)