RBAC for tapping

[danibaeyens]

Hi,
I'm (yeah, still) using linkerd 2.10 and I run a multi-tenant cluster with RBAC for different users (each user has their own namespace).
One of them has set up a service profile for an ExternalName service and docs say:

Note that at present, you cannot view statistics gathered for routes in this ServiceProfile in the web 
dashboard. You can get the statistics using the CLI.

I've tried to grant them permissions for running a linkerd viz routes command but they seem to need:

Cannot connect to Linkerd Viz: secrets "tap-k8s-tls" is forbidden: User ".... user ...." cannot get
resource "secrets" in API group "" in the namespace "linkerd-viz"

Validate the install with: linkerd viz check

I don't see anywhere in the docs which specific permissions I need to grant... is this one really required?
Is this, maybe, RBAC documented anywhere?
The references I find related to linkerd-linkerd-viz-tap-admin does not seem to grant get/list/watch for secrets, i.e.:

linkerd2/viz/cmd/testdata/install_prometheus_disabled.golden

Line 67 in 7ac79b8

### Tap RBAC

It looks like it's needed to extend the permissions even further.

So basically, is there any doc? What are the needed permissions and the reason for granting so much for tapping? (up to secrets in linkerd-viz namespace)

[olix0r]

I think there are a few issues here.

Note that at present, you cannot view statistics gathered for routes in this ServiceProfile in the web 
dashboard. You can get the statistics using the CLI.

As far as I understand, this shouldn't require tap. linkerd viz routes uses prometheus and should only require access to the metrics-api pod.

I've tried to grant them permissions for running a linkerd viz routes command but they seem to need:

What did you do, specifically?

Cannot connect to Linkerd Viz: secrets "tap-k8s-tls" is forbidden: User ".... user ...." cannot get
resource "secrets" in API group "" in the namespace "linkerd-viz"

Validate the install with: linkerd viz check

What command generated this output?

I don't see anywhere in the docs which specific permissions I need to grant... is this one really required?

No, you shouldn't need secret access to use Tap.

https://linkerd.io/2.11/tasks/securing-your-cluster/ includes details on configuring Tap RBAC.

[danibaeyens]

I think there are a few issues here.

Note that at present, you cannot view statistics gathered for routes in this ServiceProfile in the web 
dashboard. You can get the statistics using the CLI.

As far as I understand, this shouldn't require tap. linkerd viz routes uses prometheus and should only require access to the metrics-api pod.

I think you are right and I'm mixing topics here. I'm reviewing my chats with the user and we were talking about routes and tap without making differences.

Anyway...

I've tried to grant them permissions for running a linkerd viz routes command but they seem to need:

What did you do, specifically?
What command generated this output?

My fault! he was not doing a linkerd viz routes but linkerd viz tap.

He was following:
https://linkerd.io/2.10/tasks/setting-up-service-profiles/

And he was trying to do:
linkerd viz tap -n booksapp deploy/webapp -o wide

And he got his first permissions issue, so I created a new clusterrole/clusterrolebinding assigned to his user, and started to allow permissions one by one to see which ones were required. Basically he got requested:

• list / namespaces
• list / clusterroles
• list / clusterrolebindings
• get / apiservices for v1alpha1.tap.linkerd.io
  And finally
• get /secrets for tap-k8s-tls

I don't see anywhere in the docs which specific permissions I need to grant... is this one really required?

No, you shouldn't need secret access to use Tap.

https://linkerd.io/2.11/tasks/securing-your-cluster/ includes details on configuring Tap RBAC.

[olix0r]

Ah, OK. It's been a while since I've looked at this, so it make take a little testing to get right. (and we should update the docs once it gets figured out!).

I think you should be able to create a role like:

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: booksapp
  name: tap
rules:
- apiGroups: ["tap.linkerd.io"]
  resources: ["*"]
  verbs: ["watch"]

and then bind the user's account to it with a role binding.

[danibaeyens]

Considering that cluster users have zero access to other namespaces, this is what I've finally needed to avoid complains from linkerd viz check and linkerd viz tap:

One ClusterRole with a ClusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cluster-wide-read-linkerd-users
rules:
- apiGroups:
  - apiregistration.k8s.io
  resources:
  - apiservices
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - list
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - clusterroles
  - clusterrolebindings
  verbs:
  - list
- apiGroups:
  - tap.linkerd.io
  resources:
  - '*'
  verbs:
  - watch

And another one like with a RoleBinding to provide access just to linkerd-viz namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: linkerd-viz-read-linkerd-users
rules:
- apiGroups:
  - ""
  resourceNames:
  - tap-k8s-tls
  resources:
  - secrets
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - pods
  - configmaps
  verbs:
  - list
- apiGroups:
  - apps
  resources:
  - replicasets
  verbs:
  - list
  - get
- apiGroups:
  - ""
  resources:
  - pods/portforward
  verbs:
  - create

Otherwise it claimed it had no permissions for one or the other operation

[jrhunger]

@danibaeyens Thanks for this. The grants you used worked for me, but each user could then see all linkerd resources (routes, edges, stats, etc.) from all namespaces, not just their own. Is that your experience as well?

[danibaeyens]

In our case, it is like that. We have hard multitenancy on the clusters (users cannot see or interact on others' namespaces) but on the observability tools they are allowed to see other users' metrics.
As soon as you are allowing port forwarding to linkerd-viz's metrics-api pod, they can query anything there, as far as I understand.

[jrhunger]

Looks like if you remove this part from the cluster-wide-read-linkerd-users ClusterRole, and apply it to a separate role/clusterrole bound within the user's namespace then they at least can't tap resources in other namespaces:

- apiGroups:
  - tap.linkerd.io
  resources:
  - '*'
  verbs:
  - watch

example:

$ linkerd viz tap pod emoji-699d77c79-mr9th
req id=2:0 proxy=in  src=172.16.66.179:45958 dst=172.16.66.189:8080 tls=true :method=POST :authority=emoji-svc.emojivoto:8080 :path=/emojivoto.v1.EmojiService/ListAll
rsp id=2:0 proxy=in  src=172.16.66.179:45958 dst=172.16.66.189:8080 tls=true :status=200 latency=597µs
...
$ linkerd viz tap pod backend-757f4496-cttd9 -n trafficsplit-sample
HTTP error, status Code [403] (pods.tap.linkerd.io "backend-757f4496-cttd9" is forbidden: User "system:serviceaccount:emojivoto:access" cannot watch resource "pods/tap" in API group "tap.linkerd.io" in the namespace "trafficsplit-sample")

The other linkerd viz subcommands still work against all namespaces, though.

[danibaeyens]

Yeah, you are right. Our users have different namespaces and I granted them tap permissions on their rolebinding (not the clusterrolebinding) 😊

About the others, maybe somebody from Buoyant can throw some light.

Thanks for the note 😊😊