Skip to content

Commit 0b45871

Browse files
thossain-nvthossain-nv
committed
chore(doc): Add KeyCloak based Tenancy management doc
1 parent c6ebc75 commit 0b45871

13 files changed

Lines changed: 426 additions & 22 deletions

‎docs/configuration/org-permissions.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
NICo does not maintain its own user directory. Identity, org membership, and role assignments are all managed in the upstream identity provider. The REST API reads role claims from the authentication token on every request. Adding or removing a user is done in the identity provider, not through nicocli.
44

5-
NICo accepts tokens from any OIDC-compatible IdP. The bundled dev Keycloak (deployed by `setup.sh` and documented in the [Quick Start Guide](../getting-started/quick-start.md)) is the recommended starting point and the reference implementation for IdP wiring -- you can use it as-is for evaluation, or model your production IdP setup after it. Configure additional or replacement IdPs via the `issuers` block in `nico-rest-api`'s config; see the [Reference Installation](../getting-started/installation-options/reference-install.md) guide for the configuration surface and the claim mappings NICo expects (org name, display name, role claim).
5+
NICo accepts tokens from any OIDC-compatible IdP. The bundled dev Keycloak (deployed by `setup.sh` and documented in the [Quick Start Guide](../getting-started/quick-start.md)) is the recommended starting point and the reference implementation for IdP wiring -- you can use it as-is for evaluation, or model your production IdP setup after it. Configure additional or replacement IdPs via the `issuers` block in `nico-rest-api`'s config; see [Authentication and Authorization](https://docs.nvidia.com/infra-controller/rest-api-reference/authentication-and-authorization) for the configuration surface, the claim mappings NICo expects (org name, display name, role claim), and the validation rules that apply before rollout.
66

77
## Roles
88

@@ -22,7 +22,7 @@ A single user can hold roles in multiple orgs simultaneously. On dev/service-acc
2222
2. Assign the `TENANT_ADMIN` role at the org level.
2323
3. Have the user authenticate with nicocli and verify: `nicocli user get`
2424

25-
The exact steps depend on your IdP. For the bundled dev Keycloak, this is realm administration in the Keycloak admin console -- create the user, add them to the realm group that maps to the tenant org, and assign the role. See the [Quick Start Guide](../getting-started/quick-start.md) for the realm layout.
25+
The exact steps depend on your IdP. For Keycloak, NICo reads org membership from realm roles named `<orgName>:<ROLE>`, so "add the user to the org" means assigning that realm role, either directly or through a group whose role mapping includes it. [Tenant Management with Keycloak](tenant-management-keycloak.md) has the commands. See the [Quick Start Guide](../getting-started/quick-start.md) for the bundled realm layout.
2626

2727
## Adding a Provider Admin
2828

@@ -34,7 +34,7 @@ Same caveat as above -- this is an IdP admin task, not a nicocli operation.
3434

3535
## Verifying Your Identity
3636

37-
```
37+
```bash
3838
nicocli user get
3939
```
4040

@@ -63,7 +63,7 @@ NICo does not expose a "list users in this tenant" endpoint. Use the IdP's admin
6363

6464
For audit purposes, NICo's audit log records which user performed each operation:
6565

66-
```
66+
```bash
6767
nicocli tui
6868
> audit list
6969
```

‎docs/configuration/tenant-management-keycloak.md‎

Lines changed: 327 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,327 @@
1+
# Tenant Management with Keycloak
2+
3+
Keycloak-side administration for onboarding a Tenant: creating the realm roles that
4+
NICo reads as org membership, creating the Tenant's identity, and granting the
5+
privileged Tenant capability.
6+
7+
[Tenant Management](tenant_management.md) covers the NICo side of the Day 1 workflow
8+
with `nicocli`, and states that meeting its role and membership conditions is an
9+
identity-provider task. This page is that task, for deployments running Keycloak.
10+
11+
NICo never writes to Keycloak. It reads role claims out of each request's token and
12+
derives org membership from them, so every step below is performed against Keycloak
13+
directly and none of it has a NICo API equivalent.
14+
15+
## Before You Start
16+
17+
You should already have:
18+
19+
- A NICo deployment with `keycloak.enabled: true`, which is the default for
20+
`helm-prereqs/setup.sh` installs.
21+
- Keycloak realm administrator credentials. The bundled dev instance uses
22+
`admin` / `admin`.
23+
- The `PROVIDER_ADMIN` role in your own org, for the NICo-side steps.
24+
25+
Confirm the deployment is in Keycloak mode and read the values you will need:
26+
27+
```bash
28+
kubectl -n nico-rest get configmap nico-rest-api-config \
29+
-o jsonpath='{.data.config\.yaml}' | grep -A 8 '^keycloak:'
30+
```
31+
32+
A `setup.sh` install reports:
33+
34+
```yaml
35+
keycloak:
36+
baseURL: http://keycloak.nico-rest:8082
37+
clientID: nico-rest
38+
clientSecretPath: /var/secrets/keycloak/client-secret
39+
enabled: true
40+
externalBaseURL: http://keycloak.nico-rest:8082
41+
realm: nico
42+
serviceAccount: true
43+
```
44+
45+
Three of these govern the rest of this page.
46+
47+
`realm` is the only realm NICo reads. A single `nico-rest-api` release accepts tokens
48+
from exactly one realm, so the Provider org and every Tenant org are realm roles inside
49+
`nico`. There is no list-valued realm setting.
50+
51+
`externalBaseURL` is the issuer prefix NICo validates against the token's `iss` claim.
52+
When it is an in-cluster address, as it is above, **tokens must be requested from inside
53+
the cluster**. A token fetched through a port-forward to `localhost` carries a
54+
non-matching `iss` and is rejected with `401`.
55+
56+
`serviceAccount: true` permits `client_credentials` tokens. Leave it enabled if the
57+
Tenant will authenticate as a service account rather than as a user.
58+
59+
If `keycloak.enabled` is `false`, your deployment uses the `issuers` block instead and
60+
this page does not apply. Onboarding a Tenant is then a configuration change rather than
61+
realm administration, and the recipe is the "Provider with Multiple Tenant IdPs" example in
62+
[Authentication and Authorization](https://docs.nvidia.com/infra-controller/rest-api-reference/authentication-and-authorization).
63+
64+
That page is also the authoritative field reference for the `keycloak` block itself, so use
65+
it when you need the meaning of a setting rather than the procedure for using it.
66+
67+
## How NICo Derives Orgs and Roles
68+
69+
NICo reads the `realm_access.roles` claim and splits each realm role on a colon. The
70+
part before the colon is the org name and the part after it is the role:
71+
72+
```text
73+
acme-corp:TENANT_ADMIN -> org "acme-corp", role TENANT_ADMIN
74+
acme-infra:PROVIDER_ADMIN -> org "acme-infra", role PROVIDER_ADMIN
75+
```
76+
77+
Five properties of this mapping matter when you create roles.
78+
79+
- **Exactly one colon.** A role with none or with two is discarded without an error,
80+
which is why realm roles such as `admin` and `user` have no effect in NICo.
81+
- **Org names are lowercased.** Use a lowercase org name, and use the same value in the
82+
`{org}` path segment of every request and in `api.org` in `~/.nico/config.yaml`.
83+
- **Both role spellings are accepted.** `TENANT_ADMIN` and the legacy prefixed forms
84+
`NICO_TENANT_ADMIN` and `FORGE_TENANT_ADMIN` all match. The bundled realm uses the
85+
prefixed form. Use the unprefixed form for new roles.
86+
- **Groups work indirectly.** Roles inherited from a Keycloak group are present in
87+
`realm_access.roles`, so assigning the realm role to a group and adding users to that
88+
group is equivalent to assigning it to each user.
89+
- **Role changes are cached for up to one minute.** NICo stores the derived org data on
90+
the user record and refreshes it when it is older than that, so a role added in
91+
Keycloak takes effect on a request made after the cache expires.
92+
93+
The three roles and what each grants are listed in
94+
[Organization & Permissions](org-permissions.md). A Tenant needs `TENANT_ADMIN` in its
95+
own org. A Provider needs `PROVIDER_ADMIN` in the Provider org, because inviting a
96+
Tenant, creating Allocations, and granting capabilities are all Provider-side
97+
operations.
98+
99+
Human users additionally need an `oidc_id` user attribute, which is the key NICo uses to
100+
locate or create the user record. The bundled `nico-rest` client already publishes it
101+
through an `oidc-usermodel-attribute-mapper`, so no mapper work is required, but the
102+
attribute has to be set on each user. Service accounts do not need it, because NICo uses
103+
the token's `sub` claim when a `client_id` claim is present.
104+
105+
## Opening an Admin Session
106+
107+
The Keycloak Admin CLI ships inside the container image. Confirm it is present:
108+
109+
```bash
110+
kubectl -n nico-rest exec deployment/keycloak -- \
111+
bash -c 'test -x /opt/keycloak/bin/kcadm.sh && echo FOUND || echo MISSING'
112+
```
113+
114+
Open a shell and authenticate against the container's local listener on port `8080`:
115+
116+
```bash
117+
kubectl -n nico-rest exec -it deployment/keycloak -- bash
118+
119+
/opt/keycloak/bin/kcadm.sh config credentials \
120+
--server http://localhost:8080 --realm master \
121+
--user admin --password admin
122+
```
123+
124+
The admin console is an alternative for operators who prefer a UI. The bundled instance
125+
publishes no ingress, so reach it with
126+
`kubectl -n nico-rest port-forward svc/keycloak 8082:8082` and open
127+
`http://localhost:8082`. Using the console for administration is fine; the in-cluster
128+
restriction described above applies only to token requests.
129+
130+
## Onboarding a Tenant Org
131+
132+
The examples use org `acme-corp` for the Tenant and realm `nico`.
133+
134+
### Create the Tenant's realm role
135+
136+
```bash
137+
/opt/keycloak/bin/kcadm.sh create roles -r nico \
138+
-s name=acme-corp:TENANT_ADMIN \
139+
-s 'description=NICo Tenant Administrator for acme-corp'
140+
```
141+
142+
### Create the Tenant's identity
143+
144+
Choose one of the two options below.
145+
146+
**Option A, a service-account client.** Suited to automation, and the pattern the
147+
bundled `ncx-service` client uses. Requires `serviceAccount: true` in the NICo config.
148+
149+
```bash
150+
/opt/keycloak/bin/kcadm.sh create clients -r nico \
151+
-s clientId=acme-corp-service \
152+
-s enabled=true \
153+
-s publicClient=false \
154+
-s serviceAccountsEnabled=true \
155+
-s standardFlowEnabled=false \
156+
-s directAccessGrantsEnabled=false \
157+
-s secret=REPLACE_WITH_A_GENERATED_SECRET
158+
159+
/opt/keycloak/bin/kcadm.sh add-roles -r nico \
160+
--uusername service-account-acme-corp-service \
161+
--rolename acme-corp:TENANT_ADMIN
162+
```
163+
164+
**Option B, a human user.** The bundled realm ships no human users, so create one, set
165+
`oidc_id`, and assign the role. Interactive login goes through the `nico-rest` client,
166+
which already has the standard flow enabled.
167+
168+
```bash
169+
/opt/keycloak/bin/kcadm.sh create users -r nico \
170+
-s username=tenant-admin@acme-corp.example \
171+
-s email=tenant-admin@acme-corp.example \
172+
-s emailVerified=true \
173+
-s enabled=true \
174+
-s 'attributes.oidc_id=["acme-corp-admin-001"]'
175+
176+
/opt/keycloak/bin/kcadm.sh set-password -r nico \
177+
--username tenant-admin@acme-corp.example --new-password 'REPLACE_ME'
178+
179+
/opt/keycloak/bin/kcadm.sh add-roles -r nico \
180+
--uusername tenant-admin@acme-corp.example \
181+
--rolename acme-corp:TENANT_ADMIN
182+
```
183+
184+
Any value for `oidc_id` works as long as it is unique within the realm and stable for
185+
the life of the user. Changing it later makes NICo treat the login as a new user.
186+
187+
Exit the pod shell when finished.
188+
189+
### Verify the token maps to the org
190+
191+
Request a token from inside the cluster. For Option A:
192+
193+
```bash
194+
TENANT_TOKEN=$(kubectl run -i --rm --restart=Never \
195+
--image=curlimages/curl "curl-tenant-$$" -n nico-rest --quiet -- \
196+
-sf -X POST \
197+
"http://keycloak.nico-rest:8082/realms/nico/protocol/openid-connect/token" \
198+
-H "Content-Type: application/x-www-form-urlencoded" \
199+
-d "grant_type=client_credentials&client_id=acme-corp-service&client_secret=REPLACE_WITH_A_GENERATED_SECRET" \
200+
2>/dev/null | python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])')
201+
```
202+
203+
`helm-prereqs/keycloak/get-token.sh` does the same thing for the bundled `ncx-service`
204+
client and is a working reference for the pattern.
205+
206+
Then confirm NICo resolves the identity, using a port-forward for the API call:
207+
208+
```bash
209+
kubectl -n nico-rest port-forward svc/nico-rest-api 8388:8388 &
210+
211+
curl -sS "http://localhost:8388/v2/org/acme-corp/nico/user/current" \
212+
-H "Authorization: Bearer $TENANT_TOKEN"
213+
```
214+
215+
A `200` response means the issuer validated, a realm role parsed into org
216+
`acme-corp`, and the user record exists. Decode the token payload if it does not, then
217+
check that `realm_access.roles` contains `acme-corp:TENANT_ADMIN`:
218+
219+
```bash
220+
echo "$TENANT_TOKEN" | cut -d. -f2 \
221+
| awk '{l=length($0)%4; if(l) $0=$0 substr("====",1,4-l)}1' \
222+
| base64 --decode | python3 -m json.tool
223+
```
224+
225+
## Completing the Setup in NICo
226+
227+
With the realm role in place, the remaining steps are the standard flow in
228+
[Tenant Management](tenant_management.md). Two points are specific to this path.
229+
230+
**The Tenant must call `nicocli tenant current` before accepting an invitation.** That
231+
call creates the Tenant record and links any invitation matching the org name. Until it
232+
runs, reading the Tenant Account returns `403` and accepting it returns
233+
`404 Org does not have tenant`, because the account's `tenantId` is still empty.
234+
235+
**A Tenant should not call `nicocli service-account current`.** In a deployment with
236+
`serviceAccount: true`, that endpoint creates an Infrastructure Provider, a Tenant, and
237+
an already-accepted Tenant Account for the calling org, which makes the org its own
238+
Provider instead of a Tenant of yours. It is the bootstrap path for the Provider org, not
239+
for an invited Tenant.
240+
241+
The order is:
242+
243+
1. Provider Admin creates the invitation. See
244+
[Creating the Link](tenant_management.md#creating-the-link).
245+
2. Tenant Admin runs `nicocli tenant current` to create the Tenant and link the
246+
invitation.
247+
3. Tenant Admin accepts, moving the account to `Ready`. See
248+
[Accepting the Invitation](tenant_management.md#accepting-the-invitation-tenant-side).
249+
4. Provider Admin creates one Allocation per Site, which is what gives the Tenant
250+
capacity. See [Assigning Resources with Allocations](tenant_management.md#assigning-resources-with-allocations).
251+
252+
## Granting the Privileged Tenant Capability
253+
254+
Nothing about `targetedInstanceCreation` is Keycloak-specific. It is not a realm role and
255+
it has no claim: a Provider Admin grants it on the Tenant Account after the Tenant
256+
accepts, using `siteCapabilities`.
257+
258+
```bash
259+
nicocli tenant-account update \
260+
--data '{"siteCapabilities":[{"siteIds":[],"targetedInstanceCreation":true}]}' \
261+
<account-id>
262+
```
263+
264+
See [Granting Targeted Instance Creation](tenant_management.md#granting-targeted-instance-creation)
265+
for the payload rules, the per-Site override behavior, how the effective value resolves,
266+
plus how to read the current value.
267+
268+
The distinction worth keeping straight on this page is that Keycloak decides **who the
269+
caller is and which org they act in**, while the Tenant Account decides **what that org is
270+
allowed to do**. A realm role of `acme-corp:TENANT_ADMIN` makes someone a Tenant Admin for
271+
`acme-corp`; it does not make them privileged. Adding a role such as
272+
`acme-corp:PROVIDER_ADMIN` does not grant the capability either, it makes the org its own
273+
Provider.
274+
275+
## Making Realm Changes Permanent
276+
277+
Changes made with `kcadm.sh` or the admin console live in Keycloak's database. They
278+
survive pod restarts, but they are not represented in your configuration, so a rebuilt
279+
realm loses them.
280+
281+
The bundled Keycloak imports `helm-prereqs/keycloak/realm-configmap.yaml` with
282+
`--import-realm`, which imports only a realm that does not already exist. **Editing that
283+
file and re-running `setup.sh` does not change an existing realm.** To fold a Tenant org
284+
into the reproducible baseline, add its roles and identities to the file and re-import
285+
from a clean state:
286+
287+
```bash
288+
helm-prereqs/keycloak/clean.sh
289+
helm-prereqs/keycloak/setup.sh
290+
```
291+
292+
`clean.sh` drops the `keycloak` database and also deletes the `keycloak-client-secret`
293+
Secret, which the `nico-rest-common` sub-chart owns. Re-run the `nico-rest` Helm upgrade
294+
to restore it, then restart the API so it re-reads the signing keys from the rebuilt
295+
realm:
296+
297+
```bash
298+
kubectl -n nico-rest rollout restart deployment/nico-rest-api
299+
```
300+
301+
Restarting the API while Keycloak is unavailable leaves it running with Keycloak
302+
authentication inactive, so confirm Keycloak is ready first.
303+
304+
## Troubleshooting
305+
306+
| Symptom | Cause | Resolution |
307+
|---------|-------|-----------|
308+
| `401 Invalid authorization token in request` | Token `iss` does not match `externalBaseURL` plus the realm. Usually a token fetched over a port-forward | Request the token from inside the cluster |
309+
| `401 Service accounts are not enabled` | Token carries a `client_id` claim but `keycloak.serviceAccount` is `false` | Enable `serviceAccount` in the values and upgrade, or use a user token |
310+
| `403 User does not have any roles assigned` | No realm role parsed into an org. Usually a role name without exactly one colon | Check `realm_access.roles` in the decoded token |
311+
| `403 Requested organization not found in token claims` | The `{org}` path segment does not match any role prefix. Often a case mismatch | Use the lowercase org name in the path and in `api.org` |
312+
| `403 User does not have Tenant Admin role with org` | Role parsed, but it is not `TENANT_ADMIN` | Assign `acme-corp:TENANT_ADMIN` and retry after the one-minute cache expires |
313+
| `404 Org does not have tenant` when accepting | `nicocli tenant current` has not been run for the Tenant org | Run it, then accept |
314+
| `400 Tenant Account status is not Invited` | The account is already `Ready` | No action needed, the invitation was already accepted |
315+
| A role added in Keycloak has no effect | Org data cached on the user record | Retry after one minute |
316+
| Realm edits to `realm-configmap.yaml` do not appear | `--import-realm` skips an existing realm | Apply with `kcadm.sh`, or re-import from clean |
317+
318+
## Related Documentation
319+
320+
- [Tenant Management](tenant_management.md), the NICo-side Day 1 workflow with `nicocli`
321+
- [Authentication and Authorization](https://docs.nvidia.com/infra-controller/rest-api-reference/authentication-and-authorization),
322+
the Day 0 auth configuration reference for both the `keycloak` and `issuers` modes
323+
- [Organization & Permissions](org-permissions.md), the role model and what each role grants
324+
- [Quick Start Guide](../getting-started/quick-start.md), deployment and token acquisition
325+
for the bundled realm
326+
- [Reference Installation](../getting-started/installation-options/reference-install.md),
327+
deployment-side authentication wiring including the `issuers` alternative

0 commit comments

Comments
 (0)