Skip to content
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/configuration/org-permissions.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

NICo does not maintain its own user directory. Identity, org membership, and role assignments are all managed in the upstream identity provider. The REST API reads role claims from the authentication token on every request. Adding or removing a user is done in the identity provider, not through nicocli.

NICo accepts tokens from any OIDC-compatible IdP. The bundled dev Keycloak (deployed by `setup.sh` and documented in the [Quick Start Guide](../getting-started/quick-start.md)) is the recommended starting point and the reference implementation for IdP wiring -- you can use it as-is for evaluation, or model your production IdP setup after it. Configure additional or replacement IdPs via the `issuers` block in `nico-rest-api`'s config; see the [Reference Installation](../getting-started/installation-options/reference-install.md) guide for the configuration surface and the claim mappings NICo expects (org name, display name, role claim).
NICo accepts tokens from any OIDC-compatible IdP. The bundled dev Keycloak (deployed by `setup.sh` and documented in the [Quick Start Guide](../getting-started/quick-start.md)) is the recommended starting point and the reference implementation for IdP wiring -- you can use it as-is for evaluation, or model your production IdP setup after it. Configure additional or replacement IdPs via the `issuers` block in `nico-rest-api`'s config; see [Authentication and Authorization](https://docs.nvidia.com/infra-controller/rest-api-reference/authentication-and-authorization) for the configuration surface, the claim mappings NICo expects (org name, display name, role claim), and the validation rules that apply before rollout.

## Roles

Expand All @@ -22,7 +22,7 @@ A single user can hold roles in multiple orgs simultaneously. On dev/service-acc
2. Assign the `TENANT_ADMIN` role at the org level.
3. Have the user authenticate with nicocli and verify: `nicocli user get`

The exact steps depend on your IdP. For the bundled dev Keycloak, this is realm administration in the Keycloak admin console -- create the user, add them to the realm group that maps to the tenant org, and assign the role. See the [Quick Start Guide](../getting-started/quick-start.md) for the realm layout.
The exact steps depend on your IdP. For Keycloak, NICo reads org membership from realm roles named `<orgName>:<ROLE>`, so "add the user to the org" means assigning that realm role, either directly or through a group whose role mapping includes it. [Tenant Management with Keycloak](tenant-management-keycloak.md) has the commands. See the [Quick Start Guide](../getting-started/quick-start.md) for the bundled realm layout.

## Adding a Provider Admin

Expand All @@ -34,7 +34,7 @@ Same caveat as above -- this is an IdP admin task, not a nicocli operation.

## Verifying Your Identity

```
```bash
nicocli user get
```

Expand Down Expand Up @@ -63,7 +63,7 @@ NICo does not expose a "list users in this tenant" endpoint. Use the IdP's admin

For audit purposes, NICo's audit log records which user performed each operation:

```
```bash
nicocli tui
> audit list
```
Expand Down
371 changes: 371 additions & 0 deletions docs/configuration/tenant-management-keycloak.md

Large diffs are not rendered by default.

82 changes: 77 additions & 5 deletions docs/configuration/tenant_management.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ This guide assumes you have completed the [Quick Start Guide](../getting-started
- At least one site registered and in `Registered` status, with machines discovered and available for allocation.
- `nicocli` installed (`make nico-cli` from the `rest-api/` directory of the `infra-controller` repo) and reachable on `$PATH`.

If you plan to enable SPIFFE JWT-SVID **machine identity**, complete [Day 0 Machine Identity](../getting-started/installation-options/day0-machine-identity.md) before provisioning instances, then configure per-org identity after tenants exist — see [Machine Identity](machine_identity.md).
If you plan to enable SPIFFE JWT-SVID **machine identity**, complete [Day 0 Machine Identity](../getting-started/installation-options/day0-machine-identity.md) before provisioning instances, then configure per-org identity after tenants exist. See [Machine Identity](machine_identity.md).

> **Note on CLI naming**: Older docs reference `carbidecli` (built via `make carbide-cli`). It's the same source under a previous name. This guide uses `nicocli` (built via `make nico-cli`) consistently.

Expand All @@ -32,6 +32,8 @@ NICo's authorization model has three roles, all managed in the upstream identity

A single user can hold roles in multiple orgs simultaneously. On dev/service-account orgs, one user typically holds both Provider Admin and Tenant Admin in the same org.

These roles are not settable through NICo. They are assigned in the identity provider and read from each request's token, which makes them a Day 0 concern that has to be in place before any of the Day 1 steps below will work. [Authentication and Authorization](https://docs.nvidia.com/infra-controller/rest-api-reference/authentication-and-authorization) is the authoritative reference for that configuration, covering both the `issuers` and `keycloak` modes, the four claim-mapping recipes, and the deployment surfaces where each value belongs.

### Authentication

The Quick Start covers `nicocli login` end-to-end for Keycloak-backed deployments (the default for `setup.sh` installs). For other identity providers and for the static-token / token-command flows useful in automation, see the nicocli reference guide.
Expand Down Expand Up @@ -73,7 +75,9 @@ nicocli tui
- The authenticated user must be a member of the organization specified in the nicocli config (`api.org`).
- The user must hold the Tenant Admin role within that org.

If either condition is not met, the API returns HTTP 403. NICo trusts whatever the IdP says in the token's claims, so getting these conditions met is an IdP administration task -- it is not done through nicocli or the NICo API. The [Quick Start Guide](../getting-started/quick-start.md) walks through the bundled Keycloak reference implementation (a dev Keycloak deployed by `setup.sh` with a pre-loaded realm), which is the simplest path for first-time setup. For production, point NICo at any OIDC-compatible IdP (Keycloak, Okta, Auth0, your existing enterprise IdP) by configuring the `issuers` block in `nico-rest-api`'s config -- see [`getting-started/installation-options/reference-install.md`](../getting-started/installation-options/reference-install.md) for the deployment-side wiring.
If either condition is not met, the API returns HTTP 403. NICo trusts whatever the IdP says in the token's claims, so getting these conditions met is an IdP administration task -- it is not done through nicocli or the NICo API. The [Quick Start Guide](../getting-started/quick-start.md) walks through the bundled Keycloak reference implementation (a dev Keycloak deployed by `setup.sh` with a pre-loaded realm), which is the simplest path for first-time setup. For production, point NICo at any OIDC-compatible IdP (Keycloak, Okta, Auth0, your existing enterprise IdP) by configuring the `issuers` block in `nico-rest-api`'s config -- see [Authentication and Authorization](https://docs.nvidia.com/infra-controller/rest-api-reference/authentication-and-authorization) for the deployment-side wiring. Its "Provider with Multiple Tenant IdPs" example is the recipe for one provider org serving tenants that each authenticate through their own identity provider.

For Keycloak deployments, [Tenant Management with Keycloak](tenant-management-keycloak.md) covers the realm-side steps in full: the realm role naming convention NICo reads as org membership, creating the Tenant's identity, and granting the privileged Tenant capability.

### Worked Example

Expand All @@ -96,7 +100,7 @@ $ nicocli tenant current
| `id` | UUID identifier for the tenant, used in all subsequent API calls |
| `org` | Organization name (matches your config `api.org`) |
| `orgDisplayName` | Human-readable name pulled from the IdP's org metadata |
| `capabilities.targetedInstanceCreation` | Whether this tenant can specify a particular machine ID when creating instances. Set during initial tenant creation: lazy-create via `tenant current` typically leaves it `false`; the service-account bootstrap path (`service-account current`) sets it `true` for self-tenants. |
| `capabilities.targetedInstanceCreation` | **Deprecated, removal scheduled for October 1, 2026.** A read-only aggregate across the tenant's `Ready` tenant accounts, not a setting on the tenant. It is `true` only when every such account enables the capability and no site override disables it, and it is absent rather than `false` when disabled: the field is omitted, and on the `tenantSummary` objects embedded in other resources `capabilities` is always `{}`. A Provider Admin configures the capability per tenant account, and optionally per site. See [Granting Targeted Instance Creation](#granting-targeted-instance-creation). |

### Verifying the Tenant

Expand Down Expand Up @@ -196,6 +200,72 @@ $ nicocli tenant-account update --data '{}' <account-id>
Error: API error 400: Tenant Account status is not Invited
```

### Granting Targeted Instance Creation

`targetedInstanceCreation` is the privileged tenant capability. A tenant that has it can:

- Create an instance against a specific machine ID, or narrow placement with a machine label selector.
- Set `isRepairTenant: true` when releasing an instance, which is what the repair tenant workflow requires.
- Read a set of otherwise provider-only resources, including machines, machine health, SKUs, racks, trays, and expected machines, at the sites of a provider it holds a `Ready` tenant account with, wherever the capability is effective. A site override that disables it also removes these reads at that site.
- Receive alternative VPC routing profiles from `tenant/current/routing-profile`.

A Provider Admin configures it per tenant account with the `siteCapabilities` field. This is
the only supported way to grant it, and it applies to regular tenants as well as
service-account orgs:

```bash
nicocli tenant-account update \
--data '{"siteCapabilities":[{"siteIds":[],"targetedInstanceCreation":true}]}' \
<account-id>
```

#### Payload rules

- Exactly one entry must have an empty or omitted `siteIds`. That entry sets the account-level default.
- Any further entry lists site IDs that override the default. Each site must already be associated with the tenant, which happens when the tenant's first allocation at that site is created.
- No site ID may appear in more than one entry.
- `targetedInstanceCreation` is required on every entry.
- The request must not also carry `tenantContactId`, which is the invitation-acceptance field. Sending both returns HTTP 400.

Updates use replace semantics. A per-site override whose site ID is omitted from a later
payload is cleared. To leave the capability on everywhere except one site, send both
entries together:

```json
{
"siteCapabilities": [
{"siteIds": [], "targetedInstanceCreation": true},
{"siteIds": ["<site-uuid>"], "targetedInstanceCreation": false}
]
}
```

#### How the effective value resolves

Three inputs decide whether the capability is in force for a tenant at a given site:

1. The tenant account must be `Ready`. Setting the capability on an `Invited` account grants nothing until the tenant accepts.
2. The account-level default applies where no site override exists.
3. A site override, when present, wins over the account default.

Because the account is per provider, a tenant with accounts at two providers can be
privileged at one and not the other.

#### Reading the current value

`nicocli tenant-account list` returns `siteCapabilities` and is the source of truth:

```bash
nicocli tenant-account list --tenant-id <tenant-uuid>
```

> **Do not read `capabilities.targetedInstanceCreation` from `nicocli tenant current`.** It is
> a deprecated read-only aggregate, scheduled for removal on **October 1, 2026**. It reports
> `true` only when every `Ready` tenant account enables the capability and no site override
> disables it, and it is omitted rather than returned as `false` otherwise, so a client cannot
> branch on a boolean here. On the `tenantSummary` objects embedded in other resources it is
> always omitted, leaving `"capabilities": {}`. Use `tenant-account list` instead.

## Instance Types

Instance types define hardware classes -- they map a named category (like "GB200-NVL72" or "DGX-H100") to a set of physical machines with specific GPU, CPU, and network configurations. Before a tenant can create compute allocations, the instance types must exist and machines must be associated with them.
Expand Down Expand Up @@ -518,7 +588,7 @@ An instance in NICo is a bare-metal machine assigned to a tenant within a VPC. C
| `--name` | yes | |
| `--tenant-id` | yes | Owning tenant -- often missed in older docs |
| `--vpc-id` | yes | Parent VPC |
| `--machine-id` | no | Pin to a specific machine (requires `targetedInstanceCreation: true` on the tenant) |
| `--machine-id` | no | Pin to a specific machine (requires [targeted instance creation](#granting-targeted-instance-creation) on the tenant's account for that site) |
| `--instance-type-id` | no | Pick from the pool of machines of this type (alternative to `--machine-id`) |
| `--operating-system-id` | no | OS for PXE provisioning |
| `--allow-unhealthy-machine` | no | Override health checks |
Expand Down Expand Up @@ -546,7 +616,7 @@ nicocli instance create --data-file - <<'EOF'
EOF
```

If you want to target a specific machine instead, replace `instanceTypeId` with `machineId`. Machine targeting requires the tenant to have `capabilities.targetedInstanceCreation: true`.
If you want to target a specific machine instead, replace `instanceTypeId` with `machineId`. Machine targeting requires the tenant's account with the site's provider to be `Ready` and to have the capability enabled for that site. See [Granting Targeted Instance Creation](#granting-targeted-instance-creation).

TUI flow:

Expand Down Expand Up @@ -962,6 +1032,8 @@ Flag-first ordering -- always put flags before positional args.

## Related Documentation

- [Authentication and Authorization](https://docs.nvidia.com/infra-controller/rest-api-reference/authentication-and-authorization) -- Day 0 auth configuration: `issuers` and `keycloak` modes, claim mappings, validation rules
- [Tenant Management with Keycloak](tenant-management-keycloak.md) -- Realm-side steps for onboarding a tenant on Keycloak deployments
- [Network Isolation](network-isolation.md) -- Per-plane tenant isolation (Ethernet, InfiniBand, NVLink)
- [Organization & Permissions](org-permissions.md) -- IdP-managed roles and user setup
- [Quick Start Guide](../getting-started/quick-start.md) -- NICo deployment and Day Zero walkthrough
Expand Down
4 changes: 2 additions & 2 deletions docs/getting-started/quick-start.md
Original file line number Diff line number Diff line change
Expand Up @@ -512,8 +512,8 @@ This `GET` endpoint lazily initializes the org on first call as follows:

1. Checks if service account is enabled in the auth config
2. Creates an **InfrastructureProvider** for the org if one doesn't exist
3. Creates a **Tenant** with targeted instance creation enabled if one doesn't exist
4. Creates a **TenantAccount** linking the provider and tenant if one doesn't exist
3. Creates a **Tenant** for the org if one doesn't exist
4. Creates a **TenantAccount** linking the provider and tenant if one doesn't exist, already in `Ready` status with the `targetedInstanceCreation` capability enabled
5. Returns the service account status with the provider and tenant IDs

Without this call, site operations return 404. Subsequent calls are read-only.
Expand Down
2 changes: 2 additions & 0 deletions docs/index.yml
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,8 @@ navigation:
path: manuals/networking/infiniband_partitioning.md
- page: Tenant Management
path: configuration/tenant_management.md
- page: Tenant Management with Keycloak
path: configuration/tenant-management-keycloak.md
- page: Image-Based Operating Systems
path: configuration/image-based-operating-systems.md
- page: Organization & Permissions
Expand Down
2 changes: 1 addition & 1 deletion docs/manuals/repair/online_repair.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ If online repair cannot fix the issue, clear online repair first and then releas

This page is intended for tenant admins and platform operators writing tenant-facing runbooks.

The caller must have access to the Infra Controller REST API through an API profile such as `nico-stg`. The online repair operation is allowed for provider admins and privileged tenant admins. In tenant workflows, this means the tenant must have the required privileged capability for repair operations, such as targeted instance creation access.
The caller must have access to the Infra Controller REST API through an API profile such as `nico-stg`. The online repair operation is allowed for provider admins and privileged tenant admins. In tenant workflows, this means the tenant must have the required privileged capability for repair operations, such as effective targeted instance creation at the site. See [Granting Targeted Instance Creation](../../configuration/tenant_management.md#granting-targeted-instance-creation) for what makes it effective.
Comment thread
thossain-nv marked this conversation as resolved.
Outdated

## What Online Repair Does

Expand Down
8 changes: 5 additions & 3 deletions docs/manuals/repair/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,13 @@ In a multi-Tenant Site, NICo allows certain Tenants to be designated as privileg
- Update the labels of a Machine to indicate repair related metadata
- Create a new Instance by specifying a Machine ID

Tenant privileges are described in terms of Capabilities. The capabilities of a Tenant can be viewed by calling the [`GET /v2/org/{org}/nico/tenant/current` REST API endpoint](https://docs.nvidia.com/infra-controller/rest-api-reference/api-reference/tenant/get-current-tenant) and inspecting the `capabilities` field. To elevate a Tenants privileges so they can access the above endpoints, the Tenant must have the `targetedInstanceCreation` capability enabled.
Tenant privileges are described in terms of Capabilities. A Tenant is privileged at a Site when its Tenant Account with that Site's Provider is `Ready` and `targetedInstanceCreation` is effective there, meaning the account default enables it and no Site override turns it off.

In service account mode, the `targetedInstanceCreation` capability is granted to Service Account Tenant when [`GET /v2/org/{org}/nico/service-account/current` REST API endpoint](https://docs.nvidia.com/infra-controller/rest-api-reference/api-reference/service-account/get-current-service-account) is called.
A Provider Admin grants the capability by setting `siteCapabilities` on the Tenant Account, through [`PATCH /v2/org/{org}/nico/tenant/account/{accountId}`](https://docs.nvidia.com/infra-controller/rest-api-reference/api-reference/tenant-account/update-tenant-account). This works for regular Tenants as well as Service Account orgs, and the capability can be set as an account-wide default or overridden per Site. For the payload rules, the per-Site override behavior, and how the effective value resolves, see [Granting Targeted Instance Creation](../../configuration/tenant_management.md#granting-targeted-instance-creation).

At present turning this capability on for regular Tenants (who are not part of a Service Account org) is not supported via the REST API. However the feature is in active development, relevant issue can be tracked [here](https://github.com/NVIDIA/infra-controller/issues/2104).
To read the current value, call [`GET /v2/org/{org}/nico/tenant/account`](https://docs.nvidia.com/infra-controller/rest-api-reference/api-reference/tenant-account/get-all-tenant-account) and inspect `siteCapabilities`. Do not use the `capabilities` field on `GET /tenant/current`: it is a deprecated read-only aggregate scheduled for removal on October 1, 2026, and it reports `false` on embedded Tenant summaries regardless of the real value.
Comment thread
thossain-nv marked this conversation as resolved.
Outdated

In Service Account mode, [`GET /v2/org/{org}/nico/service-account/current`](https://docs.nvidia.com/infra-controller/rest-api-reference/api-reference/service-account/get-current-service-account) creates a `Ready` Tenant Account with the capability already enabled, so a Service Account org is privileged without a separate grant.

NOTE: Privileged Tenants still need Network Allocations from Provider in order to create Instances.

Expand Down
Loading
Loading