Skip to content

Commit cd3031e

Browse files
micromaomaomicromaomao
committed
Add various agent instructions for working with confidential containers.
This was generated via a long prompt plus ingesting a bunch of previous PRs doing confidential fixes, then manually curated by me. Assisted-by: GitHub Copilot:claude-opus-4.7 Signed-off-by: Tingmao Wang <tingmaowang@microsoft.com>
1 parent aad6303 commit cd3031e

3 files changed

Lines changed: 458 additions & 0 deletions

File tree

‎.github/instructions/about-confidential-containers.instructions.md‎

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
---
2+
description: Use this when working with confidential containers (C-LCOW or C-WCOW).
3+
---
4+
5+
# Confidential Containers (C-LCOW and C-WCOW)
6+
7+
C-LCOW (Confidential Linux Containers on Windows) and C-WCOW (Confidential
8+
Windows Containers on Windows) are containers running in an SNP UVM (a trusted
9+
execution environment backed by AMD SEV-SNP). Inside the UVM, all sensitive
10+
operations coming from the (untrusted) host are gated by a Rego-based security
11+
policy. This functionality backs C-ACI, the confidential SKU of ACI.
12+
13+
## Trust model
14+
15+
- The UVM is launched with a "host data" field set to the SHA-256 of the
16+
customer's security policy. When the GCS (on Linux) or gcs-sidecar (on
17+
Windows) starts, it receives the policy from the host via an RPC (a
18+
`ModifySettings` request for `guestresource.ResourceTypeSecurityPolicy`).
19+
That RPC is itself untrusted — the host could send any policy — but before
20+
the GCS constructs a `regoEnforcer` from it, it requests an SNP attestation
21+
report from the PSP and verifies that the SHA-256 of the received policy
22+
matches the `host_data` field baked into the report at launch. If the
23+
hashes don't match, the policy is rejected. Until this RPC has been
24+
processed, the enforcer in use is the `ClosedDoorSecurityPolicyEnforcer`,
25+
so nothing meaningful can happen.
26+
- After that point, **every** subsequent request from the host is treated as
27+
untrusted input. The policy describes which containers may run (layer hashes, command,
28+
env, mounts, user, capabilities, seccomp, privileged, etc.), which external
29+
processes may exec, which fragments may be loaded, and so on.
30+
- For C-LCOW the GCS itself performs the requested operation after enforcement
31+
(mount filesystems, spawn processes, invoke runc). For C-WCOW the
32+
gcs-sidecar validates and rewrites the request, then forwards it to the
33+
Windows "inbox GCS" (a built-in C++ component) which performs the operation.
34+
The sidecar is the only enforcement boundary — anything passed unchecked
35+
through to the inbox GCS is effectively trusted.
36+
37+
## Rego modules
38+
39+
The interpreter (see [internal/regopolicyinterpreter](internal/regopolicyinterpreter))
40+
loads several Rego modules into one OPA evaluation:
41+
42+
- `policy` — the customer's policy, generated by tooling from the container
43+
group spec. Declares allowed containers, external processes, fragments, and
44+
a set of base flags (`allow_properties_access`, `allow_unencrypted_scratch`,
45+
`allow_environment_variable_dropping`, `allow_capability_dropping`, …).
46+
Customer policies typically just **delegate** each enforcement point to
47+
`data.framework.<point>`; the policy mainly supplies data.
48+
- `framework` — [pkg/securitypolicy/framework.rego](pkg/securitypolicy/framework.rego),
49+
bundled with GCS. Contains all the actual matching/narrowing logic for
50+
every enforcement point, plus the error-reason rules used to build denial
51+
messages. Has a `version` (see `version_framework`) that the policy's
52+
`framework_version` is compared against; the framework includes backward
53+
compatibility shims (`check_container`, `apply_defaults`, etc.) so newer
54+
GCS can run older policies.
55+
> **The framework must never break existing policies.** Customer
56+
> policies are generated by tooling and bundled with images that may not
57+
> be regenerated for a long time. Any framework change must preserve the
58+
> exact decision for every previously-valid policy: gate new behavior on
59+
> `policy_framework_version` / `policy_api_version`, add an
60+
> `apply_defaults` / `check_*` shim for any new object field, and bump
61+
> `version_framework` / `version_api` accordingly. The same rule applies
62+
> to fragments (`fragment_framework_version`).
63+
- `api` — [pkg/securitypolicy/api.rego](pkg/securitypolicy/api.rego). Declares
64+
every enforcement point, the version it was introduced in, its default
65+
result if missing, and whether `use_framework` is set (so the enforcer can
66+
fall back to `data.framework.<point>` when the policy itself does not
67+
define the rule).
68+
- **Fragments** — additional Rego modules loaded at runtime after doing
69+
signature verification and having the issuer checked by the `load_fragment`
70+
enforcement point. Each fragment lives in its own namespace (parsed from the
71+
required `package <ns>` first line; reserved namespaces `framework`, `api`,
72+
`policy`, `metadata` are rejected — see `parseNamespace`). Fragments can
73+
contribute additional allowed `containers`, `external_processes`, and nested
74+
`fragments`.
75+
76+
## Enforcement point lifecycle
77+
78+
The call path for a host request to a policy decision:
79+
80+
1. The host sends a bridge message. For C-LCOW this lands in the bridge in
81+
[internal/guest/bridge](internal/guest/bridge); for C-WCOW it lands in
82+
[internal/gcs-sidecar](internal/gcs-sidecar).
83+
2. The request is unmarshalled and dispatched, typically into
84+
[internal/guest/runtime/hcsv2/uvm.go](internal/guest/runtime/hcsv2/uvm.go)
85+
(`Host.CreateContainer`, `Host.modifyMappedVirtualDisk`,
86+
`Host.modifyCombinedLayers`, `Host.ExecProcess`, …) on Linux, or
87+
equivalent sidecar handlers on Windows.
88+
3. The handler calls into a `SecurityPolicyEnforcer` method (see
89+
[pkg/securitypolicy/securitypolicyenforcer.go](pkg/securitypolicy/securitypolicyenforcer.go))
90+
such as `EnforceDeviceMountPolicy`, `EnforceOverlayMountPolicy`,
91+
`EnforceCreateContainerPolicyV2`, `EnforceExecInContainerPolicyV2`, etc.
92+
4. The `regoEnforcer` (see
93+
[pkg/securitypolicy/securitypolicyenforcer_rego.go](pkg/securitypolicy/securitypolicyenforcer_rego.go))
94+
builds an `inputData` map for the enforcement point and calls
95+
`policy.enforce(ctx, name, input)`, which queries
96+
`data.policy.<name>` (with fallback to `data.framework.<name>` via
97+
`applyDefaults` when the policy delegates or the rule pre-dates the
98+
policy's `api_version`).
99+
5. The Rego result is either `{"allowed": true, ...extra}` or `{"allowed": false}`.
100+
On deny, the enforcer queries `data.policy.reason` to build a structured
101+
error (`denyWithReason` / `policyDecisionToError`, truncating
102+
`error_objects`, then `input`, then `reason` until under
103+
`maxErrorMessageLength`).
104+
6. Extra outputs are interpreted by the Go side: e.g. `env_list` filters the
105+
env that survives "env dropping", `caps_list` filters capabilities,
106+
`allow_stdio_access` decides whether the container's stdio uses the real
107+
vsock transport or `/dev/null`, `add_module` decides whether a loaded
108+
fragment is kept.
109+
110+
## Metadata and revertable sections
111+
112+
Rego is functional, but enforcement needs state (which devices are mounted,
113+
which overlays exist, which containers have been started, which fragments
114+
have been loaded, fragment parameter sets, …). We invented **metadata** for
115+
this: a Rego rule can return a `metadata` array of operations
116+
(`add`/`update`/`remove` on named maps, plus the newer "set" type) and the
117+
interpreter applies them after a successful query. See `framework.rego`
118+
usage of `data.metadata.devices`, `data.metadata.matches`,
119+
`data.metadata.started`, `data.metadata.overlayTargets`,
120+
`data.metadata.scratch_mounts`, `data.metadata.issuers`,
121+
`data.metadata.fragment_parameters`, etc.
122+
123+
Because metadata mutates real state, mount/unmount handlers wrap their work
124+
in a **revertable section** (`StartRevertableSection` / `Commit` / `Rollback`,
125+
see the methods on `regoEnforcer` and `commitOrRollbackPolicyRevSection` in
126+
`hcsv2/uvm.go`). On any error the metadata is rolled back so it stays in
127+
sync with reality. Any new side-effecting handler **must** use this pattern,
128+
and the underlying operation **must** clean up its own partial state on
129+
failure (the revert only undoes the Rego metadata, not files/mounts/dm
130+
devices). If a clean revert is not possible, there should be a mechanism to
131+
stop further operations to prevent exploits (e.g. the `mountsBroken` mechanism
132+
for mounts).
133+
134+
## Threat-model rule of thumb
135+
136+
Treat **everything** coming from the host as adversarial. When changing
137+
any in-guest code path, work through the checklist in the
138+
[review-confidential-security](../skills/review-confidential-security/SKILL.md)
139+
skill.
140+
141+
## C-WCOW specifics
142+
143+
- The enforcement boundary is [internal/gcs-sidecar](internal/gcs-sidecar),
144+
not the inbox GCS. The sidecar unmarshals the request, runs policy
145+
enforcement, potentially rewrites the payload (e.g. filtering env or ETW
146+
providers), and only then forwards it.
147+
- Anything the sidecar does **not** validate is effectively delegated to the
148+
inbox GCS, which is not part of this repo. New resource types added to the
149+
sidecar dispatcher must come with explicit enforcement, not a pass-through.
150+
- The same Rego enforcer is shared with C-LCOW (`policy.osType` switches
151+
per-input shape inside `EnforceCreateContainerPolicyV2` etc.), so changes
152+
to framework.rego must consider both Linux and Windows code paths.
153+
154+
## Where to look
155+
156+
- Rego: [pkg/securitypolicy/framework.rego](pkg/securitypolicy/framework.rego),
157+
[pkg/securitypolicy/api.rego](pkg/securitypolicy/api.rego),
158+
[pkg/securitypolicy/policy.rego](pkg/securitypolicy/policy.rego)
159+
- Go enforcer: [pkg/securitypolicy/securitypolicyenforcer_rego.go](pkg/securitypolicy/securitypolicyenforcer_rego.go),
160+
[pkg/securitypolicy/securitypolicyenforcer.go](pkg/securitypolicy/securitypolicyenforcer.go)
161+
- Interpreter: [internal/regopolicyinterpreter](internal/regopolicyinterpreter)
162+
- C-LCOW handlers: [internal/guest/runtime/hcsv2/uvm.go](internal/guest/runtime/hcsv2/uvm.go)
163+
- C-WCOW handlers: [internal/gcs-sidecar](internal/gcs-sidecar)
164+
- Bridge sequential mode: [internal/guest/bridge](internal/guest/bridge)
165+
- Tests: `regopolicy_linux_test.go`, `regopolicy_windows_test.go`, `api_test.rego`
166+
167+
## Build tag
168+
169+
The Rego enforcer is gated behind the `rego` Go build tag (see the
170+
`//go:build rego` headers on `securitypolicyenforcer_rego.go` and the
171+
test files). Non-confidential LCOW builds of GCS deliberately do not
172+
include any Rego code. When building or testing any confidential change
173+
you must pass `-tags rego`, e.g. `go build -tags rego ./...` or
174+
`go test -tags rego ./pkg/securitypolicy/...`. Without it, your changes
175+
to the Rego enforcer will not be compiled and the tests will silently
176+
not run.

‎.github/skills/add-enforcement-point/SKILL.md‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
---
2+
name: add-enforcement-point
3+
description: Step-by-step procedure for adding a new Rego enforcement point to the hcsshim confidential containers (C-LCOW / C-WCOW) policy. Use when introducing a new host-driven operation that needs to be gated by the security policy, or when extending an existing enforcement point with new inputs.
4+
---
5+
6+
# Adding a new enforcement point
7+
8+
Before starting, read
9+
[.github/instructions/about-confidential-containers.instructions.md](../../.github/instructions/about-confidential-containers.instructions.md)
10+
for background on the trust model, modules, metadata, and revertable sections.
11+
12+
> **Backward compatibility is non-negotiable.** Customer policies are
13+
> generated by tooling and shipped alongside container images; we do not
14+
> control when they are regenerated. Any change to `framework.rego`,
15+
> `api.rego`, or the container/external_process/fragment object shapes
16+
> **must** continue to evaluate identically for every policy that was
17+
> valid before the change. Concretely: never remove or rename existing
18+
> rules or fields, never change the meaning of an existing input field,
19+
> always gate new framework behavior on `policy_framework_version` /
20+
> `policy_api_version`, and always provide an `apply_defaults` / `check_*`
21+
> shim for new fields so older policies see a sensible default. If your
22+
> change cannot be expressed this way, it is the wrong shape — stop and
23+
> rethink.
24+
25+
## 1. Decide the shape
26+
27+
- **Name** — snake_case, matches the Go method and the Rego rule name
28+
(`mount_device`, `create_container`, `signal_container_process`, …).
29+
- **Inputs** — every field the policy may need to make a decision. Anything
30+
the host controls is fair game and should be passed. Things GCS itself
31+
derives (e.g. dynamically added devices for privileged containers, or
32+
`/dev/sev-guest`) should be stripped before passing, otherwise the
33+
customer policy would have to whitelist them.
34+
- **Outputs** — `allowed` is required. Optional extras: `metadata` (state
35+
updates), `env_list`, `caps_list`, `allow_stdio_access`, `add_module`,
36+
any new field you need (must also be wired into the Go side).
37+
- **Default result** — what the framework should return when the policy
38+
pre-dates this enforcement point (`use_framework: false` case) or has no
39+
rule defined.
40+
41+
## 2. Register the enforcement point
42+
43+
Edit [pkg/securitypolicy/api.rego](../../pkg/securitypolicy/api.rego) and
44+
add an entry to `enforcement_points`. Bump the api `version` if you're
45+
introducing a new version (and update `version_api`). Choose
46+
`use_framework` based on intent:
47+
48+
- `use_framework: true` — the policy may omit a rule for this point and
49+
the enforcer will fall back to `data.framework.<your_point>`. Use when
50+
the framework can make a complete decision from existing policy data
51+
(containers, fragments, flags) without requiring customers to write
52+
anything new.
53+
- `use_framework: false` with a permissive `default_results` (e.g.
54+
`{"allowed": true}`) — older policies that don't know about this point
55+
get the safe-by-default behavior.
56+
- `use_framework: false` with `default_results: {"allowed": false}` —
57+
required when the new feature needs additional fields in the customer's
58+
policy (or in a fragment) to make a decision. An older policy without
59+
those fields cannot meaningfully allow the operation, so it must deny.
60+
Customers who want the feature must regenerate their policy with the
61+
new tooling.
62+
63+
## 3. Implement in framework.rego
64+
65+
In [pkg/securitypolicy/framework.rego](../../pkg/securitypolicy/framework.rego):
66+
67+
- Add `default <your_point> := {"allowed": false}` (or the right default).
68+
- Add one or more rule bodies. Use existing patterns:
69+
- **Narrowing pattern** (`create_container`, `exec_in_container`): start
70+
with `data.metadata.matches[input.containerID]`, repeatedly filter
71+
with `count(...) > 0` after each criterion. Each narrowing step should
72+
have a matching `errors[...]` rule (see below) so denials produce
73+
useful messages.
74+
- **Mount/unmount pattern**: check the target is not already mounted (or
75+
is mounted), validate the path/regex, look up the expected hash/etc
76+
against `data.metadata.devices` / `data.policy.containers[_].layers`,
77+
return a `metadata` op (`add`/`remove`/`update`).
78+
- For `is_linux` / `is_windows` differences provide separate rule bodies
79+
with the relevant guard.
80+
- If you need persistent state, use the metadata machinery rather than
81+
trying to compute it from inputs.
82+
- Bump `version` (and `version_framework`) if you broke compatibility, and
83+
add a corresponding `check_*` / `apply_defaults` shim so older policies
84+
keep working.
85+
86+
## 4. Add error rules
87+
88+
For every narrowing step, add `errors["human readable reason"] { input.rule == "<your_point>"; <condition that means the step failed> }`. These are
89+
queried via `data.policy.reason` on a deny to build the structured error
90+
message. Without them the user sees `noReasonMessage`.
91+
92+
Also extend `error_objects` if your decision produces a useful set of
93+
candidate objects (containers, processes, fragments) that should appear in
94+
the denial.
95+
96+
## 5. Wire the Go side
97+
98+
In [pkg/securitypolicy/securitypolicyenforcer.go](../../pkg/securitypolicy/securitypolicyenforcer.go),
99+
add the method to the `SecurityPolicyEnforcer` interface. Implement it as a
100+
no-op (or always-allow) on the non-Rego enforcers (`open_door`,
101+
`closed_door`, JSON enforcer if still present, `standard`).
102+
103+
In [pkg/securitypolicy/securitypolicyenforcer_rego.go](../../pkg/securitypolicy/securitypolicyenforcer_rego.go):
104+
105+
```go
106+
func (policy *regoEnforcer) EnforceMyThingPolicy(ctx context.Context, ...) (..., error) {
107+
input := inputData{
108+
"field1": ...,
109+
...
110+
}
111+
results, err := policy.enforce(ctx, "my_thing", input)
112+
if err != nil {
113+
return ..., err
114+
}
115+
// extract any extra outputs from results
116+
return ..., nil
117+
}
118+
```
119+
120+
- For OS-dependent input shape, switch on `policy.osType`.
121+
- For sensitive fields, extend `redactSensitiveData` /
122+
`replaceCapabilitiesWithPlaceholders` so they don't leak in error
123+
messages.
124+
- If your point mutates state, wrap the call site in a revertable section
125+
(`StartRevertableSection` + `commitOrRollbackPolicyRevSection`) and make
126+
sure the underlying operation cleans up partial state on its own failure.
127+
128+
## 6. Call from the handler
129+
130+
Linux: add the call in the appropriate handler in
131+
[internal/guest/runtime/hcsv2/uvm.go](../../internal/guest/runtime/hcsv2/uvm.go)
132+
(or wherever the host request first enters the trusted code). Validate
133+
host-controlled fields on the Go side **first** (length, regex, path
134+
equality, etc.) where the policy cannot meaningfully check them — e.g.
135+
`checkValidContainerID`, `checkContainerSettings`.
136+
137+
Windows: add the call in the corresponding gcs-sidecar handler under
138+
[internal/gcs-sidecar](../../internal/gcs-sidecar). Make sure the request
139+
is enforced **before** it is forwarded to the inbox GCS, and that any
140+
rewriting (env filtering, etc.) happens after enforcement so the forwarded
141+
payload reflects what was approved.
142+
143+
## 7. Tests
144+
145+
Add enforcer-level tests (build-tagged `rego`) in
146+
[pkg/securitypolicy/regopolicy_linux_test.go](../../pkg/securitypolicy/regopolicy_linux_test.go)
147+
and/or
148+
[pkg/securitypolicy/regopolicy_windows_test.go](../../pkg/securitypolicy/regopolicy_windows_test.go)
149+
covering: allow case, each deny case (one per `errors[...]` rule), metadata
150+
state after success, and (where relevant) cross-interaction with fragments.
151+
Add an entry to
152+
[pkg/securitypolicy/api_test.rego](../../pkg/securitypolicy/api_test.rego)
153+
so the api.rego registration is exercised. Remember to run tests with
154+
`go test -tags rego ./pkg/securitypolicy/...` — without the tag they are
155+
silently skipped.
156+
157+
## 8. Compatibility checklist
158+
159+
- Did you bump `version_api` / `version_framework` if the change is
160+
observable to existing policies?
161+
- Did you add `apply_defaults` / `check_*` shims for any new fields on
162+
container / external_process / fragment objects?
163+
- Does the no-op behavior on the non-Rego enforcers match the policy's
164+
default (typically allow for backward compatibility)?
165+
- For C-WCOW: does your input shape work for both Linux and Windows, or do
166+
you need an `is_linux` / `is_windows` split?

0 commit comments

Comments
 (0)