Skip to content

Commit ecd01df

Browse files
AmandaAZAmandaAZ
authored
Merge pull request #8885 from mayasingh17/msingh-troubleshoot-seccomp
AB#6013: Add seccomp profile TSG
2 parents 3dedc07 + c672a0e commit ecd01df

File tree

2 files changed

+236
-0
lines changed

2 files changed

+236
-0
lines changed

‎support/azure/azure-kubernetes/security/troubleshoot-seccomp-profiles.md‎

Lines changed: 231 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,231 @@
1+
---
2+
title: Troubleshoot Seccomp Profiles in Azure Kubernetes Service
3+
description: Helps you troubleshoot Azure Kubernetes Service workload issues caused by seccomp profiles.
4+
ms.reviewer: mayasingh, josebl, qasimsarfraz, v-weizhu
5+
ms.service: azure-kubernetes-service
6+
ms.topic: troubleshooting-general #Don't change.
7+
ms.date: 06/06/2025
8+
#customer intent: As a DevOps Engineer, I want to set up an effective Seccomp profile so that I can run my workloads successfully while making them as secure as possible.
9+
---
10+
# Troubleshoot seccomp profile configuration in Azure Kubernetes Service
11+
12+
[Secure computing (seccomp)](https://www.man7.org/linux/man-pages/man2/seccomp.2.html) is a Linux kernel feature that restricts the system calls (syscalls) containers can make, enhancing the security of containerized workloads. In Azure Kubernetes Service (AKS), the [containerd](https://containerd.io/) runtime used by AKS nodes natively supports seccomp. Enabling a seccomp profile might cause AKS workloads to fail because workload critical syscalls are blocked. This article introduces what seccomp profiles are, how they work, and how to troubleshoot them using the open source project [Inspektor Gadget](https://go.microsoft.com/fwlink/?linkid=2260072).
13+
14+
## Background
15+
16+
Syscalls are the interface that allows user space programs to request kernel services. Seccomp profiles specify the syscalls that are allowed or denied for a specific container. AKS supports two values:
17+
18+
- `RuntimeDefault`: Use the default seccomp profile given by the runtime.
19+
- `Unconfined`: All syscalls are allowed.
20+
21+
To enable seccomp on your AKS node pools, see [Secure container access to resources using built-in Linux security features](/azure/aks/secure-container-access). You can also configure a custom profile to meet your workload's specific needs, see [Configure a custom seccomp profile](/azure/aks/secure-container-access#configure-a-custom-seccomp-profile) for details.
22+
23+
When using seccomp profiles, it's important to test and validate the effect on your workloads. Some workloads might require a lower number of syscall restrictions than others. This means that if workloads require syscalls that aren't included in the configured profile, they might fail during runtime.
24+
25+
This article shows how to use the open source project [Inspektor Gadget](https://go.microsoft.com/fwlink/?linkid=2260072) to diagnose issues and gain visibility into blocked syscalls.
26+
27+
## Symptoms
28+
29+
After you configure AKS workloads to use a seccomp profile, the workloads exit unexpectedly with one of the following errors:
30+
31+
- > permission denied
32+
- > function not implemented
33+
34+
## Prerequisites
35+
36+
- A tool to connect to the Kubernetes cluster, such as the `kubectl` tool. To install `kubectl` using the [Azure CLI](/cli/azure/install-azure-cli), run the [az aks install-cli](/cli/azure/aks#az-aks-install-cli) command.
37+
- The [krew](https://sigs.k8s.io/krew) package manager for installing [Inspektor Gadget](https://go.microsoft.com/fwlink/?linkid=2260072)'s plugin. You can follow the [krew quickstart guide](https://krew.sigs.k8s.io/docs/user-guide/quickstart/) to install it.
38+
- The seccomp profile that you're trying to troubleshoot.
39+
- The open source project [Inspektor Gadget](/troubleshoot/azure/azure-kubernetes/logs/capture-system-insights-from-aks#how-to-install-inspektor-gadget-in-an-aks-cluster).
40+
41+
## Troubleshooting checklist
42+
43+
### Step 1: Modify your seccomp profile
44+
45+
Create a custom seccomp profile matching the one you're troubleshooting and replace its default action such as `SCMP_ACT_ERRNO` with `SCMP_ACT_LOG` to log blocked syscalls instead of failing them.
46+
47+
Your custom seccomp profile might look like this:
48+
49+
```json
50+
{
51+
"defaultAction": "SCMP_ACT_ALLOW",
52+
"syscalls": [
53+
{
54+
"names": ["acct",
55+
"add_key",
56+
"bpf",
57+
"clock_adjtime",
58+
"clock_settime",
59+
"clone",
60+
"create_module",
61+
"delete_module",
62+
"finit_module",
63+
"get_kernel_syms",
64+
"get_mempolicy",
65+
"init_module",
66+
"ioperm",
67+
"iopl",
68+
"kcmp",
69+
"kexec_file_load",
70+
"kexec_load",
71+
"keyctl",
72+
"lookup_dcookie",
73+
"mbind",
74+
"mount",
75+
"move_pages",
76+
"nfsservctl",
77+
"open_by_handle_at",
78+
"perf_event_open",
79+
"personality",
80+
"pivot_root",
81+
"process_vm_readv",
82+
"process_vm_writev",
83+
"ptrace",
84+
"query_module",
85+
"quotactl",
86+
"reboot",
87+
"request_key",
88+
"set_mempolicy",
89+
"setns",
90+
"settimeofday",
91+
"stime",
92+
"swapon",
93+
"swapoff",
94+
"sysfs",
95+
"_sysctl",
96+
"umount",
97+
"umount2",
98+
"unshare",
99+
"uselib",
100+
"userfaultfd",
101+
"ustat",
102+
"vm86",
103+
"vm86old"],
104+
"action": "SCMP_ACT_LOG"
105+
}
106+
]
107+
}
108+
```
109+
110+
The article [Configure a custom seccomp profile](/azure/aks/secure-container-access#configure-a-custom-seccomp-profile) shows how you can apply your custom seccomp profile to your AKS cluster. Alternatively, you can follow these steps:
111+
112+
1. Get the names of the nodes in your AKS cluster by running the following command:
113+
114+
```console
115+
kubectl get nodes
116+
```
117+
118+
2. Use the `kubectl debug` command to start a debug pod on the node and make sure the **seccomp** folder exists and the `tar` tool is installed (for copying the profile into the node in the next step):
119+
120+
```console
121+
kubectl debug node/<node-name> -it --image=mcr.microsoft.com/azurelinux/base/core:3.0
122+
root [ / ]# mkdir -p /host/var/lib/kubelet/seccomp
123+
root [ / ]# tdnf install -y tar
124+
```
125+
126+
3. Copy the pod name printed when running the `kubectl debug` command. It looks like `node-debugger-<node-name>-<random-sufix>`. It can also be retrieved by listing the pods in the `default` namespace.
127+
128+
4. In another terminal, transfer the seccomp profile file to the node directly:
129+
130+
```console
131+
kubectl cp <the path of the local seccomp profile>/my-profile.json <pod name>:/host/var/lib/kubelet/seccomp/my-profile.json
132+
```
133+
134+
> [!NOTE]
135+
> Repeat the preceding steps for each node in the cluster to ensure that the seccomp profile is available on all nodes where your workload might run.
136+
137+
Now, you can modify the `seccompProfile` specification of the target pod, which should be confined to the recorded syscalls. For example:
138+
139+
```yaml
140+
apiVersion: v1
141+
kind: Pod
142+
metadata:
143+
name: default-pod
144+
labels:
145+
app: default-pod
146+
spec:
147+
securityContext:
148+
seccompProfile:
149+
type: Localhost
150+
localhostProfile: my-profile.json
151+
containers:
152+
- name: test-container
153+
image: docker.io/library/nginx:latest
154+
```
155+
156+
### Step 2: Install Inspektor Gadget
157+
158+
[Inspektor Gadget](https://go.microsoft.com/fwlink/?linkid=2260072) provides insights into syscalls affecting your containers. To use it, run the following commands to install the `gadget` kubectl plugin in your host and deploy [Inspektor Gadget](https://go.microsoft.com/fwlink/?linkid=2260072) into the cluster:
159+
160+
```console
161+
kubectl krew install gadget
162+
```
163+
164+
```console
165+
kubectl gadget deploy
166+
```
167+
168+
For more information, see [How to install Inspektor Gadget in an AKS cluster](../logs/capture-system-insights-from-aks.md#how-to-install-inspektor-gadget-in-an-aks-cluster).
169+
170+
### Step 3: Run the audit_seccomp gadget
171+
172+
With [Inspektor Gadget](https://go.microsoft.com/fwlink/?linkid=2260072) installed, start the [audit_seccomp gadget](https://go.microsoft.com/fwlink/?linkid=2259786) using the [kubectl gadget run command](https://go.microsoft.com/fwlink/?linkid=2259865):
173+
174+
```console
175+
kubectl gadget run audit_seccomp
176+
```
177+
178+
### Step 4: Analyze blocked syscalls
179+
180+
Run your workload using the `kubectl apply -f` command. Then, the [audit_seccomp gadget](https://go.microsoft.com/fwlink/?linkid=2259786) logs the syscalls that the seccomp profile should block, along with their associated pods, containers, and processes. You can use this information to identify the root causes of workload failures.
181+
182+
For example, if you run the above-mentioned `default-pod` pod with the `my-profile.json` profile, the output looks like the following one:
183+
184+
```output
185+
K8S.NODE K8S.NAMESPACE K8S.PODNAME K8S.CONTAINERNAME COMM PID TID CODE SYSCALL
186+
aks-nodepool1-38695788-vmss000002 default default-pod test-container docker-entrypoi 3996610 3996610 SECCOMP_RET_LOG SYS_CLONE
187+
aks-nodepool1-38695788-vmss000002 default default-pod test-container docker-entrypoi 3996610 3996610 SECCOMP_RET_LOG SYS_CLONE
188+
aks-nodepool1-38695788-vmss000002 default default-pod test-container docker-entrypoi 3996610 3996610 SECCOMP_RET_LOG SYS_CLONE
189+
aks-nodepool1-38695788-vmss000002 default default-pod test-container docker-entrypoi 3996610 3996610 SECCOMP_RET_LOG SYS_CLONE
190+
aks-nodepool1-38695788-vmss000002 default default-pod test-container docker-entrypoi 3996610 3996610 SECCOMP_RET_LOG SYS_CLONE
191+
aks-nodepool1-38695788-vmss000002 default default-pod test-container 10-listen-on-ip 3996628 3996628 SECCOMP_RET_LOG SYS_CLONE
192+
aks-nodepool1-38695788-vmss000002 default default-pod test-container 10-listen-on-ip 3996628 3996628 SECCOMP_RET_LOG SYS_CLONE
193+
aks-nodepool1-38695788-vmss000002 default default-pod test-container 10-listen-on-ip 3996632 3996632 SECCOMP_RET_LOG SYS_CLONE
194+
aks-nodepool1-38695788-vmss000002 default default-pod test-container 10-listen-on-ip 3996632 3996632 SECCOMP_RET_LOG SYS_CLONE
195+
aks-nodepool1-38695788-vmss000002 default default-pod test-container 10-listen-on-ip 3996632 3996632 SECCOMP_RET_LOG SYS_CLONE
196+
aks-nodepool1-38695788-vmss000002 default default-pod test-container 10-listen-on-ip 3996628 3996628 SECCOMP_RET_LOG SYS_CLONE
197+
aks-nodepool1-38695788-vmss000002 default default-pod test-container 10-listen-on-ip 3996628 3996628 SECCOMP_RET_LOG SYS_CLONE
198+
aks-nodepool1-38695788-vmss000002 default default-pod test-container 20-envsubst-on- 3996639 3996639 SECCOMP_RET_LOG SYS_CLONE
199+
aks-nodepool1-38695788-vmss000002 default default-pod test-container 20-envsubst-on- 3996639 3996639 SECCOMP_RET_LOG SYS_CLONE
200+
aks-nodepool1-38695788-vmss000002 default default-pod test-container 20-envsubst-on- 3996641 3996641 SECCOMP_RET_LOG SYS_CLONE
201+
aks-nodepool1-38695788-vmss000002 default default-pod test-container 30-tune-worker- 3996643 3996643 SECCOMP_RET_LOG SYS_CLONE
202+
aks-nodepool1-38695788-vmss000002 default default-pod test-container nginx 3996610 3996610 SECCOMP_RET_LOG SYS_CLONE
203+
aks-nodepool1-38695788-vmss000002 default default-pod test-container nginx 3996610 3996610 SECCOMP_RET_LOG SYS_CLONE
204+
```
205+
206+
The output indicates that the `test-container` executes the `SYS_CLONE` syscall that the seccomp profile should block. With this information, you can determine whether to permit the listed syscalls in your container. If so, adjust the seccomp profile by removing them, which prevents the workload from failing.
207+
208+
Here are some commonly blocked syscalls to watch out for. A more comprehensive list is available in [Significant syscalls blocked by default profile](/azure/aks/secure-container-access#significant-syscalls-blocked-by-default-profile).
209+
210+
| Blocked syscall |Consideration |
211+
|---|---|
212+
| `clock_settime` or `clock_adjtime` | If your workload needs accurate time synchronization, ensure this syscall isn't blocked. |
213+
| `add_key` or `key_ctl` | If your workload requires key management, these blocked syscalls prevent containers from using the kernel keyring that is used for retaining security data, authentication keys, encryption keys, and other data within the kernel. |
214+
| `clone` | This syscall prevents the cloning of new namespaces, except for `CLONE_NEWUSER`. Workloads that depend on creating new namespaces might be affected if this syscall is blocked. |
215+
| `io_uring` | This syscall is blocked with the move to `containerd` 2.0. However, it's not blocked in the profile for `containerd` 1.7. |
216+
217+
## Next steps
218+
219+
If you encounter issues with your workloads due to blocked syscalls, consider using a custom seccomp profile suited to the specific needs of your application. You can check out the [Inspektor Gadget advise_seccomp gadget](https://go.microsoft.com/fwlink/?linkid=2260408).
220+
221+
Testing and refining seccomp profiles helps maintain performance and security for AKS workloads. For further assistance, see [Secure computing](/azure/aks/secure-container-access#secure-computing-seccomp).
222+
223+
## Related content
224+
225+
[Secure container access to resources using built-in Linux security features](/azure/aks/secure-container-access#secure-computing-seccomp)
226+
227+
[!INCLUDE [Third-party information disclaimer](../../../includes/third-party-disclaimer.md)]
228+
229+
[!INCLUDE [Third-party contact information disclaimer](../../../includes/third-party-contact-disclaimer.md)]
230+
231+
[!INCLUDE [Azure Help Support](../../../includes/azure-help-support.md)]

‎support/azure/azure-kubernetes/toc.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -292,6 +292,11 @@
292292
- name: Troubleshoot pods and namespaces stuck in the Terminating state
293293
href: storage/pods-namespaces-terminating-state.md
294294

295+
- name: Security
296+
items:
297+
- name: Troubleshoot seccomp profiles in AKS
298+
href: security/troubleshoot-seccomp-profiles.md
299+
295300
- name: Extensions, Policies and Add-Ons
296301
items:
297302
- name: Troubleshoot Azure Policy Add-on for AKS

0 commit comments

Comments
 (0)