|
| 1 | +--- |
| 2 | +title: Troubleshoot Seccomp Profiles in Azure Kubernetes Service |
| 3 | +description: Helps you troubleshoot Azure Kubernetes Service workload issues caused by seccomp profiles. |
| 4 | +ms.reviewer: mayasingh, josebl, qasimsarfraz, v-weizhu |
| 5 | +ms.service: azure-kubernetes-service |
| 6 | +ms.topic: troubleshooting-general #Don't change. |
| 7 | +ms.date: 06/06/2025 |
| 8 | +#customer intent: As a DevOps Engineer, I want to set up an effective Seccomp profile so that I can run my workloads successfully while making them as secure as possible. |
| 9 | +--- |
| 10 | +# Troubleshoot seccomp profile configuration in Azure Kubernetes Service |
| 11 | + |
| 12 | +[Secure computing (seccomp)](https://www.man7.org/linux/man-pages/man2/seccomp.2.html) is a Linux kernel feature that restricts the system calls (syscalls) containers can make, enhancing the security of containerized workloads. In Azure Kubernetes Service (AKS), the [containerd](https://containerd.io/) runtime used by AKS nodes natively supports seccomp. Enabling a seccomp profile might cause AKS workloads to fail because workload critical syscalls are blocked. This article introduces what seccomp profiles are, how they work, and how to troubleshoot them using the open source project [Inspektor Gadget](https://go.microsoft.com/fwlink/?linkid=2260072). |
| 13 | + |
| 14 | +## Background |
| 15 | + |
| 16 | +Syscalls are the interface that allows user space programs to request kernel services. Seccomp profiles specify the syscalls that are allowed or denied for a specific container. AKS supports two values: |
| 17 | + |
| 18 | +- `RuntimeDefault`: Use the default seccomp profile given by the runtime. |
| 19 | +- `Unconfined`: All syscalls are allowed. |
| 20 | + |
| 21 | +To enable seccomp on your AKS node pools, see [Secure container access to resources using built-in Linux security features](/azure/aks/secure-container-access). You can also configure a custom profile to meet your workload's specific needs, see [Configure a custom seccomp profile](/azure/aks/secure-container-access#configure-a-custom-seccomp-profile) for details. |
| 22 | + |
| 23 | +When using seccomp profiles, it's important to test and validate the effect on your workloads. Some workloads might require a lower number of syscall restrictions than others. This means that if workloads require syscalls that aren't included in the configured profile, they might fail during runtime. |
| 24 | + |
| 25 | +This article shows how to use the open source project [Inspektor Gadget](https://go.microsoft.com/fwlink/?linkid=2260072) to diagnose issues and gain visibility into blocked syscalls. |
| 26 | + |
| 27 | +## Symptoms |
| 28 | + |
| 29 | +After you configure AKS workloads to use a seccomp profile, the workloads exit unexpectedly with one of the following errors: |
| 30 | + |
| 31 | +- > permission denied |
| 32 | +- > function not implemented |
| 33 | +
|
| 34 | +## Prerequisites |
| 35 | + |
| 36 | +- A tool to connect to the Kubernetes cluster, such as the `kubectl` tool. To install `kubectl` using the [Azure CLI](/cli/azure/install-azure-cli), run the [az aks install-cli](/cli/azure/aks#az-aks-install-cli) command. |
| 37 | +- The [krew](https://sigs.k8s.io/krew) package manager for installing [Inspektor Gadget](https://go.microsoft.com/fwlink/?linkid=2260072)'s plugin. You can follow the [krew quickstart guide](https://krew.sigs.k8s.io/docs/user-guide/quickstart/) to install it. |
| 38 | +- The seccomp profile that you're trying to troubleshoot. |
| 39 | +- The open source project [Inspektor Gadget](/troubleshoot/azure/azure-kubernetes/logs/capture-system-insights-from-aks#how-to-install-inspektor-gadget-in-an-aks-cluster). |
| 40 | + |
| 41 | +## Troubleshooting checklist |
| 42 | + |
| 43 | +### Step 1: Modify your seccomp profile |
| 44 | + |
| 45 | +Create a custom seccomp profile matching the one you're troubleshooting and replace its default action such as `SCMP_ACT_ERRNO` with `SCMP_ACT_LOG` to log blocked syscalls instead of failing them. |
| 46 | + |
| 47 | +Your custom seccomp profile might look like this: |
| 48 | + |
| 49 | +```json |
| 50 | +{ |
| 51 | + "defaultAction": "SCMP_ACT_ALLOW", |
| 52 | + "syscalls": [ |
| 53 | + { |
| 54 | + "names": ["acct", |
| 55 | + "add_key", |
| 56 | + "bpf", |
| 57 | + "clock_adjtime", |
| 58 | + "clock_settime", |
| 59 | + "clone", |
| 60 | + "create_module", |
| 61 | + "delete_module", |
| 62 | + "finit_module", |
| 63 | + "get_kernel_syms", |
| 64 | + "get_mempolicy", |
| 65 | + "init_module", |
| 66 | + "ioperm", |
| 67 | + "iopl", |
| 68 | + "kcmp", |
| 69 | + "kexec_file_load", |
| 70 | + "kexec_load", |
| 71 | + "keyctl", |
| 72 | + "lookup_dcookie", |
| 73 | + "mbind", |
| 74 | + "mount", |
| 75 | + "move_pages", |
| 76 | + "nfsservctl", |
| 77 | + "open_by_handle_at", |
| 78 | + "perf_event_open", |
| 79 | + "personality", |
| 80 | + "pivot_root", |
| 81 | + "process_vm_readv", |
| 82 | + "process_vm_writev", |
| 83 | + "ptrace", |
| 84 | + "query_module", |
| 85 | + "quotactl", |
| 86 | + "reboot", |
| 87 | + "request_key", |
| 88 | + "set_mempolicy", |
| 89 | + "setns", |
| 90 | + "settimeofday", |
| 91 | + "stime", |
| 92 | + "swapon", |
| 93 | + "swapoff", |
| 94 | + "sysfs", |
| 95 | + "_sysctl", |
| 96 | + "umount", |
| 97 | + "umount2", |
| 98 | + "unshare", |
| 99 | + "uselib", |
| 100 | + "userfaultfd", |
| 101 | + "ustat", |
| 102 | + "vm86", |
| 103 | + "vm86old"], |
| 104 | + "action": "SCMP_ACT_LOG" |
| 105 | + } |
| 106 | + ] |
| 107 | + } |
| 108 | + ``` |
| 109 | + |
| 110 | +The article [Configure a custom seccomp profile](/azure/aks/secure-container-access#configure-a-custom-seccomp-profile) shows how you can apply your custom seccomp profile to your AKS cluster. Alternatively, you can follow these steps: |
| 111 | + |
| 112 | +1. Get the names of the nodes in your AKS cluster by running the following command: |
| 113 | + |
| 114 | + ```console |
| 115 | + kubectl get nodes |
| 116 | + ``` |
| 117 | + |
| 118 | +2. Use the `kubectl debug` command to start a debug pod on the node and make sure the **seccomp** folder exists and the `tar` tool is installed (for copying the profile into the node in the next step): |
| 119 | + |
| 120 | + ```console |
| 121 | + kubectl debug node/<node-name> -it --image=mcr.microsoft.com/azurelinux/base/core:3.0 |
| 122 | + root [ / ]# mkdir -p /host/var/lib/kubelet/seccomp |
| 123 | + root [ / ]# tdnf install -y tar |
| 124 | + ``` |
| 125 | + |
| 126 | +3. Copy the pod name printed when running the `kubectl debug` command. It looks like `node-debugger-<node-name>-<random-sufix>`. It can also be retrieved by listing the pods in the `default` namespace. |
| 127 | + |
| 128 | +4. In another terminal, transfer the seccomp profile file to the node directly: |
| 129 | + |
| 130 | + ```console |
| 131 | + kubectl cp <the path of the local seccomp profile>/my-profile.json <pod name>:/host/var/lib/kubelet/seccomp/my-profile.json |
| 132 | + ``` |
| 133 | + |
| 134 | +> [!NOTE] |
| 135 | +> Repeat the preceding steps for each node in the cluster to ensure that the seccomp profile is available on all nodes where your workload might run. |
| 136 | + |
| 137 | +Now, you can modify the `seccompProfile` specification of the target pod, which should be confined to the recorded syscalls. For example: |
| 138 | + |
| 139 | +```yaml |
| 140 | +apiVersion: v1 |
| 141 | +kind: Pod |
| 142 | +metadata: |
| 143 | + name: default-pod |
| 144 | + labels: |
| 145 | + app: default-pod |
| 146 | +spec: |
| 147 | + securityContext: |
| 148 | + seccompProfile: |
| 149 | + type: Localhost |
| 150 | + localhostProfile: my-profile.json |
| 151 | + containers: |
| 152 | + - name: test-container |
| 153 | + image: docker.io/library/nginx:latest |
| 154 | +``` |
| 155 | + |
| 156 | +### Step 2: Install Inspektor Gadget |
| 157 | + |
| 158 | +[Inspektor Gadget](https://go.microsoft.com/fwlink/?linkid=2260072) provides insights into syscalls affecting your containers. To use it, run the following commands to install the `gadget` kubectl plugin in your host and deploy [Inspektor Gadget](https://go.microsoft.com/fwlink/?linkid=2260072) into the cluster: |
| 159 | + |
| 160 | +```console |
| 161 | +kubectl krew install gadget |
| 162 | +``` |
| 163 | + |
| 164 | +```console |
| 165 | +kubectl gadget deploy |
| 166 | +``` |
| 167 | + |
| 168 | +For more information, see [How to install Inspektor Gadget in an AKS cluster](../logs/capture-system-insights-from-aks.md#how-to-install-inspektor-gadget-in-an-aks-cluster). |
| 169 | + |
| 170 | +### Step 3: Run the audit_seccomp gadget |
| 171 | + |
| 172 | +With [Inspektor Gadget](https://go.microsoft.com/fwlink/?linkid=2260072) installed, start the [audit_seccomp gadget](https://go.microsoft.com/fwlink/?linkid=2259786) using the [kubectl gadget run command](https://go.microsoft.com/fwlink/?linkid=2259865): |
| 173 | + |
| 174 | +```console |
| 175 | +kubectl gadget run audit_seccomp |
| 176 | +``` |
| 177 | + |
| 178 | +### Step 4: Analyze blocked syscalls |
| 179 | + |
| 180 | +Run your workload using the `kubectl apply -f` command. Then, the [audit_seccomp gadget](https://go.microsoft.com/fwlink/?linkid=2259786) logs the syscalls that the seccomp profile should block, along with their associated pods, containers, and processes. You can use this information to identify the root causes of workload failures. |
| 181 | + |
| 182 | +For example, if you run the above-mentioned `default-pod` pod with the `my-profile.json` profile, the output looks like the following one: |
| 183 | + |
| 184 | +```output |
| 185 | +K8S.NODE K8S.NAMESPACE K8S.PODNAME K8S.CONTAINERNAME COMM PID TID CODE SYSCALL |
| 186 | +aks-nodepool1-38695788-vmss000002 default default-pod test-container docker-entrypoi 3996610 3996610 SECCOMP_RET_LOG SYS_CLONE |
| 187 | +aks-nodepool1-38695788-vmss000002 default default-pod test-container docker-entrypoi 3996610 3996610 SECCOMP_RET_LOG SYS_CLONE |
| 188 | +aks-nodepool1-38695788-vmss000002 default default-pod test-container docker-entrypoi 3996610 3996610 SECCOMP_RET_LOG SYS_CLONE |
| 189 | +aks-nodepool1-38695788-vmss000002 default default-pod test-container docker-entrypoi 3996610 3996610 SECCOMP_RET_LOG SYS_CLONE |
| 190 | +aks-nodepool1-38695788-vmss000002 default default-pod test-container docker-entrypoi 3996610 3996610 SECCOMP_RET_LOG SYS_CLONE |
| 191 | +aks-nodepool1-38695788-vmss000002 default default-pod test-container 10-listen-on-ip 3996628 3996628 SECCOMP_RET_LOG SYS_CLONE |
| 192 | +aks-nodepool1-38695788-vmss000002 default default-pod test-container 10-listen-on-ip 3996628 3996628 SECCOMP_RET_LOG SYS_CLONE |
| 193 | +aks-nodepool1-38695788-vmss000002 default default-pod test-container 10-listen-on-ip 3996632 3996632 SECCOMP_RET_LOG SYS_CLONE |
| 194 | +aks-nodepool1-38695788-vmss000002 default default-pod test-container 10-listen-on-ip 3996632 3996632 SECCOMP_RET_LOG SYS_CLONE |
| 195 | +aks-nodepool1-38695788-vmss000002 default default-pod test-container 10-listen-on-ip 3996632 3996632 SECCOMP_RET_LOG SYS_CLONE |
| 196 | +aks-nodepool1-38695788-vmss000002 default default-pod test-container 10-listen-on-ip 3996628 3996628 SECCOMP_RET_LOG SYS_CLONE |
| 197 | +aks-nodepool1-38695788-vmss000002 default default-pod test-container 10-listen-on-ip 3996628 3996628 SECCOMP_RET_LOG SYS_CLONE |
| 198 | +aks-nodepool1-38695788-vmss000002 default default-pod test-container 20-envsubst-on- 3996639 3996639 SECCOMP_RET_LOG SYS_CLONE |
| 199 | +aks-nodepool1-38695788-vmss000002 default default-pod test-container 20-envsubst-on- 3996639 3996639 SECCOMP_RET_LOG SYS_CLONE |
| 200 | +aks-nodepool1-38695788-vmss000002 default default-pod test-container 20-envsubst-on- 3996641 3996641 SECCOMP_RET_LOG SYS_CLONE |
| 201 | +aks-nodepool1-38695788-vmss000002 default default-pod test-container 30-tune-worker- 3996643 3996643 SECCOMP_RET_LOG SYS_CLONE |
| 202 | +aks-nodepool1-38695788-vmss000002 default default-pod test-container nginx 3996610 3996610 SECCOMP_RET_LOG SYS_CLONE |
| 203 | +aks-nodepool1-38695788-vmss000002 default default-pod test-container nginx 3996610 3996610 SECCOMP_RET_LOG SYS_CLONE |
| 204 | +``` |
| 205 | + |
| 206 | +The output indicates that the `test-container` executes the `SYS_CLONE` syscall that the seccomp profile should block. With this information, you can determine whether to permit the listed syscalls in your container. If so, adjust the seccomp profile by removing them, which prevents the workload from failing. |
| 207 | + |
| 208 | +Here are some commonly blocked syscalls to watch out for. A more comprehensive list is available in [Significant syscalls blocked by default profile](/azure/aks/secure-container-access#significant-syscalls-blocked-by-default-profile). |
| 209 | + |
| 210 | +| Blocked syscall |Consideration | |
| 211 | +|---|---| |
| 212 | +| `clock_settime` or `clock_adjtime` | If your workload needs accurate time synchronization, ensure this syscall isn't blocked. | |
| 213 | +| `add_key` or `key_ctl` | If your workload requires key management, these blocked syscalls prevent containers from using the kernel keyring that is used for retaining security data, authentication keys, encryption keys, and other data within the kernel. | |
| 214 | +| `clone` | This syscall prevents the cloning of new namespaces, except for `CLONE_NEWUSER`. Workloads that depend on creating new namespaces might be affected if this syscall is blocked. | |
| 215 | +| `io_uring` | This syscall is blocked with the move to `containerd` 2.0. However, it's not blocked in the profile for `containerd` 1.7. | |
| 216 | + |
| 217 | +## Next steps |
| 218 | + |
| 219 | +If you encounter issues with your workloads due to blocked syscalls, consider using a custom seccomp profile suited to the specific needs of your application. You can check out the [Inspektor Gadget advise_seccomp gadget](https://go.microsoft.com/fwlink/?linkid=2260408). |
| 220 | + |
| 221 | +Testing and refining seccomp profiles helps maintain performance and security for AKS workloads. For further assistance, see [Secure computing](/azure/aks/secure-container-access#secure-computing-seccomp). |
| 222 | + |
| 223 | +## Related content |
| 224 | + |
| 225 | +[Secure container access to resources using built-in Linux security features](/azure/aks/secure-container-access#secure-computing-seccomp) |
| 226 | + |
| 227 | +[!INCLUDE [Third-party information disclaimer](../../../includes/third-party-disclaimer.md)] |
| 228 | + |
| 229 | +[!INCLUDE [Third-party contact information disclaimer](../../../includes/third-party-contact-disclaimer.md)] |
| 230 | + |
| 231 | +[!INCLUDE [Azure Help Support](../../../includes/azure-help-support.md)] |
0 commit comments