You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# Capture real-time system insights from an AKS cluster
15
15
16
16
This article discusses the process of gathering real-time system insights from your Microsoft Azure Kubernetes Service (AKS) cluster by using Inspektor Gadget. The article contains step-by-step instructions for installing this tool on your AKS environment. It also explores practical examples that show how Inspektor Gadget helps you gather valuable information to do effective debugging of real-world issues.
17
17
18
-
## Demo
18
+
## Demo: Real-time DNS troubleshooting and critical file-access alerting
19
19
20
-
To begin, consider the following quick demo. Suppose that you have to figure out why the DNS requests from an application fail. By using Inspektor Gadget, you can capture the DNS traffic in the Kubernetes namespace in which your application is running:
20
+
To begin, consider the following quick demo. Suppose that you have to figure out why the DNS requests from an application fail. By using Inspektor Gadget, you can run the [trace_dns gadget](https://go.microsoft.com/fwlink/?linkid=2260317) to capture the DNS traffic in the Kubernetes namespace in which your application is running:
21
21
22
22
```bash
23
-
kubectl gadget trace dns --namespace my-ns --output columns=+nameserver
From this information, you can see that the DNS requests are directed to the DNS server at IP address `1.2.3.4`, but the server never responds.
34
+
From this information, we can see that the DNS requests are directed to the DNS server at IP address `1.2.3.4` (as indicated in the `NAMESERVER` column), but we only see the queries (`Q` in the `QR` column) and no responses (`R` in the `QR` column). This means that the DNS server didn't respond to the queries, which is why the application can't resolve the domain name `www.example.com`.
35
35
36
-
Now, suppose that `1.2.3.4` isn't the default name server configuration, and you suspect that a suspicious process is modifying the configuration at runtime. In these kinds of cases, Inspektor Gadget goes beyond DNS diagnostics. It also enables you to monitor processes that access critical files (such as */etc/resolv.conf*) and have the intention of modifying those files. To use this monitoring feature, filter the flags in the output to show any of the [writing file access modes](https://linux.die.net/man/3/open) (`O_WRONLY` to open for writing only, or `O_RDWR` to open for reading and writing):
36
+
Now, suppose that `1.2.3.4` isn't the default name server configuration, and you suspect that a malicious process is modifying the configuration at runtime. In these kinds of cases, Inspektor Gadget goes beyond DNS diagnostics. It also enables you to monitor processes that access critical files (such as */etc/resolv.conf*) and have the intention of modifying those files. To do that, run the [trace_open gadget](https://go.microsoft.com/fwlink/?linkid=2260318)in the same namespace and filter the results by the file name and the flags that indicate [the intention to write to the file](https://linux.die.net/man/3/open) (`O_WRONLY` to open for writing only, or `O_RDWR` to open for reading and writing):
[Inspektor Gadget](https://go.microsoft.com/fwlink/?linkid=2260072) is a framework that's designed for building, packaging, deploying, and running tools that are dedicated to debugging and inspecting Linux and Kubernetes systems. These tools ("gadgets") are implemented as[eBPF](https://go.microsoft.com/fwlink/?linkid=2259866) programs. Their primary goal is to gather low-level kernel data to provide insights into specific system scenarios. The Inspektor Gadget framework manages the association of the collected data by using high-level references, such as Kubernetes resources. This integration makes sure that a seamless connection exists between low-level insights and their corresponding high-level context. The integration streamlines the troubleshooting process and the collection of relevant information.
52
+
[Inspektor Gadget](https://go.microsoft.com/fwlink/?linkid=2260072) is a framework that makes it easy to monitor, troubleshoot, and secure workloads running on Linux and Kubernetes. It consists of tools (*Gadgets*) that leverage[eBPF](https://go.microsoft.com/fwlink/?linkid=2259866) programs. Their primary goal is to gather low-level kernel data to provide insights into specific system scenarios. The Inspektor Gadget framework manages the association of the collected data by using high-level references, such as Kubernetes resources. This integration makes sure that a seamless connection exists between low-level insights and their corresponding high-level context. The integration streamlines the troubleshooting process and the collection of relevant information.
52
53
53
54
## Gadgets
54
55
@@ -60,7 +61,6 @@ Inspektor Gadget provides a set of built-in tools that are designed to debug and
60
61
61
62
The gadgets present the information that they collected by using different mechanisms. For instance, some gadgets can inform you about the system status at specific times. Other gadgets can report every time a given event occurs, or they can provide periodic updates.
62
63
63
-
64
64
These are just a few examples. The [official documentation](https://go.microsoft.com/fwlink/?linkid=2260507) provides detailed descriptions and examples of each gadget so that you can determine the most suitable gadget for your specific use case. However, if you find a use case that the existing gadgets don't currently cover, Inspektor Gadget allows you to run your own eBPF programs by using the [run command](https://go.microsoft.com/fwlink/?linkid=2259865). Because the Inspektor Gadget framework handles the building, packaging, and deployment of your custom programs, it streamlines the process for your unique requirements. Also, it gathers high-level metadata to enrich the data that you collect in your program.
65
65
66
66
## Use cases
@@ -70,7 +70,7 @@ To complement the demo that's presented at the beginning of this article, we com
70
70
| Problem area | Symptoms | Troubleshooting |
71
71
|--|--|--|
72
72
|**Disk-intensive applications**| High memory or CPU usage, or inconsistent node readiness | An application might consistently engage in disk read/write operations, such as extensive logging. By using Inspektor Gadget, you can identify in real time which containers generate more [block I/O](https://go.microsoft.com/fwlink/?linkid=2260070). Or, more specifically, you can find the container that causes more reads and writes into a [file](https://go.microsoft.com/fwlink/?linkid=2260071). |
73
-
|**"It's always DNS"**| High application latency, time-outs, or poor end-user experience | <p>By using Inspektor Gadget, you can [trace all the DNS](https://go.microsoft.com/fwlink/?linkid=2260317) queries and responses in the cluster. In particular, Inspektor Gadget provides the following information that helps you to determine whether the DNS is affecting your application's performance:</p> <ul> <li>Query success</li> <li>Whether the response contains an error</li> <li>The name server that's used for the lookup</li> <li>The query-response latency</li> </ul> |
73
+
|**"It's always DNS"**| High application latency, timeouts, or poor end-user experience | <p>By using Inspektor Gadget, you can [trace all the DNS](https://go.microsoft.com/fwlink/?linkid=2260317) queries and responses in the cluster. In particular, Inspektor Gadget provides the following information that helps you to determine whether the DNS is affecting your application's performance:</p> <ul> <li>Query success</li> <li>Whether the response contains an error</li> <li>The name server that's used for the lookup</li> <li>The query-response latency</li> </ul> |
74
74
| **File system access** | Application misbehaves or can't function correctly | <p>The application might be unable to access specific configurations, logs, or other vital files in the file system. In such scenarios, Inspektor Gadget enables you to [trace all the opened files](https://go.microsoft.com/fwlink/?linkid=2260318) inside pods to diagnose access issues. Whenever your application tries to open a file, you can discover the following information:</p> <ul> <li>The flags that are used to open the file (for example, [O_RDONLY, O_WRONLY, O_RDWR](https://linux.die.net/man/3/open), and so on)</li> <li>Whether the file opening attempt succeeds</li> <li>The returned error (if the file opening attempt fails)</li> </ul> <p>For instance, if the attempt to open the file fails because of error 2 ([ENOENT](https://man7.org/linux/man-pages/man3/errno.3.html)), the application is probably trying to open a file that doesn’t exist. This means that you might have a typo in the code, or the file is available in a different path.</p> |
75
75
|**Remote code execution (RCE)**| Unauthorized code execution such as [cryptojacking](https://en.wikipedia.org/wiki/Cryptojacking) that's evident in high CPU usage during application idle periods | When attackers try to make this kind of attack on a system, they usually have to run the code by using `bash`. Inspektor Gadget enables you to [trace the creation of new processes](https://go.microsoft.com/fwlink/?linkid=2260319), particularly processes that involve critical commands such as `bash`. |
76
76
@@ -86,46 +86,164 @@ By selecting the following button, an AKS cluster will be automatically created,
86
86
87
87
This section outlines the steps for installing Inspektor Gadget in your AKS cluster by running the `kubectl gadget` plug-in. The installation consists of two parts:
88
88
89
-
- Installing the `kubectl gadget` plug-in on your computer
89
+
1. Installing the `kubectl gadget` plug-in on your workstation
90
90
91
-
- Running the `kubectl gadget` plug-in to install Inspektor Gadget in the cluster
91
+
2. Running the `kubectl gadget` plug-in to deploy Inspektor Gadget in the cluster
92
92
93
93
> [!WARNING]
94
-
> Many mechanisms are available to deploy and use Inspektor Gadget. Each of these mechanisms is tailored to specific use cases and requirements. You can use the kubectl gadget plug-in to apply several of these mechanisms, but not all of them. For instance, deploying Inspektor Gadget by using the `kubectl gadget` plug-in depends on the availability of the Kubernetes API server. If you can’t depend on such a component because its availability might be occasionally compromised, we recommend that you avoid using the `kubectl gadget`deployment mechanism. For more information about this and other use cases, see the [Inspektor Gadget documentation](https://go.microsoft.com/fwlink/?linkid=2260072).
94
+
> Many mechanisms are available to deploy and use Inspektor Gadget. Each of these mechanisms is tailored to specific use cases and requirements. You can use the kubectl gadget plug-in to apply several of these mechanisms, but not all of them. For instance, deploying Inspektor Gadget by using the `kubectl gadget` plug-in depends on the availability of the Kubernetes API server. If you can’t depend on such a component because its availability might be occasionally compromised, we recommend that you avoid using the `kubectl gadget`deployment mechanism. For more information about this and other use cases, see the [Inspektor Gadget documentation](https://go.microsoft.com/fwlink/?linkid=2326106).
95
95
96
96
#### Prerequisites
97
97
98
98
- The Kubernetes [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/) command-line tool. If you have [Azure CLI](/cli/azure/install-azure-cli-linux), you can run the [az aks install-cli](/cli/azure/aks#az-aks-install-cli) command to install kubectl.
99
99
100
100
- An AKS cluster. If you don't have an AKS cluster, [create one by using Azure CLI](/azure/aks/learn/quick-kubernetes-deploy-cli) or [by using the Azure portal](/azure/aks/learn/quick-kubernetes-deploy-portal).
101
101
102
-
- The [krew](https://sigs.k8s.io/krew) package manager for plug-ins in kubectl. You can follow the [krew quickstart guide](https://krew.sigs.k8s.io/docs/user-guide/quickstart/) to install this package manager.
102
+
#### Part 1: Install the kubectl gadget plug-in on your workstation
103
103
104
-
#### Part 1: Install the kubectl plug-in gadget on your computer
104
+
Use the instructions for your OS:
105
105
106
-
We recommend that you use `krew` to install the `kubectl gadget` plug-in.
106
+
- Azure Linux 3.0
107
+
- Ubuntu 18.04 / 20.04 / 22.04
107
108
108
109
> [!NOTE]
109
-
> To install a specific release or compile it from the source, see [Install kubectl gadget](https://go.microsoft.com/fwlink/?linkid=2260075#installing-kubectl-gadget) on GitHub.
110
+
> To install a specific release or compile it from the source, see [Install kubectl gadget](https://go.microsoft.com/fwlink/?linkid=2260075#installing-the-kubectl-gadget-client) on GitHub.
110
111
111
-
```bash
112
-
kubectl krew install gadget
113
-
```
112
+
##### [Azure Linux 3.0](#tab/azurelinux30)
113
+
114
+
1. Add the Microsoft Cloud-Native repository to your system:
2. Download the Microsoft GNU Privacy Guard (GPG) public key:
144
+
145
+
```bash
146
+
curl -sSL https://packages.microsoft.com/keys/microsoft.asc | tee /usr/share/keyrings/microsoft.asc
147
+
```
148
+
149
+
3. Add the Microsoft Cloud-Native repository to your system:
150
+
151
+
```bash
152
+
echo "deb [arch=amd64,arm64,armhf signed-by=/usr/share/keyrings/microsoft.asc] https://packages.microsoft.com/ubuntu/22.04/prod jammy main" | tee /etc/apt/sources.list.d/packages-microsoft-prod.list
153
+
```
154
+
155
+
4. Update the package index:
156
+
157
+
```bash
158
+
apt update
159
+
```
160
+
161
+
5. Install the `kubectl gadget` plug-in:
162
+
163
+
```bash
164
+
apt install -y kubectl-gadget
165
+
```
166
+
167
+
##### [Ubuntu 20.04](#tab/ubuntu2004)
168
+
169
+
1. Get `curl` if you don't have it installed:
170
+
171
+
```bash
172
+
apt update && apt install -y curl
173
+
```
174
+
175
+
2. Download the Microsoft GNU Privacy Guard (GPG) public key:
176
+
177
+
```bash
178
+
curl -sSL https://packages.microsoft.com/keys/microsoft.asc | tee /usr/share/keyrings/microsoft.asc
179
+
```
180
+
181
+
3. Add the Microsoft Cloud-Native repository to your system:
182
+
183
+
```bash
184
+
echo"deb [arch=amd64,arm64,armhf signed-by=/usr/share/keyrings/microsoft.asc] https://packages.microsoft.com/ubuntu/20.04/prod focal main"| tee /etc/apt/sources.list.d/packages-microsoft-prod.list
185
+
```
186
+
187
+
4. Update the package index:
188
+
189
+
```bash
190
+
apt update
191
+
```
192
+
193
+
5. Install the `kubectl gadget` plug-in:
194
+
195
+
```bash
196
+
apt install -y kubectl-gadget
197
+
```
198
+
199
+
##### [Ubuntu 18.04](#tab/ubuntu1804)
200
+
201
+
1. Get `curl`if you don't have it installed:
202
+
203
+
```bash
204
+
apt update && apt install -y curl
205
+
```
206
+
207
+
2. Download the Microsoft GNU Privacy Guard (GPG) public key:
208
+
209
+
```bash
210
+
curl -sSL https://packages.microsoft.com/keys/microsoft.asc | tee /usr/share/keyrings/microsoft.asc
211
+
```
212
+
213
+
3. Add the Microsoft Cloud-Native repository to your system:
214
+
215
+
```bash
216
+
echo "deb [arch=amd64,arm64,armhf signed-by=/usr/share/keyrings/microsoft.asc] https://packages.microsoft.com/ubuntu/18.04/prod bionic main" | tee /etc/apt/sources.list.d/packages-microsoft-prod.list
217
+
```
218
+
219
+
4. Update the package index:
220
+
221
+
```bash
222
+
apt update
223
+
```
224
+
225
+
5. Install the `kubectl gadget` plug-in:
226
+
227
+
```bash
228
+
apt install -y kubectl-gadget
229
+
```
230
+
231
+
---
114
232
115
233
Now, verify the installation by running the `version` command:
116
234
117
235
```bash
118
236
kubectl gadget version
119
237
```
120
238
121
-
The `version`command shows you the version of the client (`kubectl gadget` plug-in), but it also tells you that it isn't installed yet on the server (the cluster):
239
+
The command output shows you the version of the client (`kubectl gadget` plug-in) and that it isn't installed yet on the server (the cluster):
122
240
123
241
```output
124
242
Client version: vX.Y.Z
125
243
Server version: not installed
126
244
```
127
245
128
-
#### Part 2: Install Inspektor Gadget in the cluster
246
+
#### Part 2: Deploy Inspektor Gadget in the cluster
129
247
130
248
The following command deploys the [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) controller.
131
249
@@ -155,6 +273,24 @@ Client version: vX.Y.Z
155
273
Server version: vX.Y.Z
156
274
```
157
275
276
+
When deploying Inspektor Gadget with the `kubectl gadget` plug-in available in the Microsoft Cloud-Native repository, the container image used for the DaemonSet is automatically pulled from the Microsoft Container Registry (MCR):
277
+
278
+
```bash
279
+
kubectl get daemonset gadget -n gadget -o jsonpath='{.spec.template.spec.containers[*].image}'
> If you use the [krew](https://sigs.k8s.io/krew) package manager to install kubectl plug-ins and prefer to install Inspektor Gadget directly from its GitHub repository, you can easily install the `kubectl gadget` plug-in and deploy Inspektor Gadget in your cluster using the following commands:
0 commit comments