Skip to content

Commit a45d7fe

Browse files
adrianreberTim Bannister
adrianreber
and
Tim Bannister
committed
Add blog post about forensic container analysis
Co-authored-by: Tim Bannister <[email protected]> Signed-off-by: Adrian Reber <[email protected]>
1 parent bf343e6 commit a45d7fe

File tree

1 file changed

+373
-0
lines changed
  • content/en/blog/_posts/2023-03-10-forensic-container-analysis

1 file changed

+373
-0
lines changed

‎content/en/blog/_posts/2023-03-10-forensic-container-analysis/index.md

Lines changed: 373 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,373 @@
1+
---
2+
layout: blog
3+
title: "Forensic container analysis"
4+
date: 2023-03-10
5+
slug: forensic-container-analysis
6+
---
7+
8+
**Authors:** Adrian Reber (Red Hat)
9+
10+
In my previous article, [Forensic container checkpointing in
11+
Kubernetes][forensic-blog], I introduced checkpointing in Kubernetes
12+
and how it has to be setup and how it can be used. The name of the
13+
feature is Forensic container checkpointing, but I did not go into
14+
any details how to do the actual analysis of the checkpoint created by
15+
Kubernetes. In this article I want to provide details how the
16+
checkpoint can be analyzed.
17+
18+
Checkpointing is still an alpha feature in Kubernetes and this article
19+
wants to provide a preview how the feature might work in the future.
20+
21+
## Preparation
22+
23+
Details about how to configure Kubernetes and the underlying CRI implementation
24+
to enable checkpointing support can be found in my [Forensic container
25+
checkpointing in Kubernetes][forensic-blog] article.
26+
27+
As an example I prepared a container image (`quay.io/adrianreber/counter:blog`)
28+
which I want to checkpoint and then analyze in this article. This container allows
29+
me to create files in the container and also store information in memory which
30+
I later want to find in the checkpoint.
31+
32+
To run that container I need a pod, and for this example I am using the following Pod manifest:
33+
34+
```yaml
35+
apiVersion: v1
36+
kind: Pod
37+
metadata:
38+
name: counters
39+
spec:
40+
containers:
41+
- name: counter
42+
image: quay.io/adrianreber/counter:blog
43+
```
44+
45+
This results in a container called `counter` running in a pod called `counters`.
46+
47+
Once the container is running I am performing following actions with that
48+
container:
49+
50+
```console
51+
$ kubectl get pod counters --template '{{.status.podIP}}'
52+
10.88.0.25
53+
$ curl 10.88.0.25:8088/create?test-file
54+
$ curl 10.88.0.25:8088/secret?RANDOM_1432_KEY
55+
$ curl 10.88.0.25:8088
56+
```
57+
58+
The first access creates a file called `test-file` with the content `test-file`
59+
in the container and the second access stores my secret information
60+
(`RANDOM_1432_KEY`) somewhere in the container's memory. The last access just
61+
adds an additional line to the internal log file.
62+
63+
The last step before I can analyze the checkpoint it to tell Kubernetes to create
64+
the checkpoint. As described in the previous article this requires access to the
65+
*kubelet* only `checkpoint` API endpoint.
66+
67+
For a container named *counter* in a pod named *counters* in a namespace named
68+
*default* the *kubelet* API endpoint is reachable at:
69+
70+
```shell
71+
# run this on the node where that Pod is executing
72+
curl -X POST "https://localhost:10250/checkpoint/default/counters/counter"
73+
```
74+
75+
For completeness the following `curl` command-line options are necessary to
76+
have `curl` accept the *kubelet*'s self signed certificate and authorize the
77+
use of the *kubelet* `checkpoint` API:
78+
79+
```shell
80+
--insecure --cert /var/run/kubernetes/client-admin.crt --key /var/run/kubernetes/client-admin.key
81+
```
82+
83+
Once the checkpointing has finished the checkpoint should be available at
84+
`/var/lib/kubelet/checkpoints/checkpoint-<pod-name>_<namespace-name>-<container-name>-<timestamp>.tar`
85+
86+
In the following steps of this article I will use the name `checkpoint.tar`
87+
when analyzing the checkpoint archive.
88+
89+
## Checkpoint archive analysis using `checkpointctl`
90+
91+
To get some initial information about the checkpointed container I am using the
92+
tool [checkpointctl][checkpointctl] like this:
93+
94+
```console
95+
$ checkpointctl show checkpoint.tar --print-stats
96+
+-----------+----------------------------------+--------------+---------+---------------------+--------+------------+------------+-------------------+
97+
| CONTAINER | IMAGE | ID | RUNTIME | CREATED | ENGINE | IP | CHKPT SIZE | ROOT FS DIFF SIZE |
98+
+-----------+----------------------------------+--------------+---------+---------------------+--------+------------+------------+-------------------+
99+
| counter | quay.io/adrianreber/counter:blog | 059a219a22e5 | runc | 2023-03-02T06:06:49 | CRI-O | 10.88.0.23 | 8.6 MiB | 3.0 KiB |
100+
+-----------+----------------------------------+--------------+---------+---------------------+--------+------------+------------+-------------------+
101+
CRIU dump statistics
102+
+---------------+-------------+--------------+---------------+---------------+---------------+
103+
| FREEZING TIME | FROZEN TIME | MEMDUMP TIME | MEMWRITE TIME | PAGES SCANNED | PAGES WRITTEN |
104+
+---------------+-------------+--------------+---------------+---------------+---------------+
105+
| 100809 us | 119627 us | 11602 us | 7379 us | 7800 | 2198 |
106+
+---------------+-------------+--------------+---------------+---------------+---------------+
107+
```
108+
109+
This gives me already some information about the checkpoint in that checkpoint
110+
archive. I can see the name of the container, information about the container
111+
runtime and container engine. It also lists the size of the checkpoint (`CHKPT
112+
SIZE`). This is mainly the size of the memory pages included in the checkpoint,
113+
but there is also information about the size of all changed files in the
114+
container (`ROOT FS DIFF SIZE`).
115+
116+
The additional parameter `--print-stats` decodes information in the checkpoint
117+
archive and displays them in the second table (*CRIU dump statistics*). This
118+
information is collected during checkpoint creation and gives an overview how much
119+
time CRIU needed to checkpoint the processes in the container and how many
120+
memory pages were analyzed and written during checkpoint creation.
121+
122+
## Digging deeper
123+
124+
With the help of `checkpointctl` I am able to get some high level information
125+
about the checkpoint archive. To be able to analyze the checkpoint archive
126+
further I have to extract it. The checkpoint archive is a *tar* archive and can
127+
be extracted with the help of `tar xf checkpoint.tar`.
128+
129+
Extracting the checkpoint archive will result in following files and directories:
130+
131+
* `bind.mounts` - this file contains information about bind mounts and is needed
132+
during restore to mount all external files and directories at the right location
133+
* `checkpoint/` - this directory contains the actual checkpoint as created by
134+
CRIU
135+
* `config.dump` and `spec.dump` - these files contain metadata about the container
136+
which is needed during restore
137+
* `dump.log` - this file contains the debug output of CRIU created during
138+
checkpointing
139+
* `stats-dump` - this file contains the data which is used by `checkpointctl`
140+
to display dump statistics (`--print-stats`)
141+
* `rootfs-diff.tar` - this file contains all changed files on the container's
142+
file-system
143+
144+
### File-system changes - `rootfs-diff.tar`
145+
146+
The first step to analyze the container's checkpoint further is to look at
147+
the files that have changed in my container. This can be done by looking at the
148+
file `rootfs-diff.tar`:
149+
150+
```console
151+
$ tar xvf rootfs-diff.tar
152+
home/counter/logfile
153+
home/counter/test-file
154+
```
155+
156+
Now the files that changed in the container can be studied:
157+
158+
```console
159+
$ cat home/counter/logfile
160+
10.88.0.1 - - [02/Mar/2023 06:07:29] "GET /create?test-file HTTP/1.1" 200 -
161+
10.88.0.1 - - [02/Mar/2023 06:07:40] "GET /secret?RANDOM_1432_KEY HTTP/1.1" 200 -
162+
10.88.0.1 - - [02/Mar/2023 06:07:43] "GET / HTTP/1.1" 200 -
163+
$ cat home/counter/test-file
164+
test-file 
165+
```
166+
167+
Compared to the container image (`quay.io/adrianreber/counter:blog`) this
168+
container is based on, I can see that the file `logfile` contains information
169+
about all access to the service the container provides and the file `test-file`
170+
was created just as expected.
171+
172+
With the help of `rootfs-diff.tar` it is possible to inspect all files that
173+
were created or changed compared to the base image of the container.
174+
175+
### Analyzing the checkpointed processes - `checkpoint/`
176+
177+
The directory `checkpoint/` contains data created by CRIU while checkpointing
178+
the processes in the container. The content in the directory `checkpoint/`
179+
consists of different [image files][image-files] which can be analyzed with the
180+
help of the tool [CRIT][crit] which is distributed as part of CRIU.
181+
182+
First lets get an overview of the processes inside of the container:
183+
184+
```console
185+
$ crit show checkpoint/pstree.img | jq .entries[].pid
186+
1
187+
7
188+
8
189+
```
190+
191+
This output means that I have three processes inside of the container's PID
192+
namespace with the PIDs: 1, 7, 8
193+
194+
This is only the view from the inside of the container's PID namespace. During
195+
restore exactly these PIDs will be recreated. From the outside of the
196+
container's PID namespace the PIDs will change after restore.
197+
198+
The next step is to get some additional information about these three processes:
199+
200+
```console
201+
$ crit show checkpoint/core-1.img | jq .entries[0].tc.comm
202+
"bash"
203+
$ crit show checkpoint/core-7.img | jq .entries[0].tc.comm
204+
"counter.py"
205+
$ crit show checkpoint/core-8.img | jq .entries[0].tc.comm
206+
"tee"
207+
```
208+
209+
This means the three processes in my container are `bash`, `counter.py` (a Python
210+
interpreter) and `tee`. For details about the parent child relations of these processes there
211+
is more data to be analyzed in `checkpoint/pstree.img`.
212+
213+
Let's compare the so far collected information to the still running container:
214+
215+
```console
216+
$ crictl inspect --output go-template --template "{{(index .info.pid)}}" 059a219a22e56
217+
722520
218+
$ ps auxf | grep -A 2 722520
219+
fedora 722520 \_ bash -c /home/counter/counter.py 2>&1 | tee /home/counter/logfile
220+
fedora 722541 \_ /usr/bin/python3 /home/counter/counter.py
221+
fedora 722542 \_ /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee /home/counter/logfile
222+
$ cat /proc/722520/comm
223+
bash
224+
$ cat /proc/722541/comm
225+
counter.py
226+
$ cat /proc/722542/comm
227+
tee
228+
```
229+
230+
In this output I am first retrieving the PID of the first process in the
231+
container and then I am looking for that PID and child processes on the system
232+
where the container is running. I am seeing three processes and the first one is
233+
"bash" which is PID 1 inside of the containers PID namespace. Then I am looking
234+
at `/proc/<PID>/comm` and I can find the exact same value
235+
as in the checkpoint image.
236+
237+
Important to remember is that the checkpoint will contain the view from within the
238+
container's PID namespace because that information is important to restore the
239+
processes.
240+
241+
One last example of what `crit` can tell us about the container is the information
242+
about the UTS namespace:
243+
244+
```console
245+
$ crit show checkpoint/utsns-12.img
246+
{
247+
"magic": "UTSNS",
248+
"entries": [
249+
{
250+
"nodename": "counters",
251+
"domainname": "(none)"
252+
}
253+
]
254+
}
255+
```
256+
257+
This tells me that the hostname inside of the UTS namespace is `counters`.
258+
259+
For every resource CRIU collected during checkpointing the `checkpoint/`
260+
directory contains corresponding image files which can be analyzed with the help
261+
of `crit`.
262+
263+
#### Looking at the memory pages
264+
265+
In addition to the information from CRIU that can be decoded with the help
266+
of CRIT, there are also files containing the raw memory pages written by
267+
CRIU to disk:
268+
269+
```console
270+
$ ls checkpoint/pages-*
271+
checkpoint/pages-1.img checkpoint/pages-2.img checkpoint/pages-3.img
272+
```
273+
274+
When I initially used the container I stored a random key (`RANDOM_1432_KEY`)
275+
somewhere in the memory. Let see if I can find it:
276+
277+
```console
278+
$ grep -ao RANDOM_1432_KEY checkpoint/pages-*
279+
checkpoint/pages-2.img:RANDOM_1432_KEY
280+
```
281+
282+
And indeed, there is my data. This way I can easily look at the content
283+
of all memory pages of the processes in the container, but it is also
284+
important to remember that anyone that can access the checkpoint
285+
archive has access to all information that was stored in the memory of the
286+
container's processes.
287+
288+
#### Using gdb for further analysis
289+
290+
Another possibility to look at the checkpoint images is `gdb`. The CRIU repository
291+
contains the script [coredump][criu-coredump] which can convert a checkpoint
292+
into a coredump file:
293+
294+
```console
295+
$ /home/criu/coredump/coredump-python3
296+
$ ls -al core*
297+
core.1 core.7 core.8
298+
```
299+
300+
Running the `coredump-python3` script will convert the checkpoint images into
301+
one coredump file for each process in the container. Using `gdb` I can also look
302+
at the details of the processes:
303+
304+
```console
305+
$ echo info registers | gdb --core checkpoint/core.1 -q
306+
307+
[New LWP 1]
308+
309+
Core was generated by `bash -c /home/counter/counter.py 2>&1 | tee /home/counter/logfile'.
310+
311+
#0 0x00007fefba110198 in ?? ()
312+
(gdb)
313+
rax 0x3d 61
314+
rbx 0x8 8
315+
rcx 0x7fefba11019a 140667595587994
316+
rdx 0x0 0
317+
rsi 0x7fffed9c1110 140737179816208
318+
rdi 0xffffffff 4294967295
319+
rbp 0x1 0x1
320+
rsp 0x7fffed9c10e8 0x7fffed9c10e8
321+
r8 0x1 1
322+
r9 0x0 0
323+
r10 0x0 0
324+
r11 0x246 582
325+
r12 0x0 0
326+
r13 0x7fffed9c1170 140737179816304
327+
r14 0x0 0
328+
r15 0x0 0
329+
rip 0x7fefba110198 0x7fefba110198
330+
eflags 0x246 [ PF ZF IF ]
331+
cs 0x33 51
332+
ss 0x2b 43
333+
ds 0x0 0
334+
es 0x0 0
335+
fs 0x0 0
336+
gs 0x0 0
337+
```
338+
339+
In this example I can see the value of all registers as they were during
340+
checkpointing and I can also see the complete command-line of my container's PID
341+
1 process: `bash -c /home/counter/counter.py 2>&1 | tee /home/counter/logfile`
342+
343+
## Summary
344+
345+
With the help of container checkpointing, it is possible to create a
346+
checkpoint of a running container without stopping the container and without the
347+
container knowing that it was checkpointed. The result of checkpointing a
348+
container in Kubernetes is a checkpoint archive; using different tools like
349+
`checkpointctl`, `tar`, `crit` and `gdb` the checkpoint can be analyzed. Even
350+
with simple tools like `grep` it is possible to find information in the
351+
checkpoint archive.
352+
353+
The different examples I have shown in this article how to analyze a checkpoint
354+
are just the starting point. Depending on your requirements it is possible to
355+
look at certain things in much more detail, but this article should give you an
356+
introduction how to start the analysis of your checkpoint.
357+
358+
## How do I get involved?
359+
360+
You can reach SIG Node by several means:
361+
362+
* Slack: [#sig-node][slack-sig-node]
363+
* Slack: [#sig-security][slack-sig-security]
364+
* [Mailing list][sig-node-ml]
365+
366+
[forensic-blog]: https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/
367+
[checkpointctl]: https://github.com/checkpoint-restore/checkpointctl
368+
[image-files]: https://criu.org/Images
369+
[crit]: https://criu.org/CRIT
370+
[slack-sig-node]: https://kubernetes.slack.com/messages/sig-node
371+
[slack-sig-security]: https://kubernetes.slack.com/messages/sig-security
372+
[sig-node-ml]: https://groups.google.com/forum/#!forum/kubernetes-sig-node
373+
[criu-coredump]: https://github.com/checkpoint-restore/criu/tree/criu-dev/coredump

0 commit comments

Comments
 (0)