Skip to content

Commit c413ab6

Browse files
aireillyaireilly
committed
more modularization of modules
1 parent 9a42115 commit c413ab6

6 files changed

+169
-164
lines changed

‎main.adoc‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,14 @@ include::modules/cnf-best-practices-linux-capabilities.adoc[leveloffset=+2]
8080

8181
include::modules/cnf-best-practices-openshift-operations.adoc[leveloffset=+2]
8282

83+
include::modules/cnf-best-practices-operations-that-can-not-be-executed-by-openshift.adoc[leveloffset=+2]
84+
85+
include::modules/cnf-best-practices-analyzing-your-application.adoc[leveloffset=+2]
86+
87+
include::modules/cnf-best-practices-finding-the-capabilities-that-an-application-needs.adoc[leveloffset=+2]
88+
89+
include::modules/cnf-best-practices-cnf-securing-cnf-networks.adoc[leveloffset=+2]
90+
8391
include::modules/cnf-best-practices-secrets-management.adoc[leveloffset=+3]
8492

8593
include::modules/cnf-best-practices-scc-permissions-for-an-application.adoc[leveloffset=+3]

‎modules/cnf-best-practices-analyzing-your-application.adoc‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
[id="cnf-best-practices-analyzing-your-application"]
2+
= Analyzing your application
3+
4+
To find out which capabilities the application needs, Red Hat has developed a SystemTap script (`container_check.stp`). With this tool, the CNF developer can find out what capabilities an application requires in order to run in a container. It also shows the syscalls which were invoked. Find more info at link:https://linuxera.org/capabilities-seccomp-kubernetes/[]
5+
6+
Another tool is `capable` which is part of the BCC tools. It can be installed on RHEL8 with `dnf install bcc`.

‎modules/cnf-best-practices-cnf-securing-cnf-networks.adoc‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
[id="cnf-best-practices-cnf-securing-cnf-networks"]
2+
= Securing CNF networks
3+
4+
CNFs must have the least permissions possible and CNFs must implement Network Policies that drop all traffic by default and permit only the relevant ports and protocols to the narrowest ranges of addresses possible.
5+
6+
.CNF requirement
7+
[IMPORTANT]
8+
====
9+
Applications must define network policies that permit only the minimum network access the application needs to function.
10+
11+
See test case link:https://github.com/test-network-function/cnf-certification-test/blob/main/CATALOG.md#networking-network-policy-deny-all[networking-network-policy-deny-all]
12+
====

‎modules/cnf-best-practices-finding-the-capabilities-that-an-application-needs.adoc‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
[id="cnf-best-practices-finding-the-capabilities-that-an-application-needs"]
2+
= Finding the capabilities that an application needs
3+
4+
Here is an example of how to find out the capabilities that an application needs. `testpmd` is a DPDK based layer-2 forwarding application. It needs the `CAP_IPC_LOCK` to allocate the hugepage memory.
5+
6+
. Use container_check.stp. We can see `CAP_IPC_LOCK` and `CAP_SYS_RAWIO` are requested by `testpmd` and the relevant syscalls.
7+
+
8+
[source,terminal]
9+
----
10+
$ $ /usr/share/systemtap/examples/profiling/container_check.stp -c 'testpmd -l 1-2 -w 0000:00:09.0 -- -a --portmask=0x8 --nb-cores=1'
11+
----
12+
+
13+
.Example output
14+
[source,terminal]
15+
----
16+
[...]
17+
capabilities used by executables
18+
executable: prob capability
19+
testpmd: cap_ipc_lock
20+
testpmd: cap_sys_rawio
21+
22+
capabilities used by syscalls
23+
executable, syscall ( capability ) : count
24+
testpmd, mlockall ( cap_ipc_lock ) : 1
25+
testpmd, mmap ( cap_ipc_lock ) : 710
26+
testpmd, open ( cap_sys_rawio ) : 1
27+
testpmd, iopl ( cap_sys_rawio ) : 1
28+
29+
failed syscalls
30+
executable, syscall = errno: count
31+
eal-intr-thread, epoll_wait = EINTR: 1
32+
lcore-slave-2, read = : 1
33+
rte_mp_handle, recvmsg = : 1
34+
stapio, = EINTR: 1
35+
stapio, execve = ENOENT: 3
36+
stapio, rt_sigsuspend = : 1
37+
testpmd, flock = EAGAIN: 5
38+
testpmd, stat = ENOENT: 10
39+
testpmd, mkdir = EEXIST: 2
40+
testpmd, readlink = ENOENT: 3
41+
testpmd, access = ENOENT: 1141
42+
testpmd, openat = ENOENT: 1
43+
testpmd, open = ENOENT: 13
44+
[...]
45+
----
46+
47+
. Use the `capable` command:
48+
+
49+
[source,terminal]
50+
----
51+
$ /usr/share/bcc/tools/capable
52+
----
53+
54+
. Start the testpmd application from another terminal, and send some test traffic to it. For example:
55+
+
56+
[source,terminal]
57+
----
58+
$ testpmd -l 18-19 -w 0000:01:00.0 -- -a --portmask=0x1 --nb-cores=1
59+
----
60+
61+
. Check the output of the `capable` command. Below, `CAP_IPC_LOCK` was requested for running `testpmd`.
62+
+
63+
[source,terminal]
64+
----
65+
[...]
66+
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
67+
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
68+
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
69+
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
70+
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
71+
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
72+
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
73+
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
74+
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
75+
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
76+
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
77+
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
78+
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
79+
[...]
80+
----
81+
82+
. Also, try to run `testpmd` without `CAP_IPC_LOCK` set with `capsh`. Now we can see that the hugepage memory cannot be allocated.
83+
84+
[source,terminal]
85+
----
86+
$ capsh --drop=cap_ipc_lock -- -c testpmd -l 18-19 -w 0000:01:00.0 -- -a --portmask=0x1 --nb-cores=1
87+
----
88+
+
89+
.Example output
90+
[source,terminal]
91+
----
92+
EAL: Detected 24 lcore(s)
93+
EAL: Detected 2 NUMA nodes
94+
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
95+
EAL: No free hugepages reported in hugepages-1048576kB
96+
EAL: Probing VFIO support...
97+
EAL: VFIO support initialized
98+
EAL: PCI device 0000:01:00.0 on NUMA socket 0
99+
EAL: probe driver: 8086:10fb net_ixgbe
100+
EAL: using IOMMU type 1 (Type 1)
101+
EAL: Ignore mapping IO port bar(2)
102+
EAL: PCI device 0000:01:00.1 on NUMA socket 0
103+
EAL: probe driver: 8086:10fb net_ixgbe
104+
EAL: PCI device 0000:07:00.0 on NUMA socket 0
105+
EAL: probe driver: 8086:1521 net_e1000_igb
106+
EAL: PCI device 0000:07:00.1 on NUMA socket 0
107+
EAL: probe driver: 8086:1521 net_e1000_igb
108+
EAL: cannot set up DMA remapping, error 12 (Cannot allocate memory) testpmd: mlockall() failed with error "Cannot allocate memory" testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=331456, size=2176, socket=0
109+
testpmd: preferred mempool ops selected: ring_mp_mc
110+
EAL: cannot set up DMA remapping, error 12 (Cannot allocate memory) testpmd: create a new mbuf pool <mbuf_pool_socket_1>: n=331456, size=2176,
111+
socket=1
112+
testpmd: preferred mempool ops selected: ring_mp_mc
113+
EAL: cannot set up DMA remapping, error 12 (Cannot allocate memory) EAL: cannot set up DMA remapping, error 12 (Cannot allocate memory)
114+
----

‎modules/cnf-best-practices-openshift-operations.adoc‎

Lines changed: 0 additions & 164 deletions
Original file line numberDiff line numberDiff line change
@@ -53,167 +53,3 @@ In OpenShift, multicast is supported for both the default interface (OVN or Open
5353
* link:https://docs.openshift.com/container-platform/latest/networking/hardware_networks/using-sriov-multicast.html#nw-using-an-sriov-interface-for-multicast_using-sriov-multicast[Configuring an SR-IOV interface for multicast]
5454
* If your application works as a multicast source and you want to utilize the additional interfaces to carry the multicast traffic, then you don’t need the `NET_ADMIN` capability. Follow the instructions in link:https://docs.openshift.com/container-platform/latest/networking/hardware_networks/using-sriov-multicast.html[Using high performance multicast] to set the correct multicast route in the pod’s routing table.
5555

56-
[id="cnf-best-practices-operations-that-can-not-be-executed-by-openshift"]
57-
== Operations that can not be executed by OpenShift
58-
59-
All the CNI plugins are only invoked during pod creation and deletion. If your CNF needs perform any operations mentioned above at runtime, the `NET_ADMIN` capability is required.
60-
61-
There are some other functionalities that are not currently supported by any of the OpenShift components which also require `NET_ADMIN` capability:
62-
63-
* Link state modification at runtime
64-
65-
* IP/MAC modification at runtime
66-
67-
* Manipulate pod’s route table or firewall rules at runtime
68-
69-
* SR/IOV VF setting at runtime
70-
71-
* Netlink configuration
72-
73-
* For example, `ethtool` can be used to configure things like rxvlan, txvlan, gso, tso, etc.
74-
75-
* Multicast
76-
+
77-
[NOTE]
78-
====
79-
If your application works as a receiving member of IGMP groups, you need to specify the NET_ADMIN capability in the pod manifest. So that the app is allowed to assign multicast addresses to the pod interface and join an IGMP group.
80-
====
81-
82-
* Set `SO_PRIORITY` to a socket to manipulate the 802.1p priority in ethernet frames
83-
84-
* Set `IP_TOS` to a socket to manipulate the DSCP value of IP packets
85-
86-
[id="cnf-best-practices-analyzing-your-application"]
87-
== Analyzing your application
88-
89-
To find out which capabilities the application needs, Red Hat has developed a SystemTap script (`container_check.stp`). With this tool, the CNF developer can find out what capabilities an application requires in order to run in a container. It also shows the syscalls which were invoked. Find more info at link:https://linuxera.org/capabilities-seccomp-kubernetes/[]
90-
91-
Another tool is `capable` which is part of the BCC tools. It can be installed on RHEL8 with `dnf install bcc`.
92-
93-
[id="cnf-best-practices-example"]
94-
=== Finding the capabilities that an application needs
95-
96-
Here is an example of how to find out the capabilities that an application needs. `testpmd` is a DPDK based layer-2 forwarding application. It needs the `CAP_IPC_LOCK` to allocate the hugepage memory.
97-
98-
. Use container_check.stp. We can see `CAP_IPC_LOCK` and `CAP_SYS_RAWIO` are requested by `testpmd` and the relevant syscalls.
99-
+
100-
[source,terminal]
101-
----
102-
$ $ /usr/share/systemtap/examples/profiling/container_check.stp -c 'testpmd -l 1-2 -w 0000:00:09.0 -- -a --portmask=0x8 --nb-cores=1'
103-
----
104-
+
105-
.Example output
106-
[source,terminal]
107-
----
108-
[...]
109-
capabilities used by executables
110-
executable: prob capability
111-
testpmd: cap_ipc_lock
112-
testpmd: cap_sys_rawio
113-
114-
capabilities used by syscalls
115-
executable, syscall ( capability ) : count
116-
testpmd, mlockall ( cap_ipc_lock ) : 1
117-
testpmd, mmap ( cap_ipc_lock ) : 710
118-
testpmd, open ( cap_sys_rawio ) : 1
119-
testpmd, iopl ( cap_sys_rawio ) : 1
120-
121-
failed syscalls
122-
executable, syscall = errno: count
123-
eal-intr-thread, epoll_wait = EINTR: 1
124-
lcore-slave-2, read = : 1
125-
rte_mp_handle, recvmsg = : 1
126-
stapio, = EINTR: 1
127-
stapio, execve = ENOENT: 3
128-
stapio, rt_sigsuspend = : 1
129-
testpmd, flock = EAGAIN: 5
130-
testpmd, stat = ENOENT: 10
131-
testpmd, mkdir = EEXIST: 2
132-
testpmd, readlink = ENOENT: 3
133-
testpmd, access = ENOENT: 1141
134-
testpmd, openat = ENOENT: 1
135-
testpmd, open = ENOENT: 13
136-
[...]
137-
----
138-
139-
. Use the `capable` command:
140-
+
141-
[source,terminal]
142-
----
143-
$ /usr/share/bcc/tools/capable
144-
----
145-
146-
. Start the testpmd application from another terminal, and send some test traffic to it. For example:
147-
+
148-
[source,terminal]
149-
----
150-
$ testpmd -l 18-19 -w 0000:01:00.0 -- -a --portmask=0x1 --nb-cores=1
151-
----
152-
153-
. Check the output of the `capable` command. Below, `CAP_IPC_LOCK` was requested for running `testpmd`.
154-
+
155-
[source,terminal]
156-
----
157-
[...]
158-
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
159-
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
160-
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
161-
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
162-
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
163-
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
164-
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
165-
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
166-
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
167-
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
168-
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
169-
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
170-
0:41:58 0 3591 testpmd CAP_IPC_LOCK 1
171-
[...]
172-
----
173-
174-
. Also, try to run `testpmd` without `CAP_IPC_LOCK` set with `capsh`. Now we can see that the hugepage memory cannot be allocated.
175-
176-
[source,terminal]
177-
----
178-
$ capsh --drop=cap_ipc_lock -- -c testpmd -l 18-19 -w 0000:01:00.0 -- -a --portmask=0x1 --nb-cores=1
179-
----
180-
+
181-
.Example output
182-
[source,terminal]
183-
----
184-
EAL: Detected 24 lcore(s)
185-
EAL: Detected 2 NUMA nodes
186-
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
187-
EAL: No free hugepages reported in hugepages-1048576kB
188-
EAL: Probing VFIO support...
189-
EAL: VFIO support initialized
190-
EAL: PCI device 0000:01:00.0 on NUMA socket 0
191-
EAL: probe driver: 8086:10fb net_ixgbe
192-
EAL: using IOMMU type 1 (Type 1)
193-
EAL: Ignore mapping IO port bar(2)
194-
EAL: PCI device 0000:01:00.1 on NUMA socket 0
195-
EAL: probe driver: 8086:10fb net_ixgbe
196-
EAL: PCI device 0000:07:00.0 on NUMA socket 0
197-
EAL: probe driver: 8086:1521 net_e1000_igb
198-
EAL: PCI device 0000:07:00.1 on NUMA socket 0
199-
EAL: probe driver: 8086:1521 net_e1000_igb
200-
EAL: cannot set up DMA remapping, error 12 (Cannot allocate memory) testpmd: mlockall() failed with error "Cannot allocate memory" testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=331456, size=2176, socket=0
201-
testpmd: preferred mempool ops selected: ring_mp_mc
202-
EAL: cannot set up DMA remapping, error 12 (Cannot allocate memory) testpmd: create a new mbuf pool <mbuf_pool_socket_1>: n=331456, size=2176,
203-
socket=1
204-
testpmd: preferred mempool ops selected: ring_mp_mc
205-
EAL: cannot set up DMA remapping, error 12 (Cannot allocate memory) EAL: cannot set up DMA remapping, error 12 (Cannot allocate memory)
206-
----
207-
208-
[id="cnf-best-practices-cnf-network-security"]
209-
== Securing CNF networks
210-
211-
CNFs must have the least permissions possible and CNFs must implement Network Policies that drop all traffic by default and permit only the relevant ports and protocols to the narrowest ranges of addresses possible.
212-
213-
.CNF requirement
214-
[IMPORTANT]
215-
====
216-
Applications must define network policies that permit only the minimum network access the application needs to function.
217-
218-
See test case link:https://github.com/test-network-function/cnf-certification-test/blob/main/CATALOG.md#networking-network-policy-deny-all[networking-network-policy-deny-all]
219-
====

‎modules/cnf-best-practices-operations-that-can-not-be-executed-by-openshift.adoc‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
[id="cnf-best-practices-operations-that-can-not-be-executed-by-openshift"]
2+
= Operations that can not be executed by OpenShift
3+
4+
All the CNI plugins are only invoked during pod creation and deletion. If your CNF needs perform any operations mentioned above at runtime, the `NET_ADMIN` capability is required.
5+
6+
There are some other functionalities that are not currently supported by any of the OpenShift components which also require `NET_ADMIN` capability:
7+
8+
* Link state modification at runtime
9+
10+
* IP/MAC modification at runtime
11+
12+
* Manipulate pod’s route table or firewall rules at runtime
13+
14+
* SR/IOV VF setting at runtime
15+
16+
* Netlink configuration
17+
18+
* For example, `ethtool` can be used to configure things like rxvlan, txvlan, gso, tso, etc.
19+
20+
* Multicast
21+
+
22+
[NOTE]
23+
====
24+
If your application works as a receiving member of IGMP groups, you need to specify the NET_ADMIN capability in the pod manifest. So that the app is allowed to assign multicast addresses to the pod interface and join an IGMP group.
25+
====
26+
27+
* Set `SO_PRIORITY` to a socket to manipulate the 802.1p priority in ethernet frames
28+
29+
* Set `IP_TOS` to a socket to manipulate the DSCP value of IP packets

0 commit comments

Comments
 (0)