Skip to content

Commit c388c1c

Browse files
author
Pavel Siska
committed
WIP
1 parent a302b38 commit c388c1c

File tree

1 file changed

+307
-37
lines changed

1 file changed

+307
-37
lines changed

‎README.md

Lines changed: 307 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -6,75 +6,345 @@
66
</div>
77
</br>
88

9-
The ipfixprobe is a high-performance flow exporter. It creates bidirectional flows from packet input and exports them to output interface. The ipfixprobe support vide variety of flow extenstion for application layer protocol information. The flow extension can be turned on with process plugins. We support TLS, QUIC, HTTP, DNS and many more. Just check our [documentation](https://ipfixprobe.cesnet.cz).
9+
The ipfixprobe is a high-performance, modular flow exporter that processes packets into bidirectional flows and exports them via a selected output plugin.
10+
It supports a wide range of application-layer protocol parsers, including TLS, QUIC, HTTP, DNS, and many others. These protocol-specific extensions can be enabled via process plugins in the configuration.
11+
12+
Need more details? Check out our [documentation](https://ipfixprobe.cesnet.cz) for a full list of supported protocols and usage examples.
1013

1114
[![](https://img.shields.io/badge/license-BSD-blue.svg)](https://github.com/CESNET/ipfixprobe/blob/master/LICENSE)
1215
[![](https://img.shields.io/badge/docs-ipfixprobe-blue.svg)](https://ipfixprobe.cesnet.cz)
1316
![Coverity Scan](https://img.shields.io/coverity/scan/22112)
1417
![GitHub top language](https://img.shields.io/github/languages/top/CESNET/ipfixprobe)
1518

19+
## ✨ Key Features
20+
21+
- Modular input–parser–output architecture
22+
- High-speed processing (DPDK, multi-threaded, NUMA-aware)
23+
- Built-in protocol parsers: TLS, QUIC, HTTP, DNS, …
24+
- Bidirectional flow (biflow) support
25+
- Real-time telemetry and statistics
26+
27+
## 📦 Installation
28+
29+
If you are running a RHEL system or one of its derivatives (e.g. Oracle Linux, Rocky Linux, CentOS Stream), the easiest way to install
30+
ipfixprobe is from our [copr repository](https://copr.fedorainfracloud.org/coprs/g/CESNET/ipfixprobe/).
31+
32+
```bash
33+
$ dnf install dnf-plugins-core # Extra step necessary on some systems
34+
$ dnf copr enable @CESNET/ipfixprobe
35+
$ dnf install ipfixprobe
36+
```
37+
38+
This installs the main `ipfixprobe` binary along with core functionality.
39+
40+
#### 🗃️ Available Packages
41+
42+
The Copr repository provides modular RPM packages, so you can install only what you need.
43+
The following packages are available and can be installed individually as needed:
44+
45+
| Package Name | Description |
46+
|---------------------------------------------|-----------------------------------------------------------------------------|
47+
| `ipfixprobe` | Core binary with common process/output plugins. |
48+
| `ipfixprobe-msec` | Core binary with common process/output plugins. Uses millisecond timestamps (compatible with Flowmon collector) |
49+
| `ipfixprobe-input-pcap` | Input plugin for PCAP files and live capture |
50+
| `ipfixprobe-input-dpdk` | High-speed input plugin using DPDK |
51+
| `ipfixprobe-input-nfb` | Input plugin for CESNET NFB/NDP cards |
52+
| `ipfixprobe-process-experimental` | Extra (possibly unstable) process plugins |
53+
54+
For other systems, follow the build instructions below.
55+
56+
## 🛠️ Build
57+
58+
You can build ipfixprobe from source using standard CMake.
59+
This lets you customize the build by enabling optional plugins and features as needed.
60+
61+
**Note:** Some plugins may require additional dependencies beyond the basic requirements.
62+
63+
### RHEL/CentOS:
64+
65+
#### 🧰 Requirements
66+
67+
```bash
68+
$ dnf install epel-release git make cmake gcc-c++ rpm-build
69+
$ dnf install libunwind-devel lz4-devel openssl-devel fuse3-devel
70+
71+
# for RHEL 8/9
72+
$ dnf install gcc-toolset-14-libatomic-devel
73+
74+
# for RHEL 10+
75+
$ dnf install libatomic
76+
```
77+
78+
### Debian/Ubuntu:
79+
#### 🧰 Requirements
80+
81+
TODO
82+
83+
#### 🧱 Build steps
84+
85+
```bash
86+
git clone https://github.com/CESNET/ipfixprobe.git
87+
cd ipfixprobe
88+
mkdir build && cd build
89+
cmake ..
90+
make -j$(nproc)
91+
# make install
92+
```
93+
94+
#### ⚙️ Optional build flags
95+
You can enable or disable optional plugins and features via CMake flags:
96+
97+
| Flag | Default | Description |
98+
| ---------------------------------- | ------- | ---------------------------------------------------------------- |
99+
| `-DENABLE_MILLISECONDS_TIMESTAMP=ON` | OFF | Use millisecond precision timestamps (for Flowmon compatibility) |
100+
| `-DENABLE_INPUT_PCAP=ON` | OFF | Enable PCAP input plugin (live & file) (requires `libpcap`) |
101+
| `-DENABLE_INPUT_DPDK=ON` | OFF | Enable high-speed DPDK input plugin (requires `dpdk-devel`) |
102+
| `-DENABLE_INPUT_NFB=ON` | OFF | Enable input plugin for CESNET NFB/NDP cards (requires `netcope-common`) |
103+
| `-DENABLE_PROCESS_EXPERIMENTAL=ON` | OFF | Enable experimental process plugins |
104+
| `-DENABLE_NEMEA=ON` | OFF | Enable support for NEMEA modules (requires `nemea-framework-devel` ) |
105+
106+
Run the command to view all available build options:
107+
108+
```bash
109+
cmake -LAH
110+
```
111+
112+
#### Example
113+
To build with DPDK and PCAP input support, and install to /usr:
114+
115+
```cmake
116+
cmake .. \
117+
-DCMAKE_INSTALL_PREFIX=/usr \
118+
-DENABLE_INPUT_PCAP=ON \
119+
-DENABLE_INPUT_DPDK=ON
120+
```
121+
122+
123+
## 🧩 Available Plugins
16124

17-
## 🛠️ Installation
18-
The RPM packages for RHEL based distributions can be downloaded from our [copr repository](https://copr.fedorainfracloud.org/coprs/g/CESNET/ipfixprobe/package/ipfixprobe/). Or just simply run:
125+
### Input Plugins
126+
List of input plugins with estimated performance and configuration complexity.
19127

128+
| Plugin | Max Throughput | Usage Complexity | Description |
129+
|---------------|----------------|------------------|-------------------------------------------|
130+
| [`pcap_live`](./src/plugins/input/pcap/README.md#pcap-live-input-plugin) | ~1 Gbps | Easy | captures packets from a live network interface |
131+
| [`pcap_file`](./src/plugins/input/pcap/README.md#pcap-file-input-plugin) | ~1 Gbps | Easy | reads packets from an offline PCAP file |
132+
| [`raw`](./src/plugins/input/raw/README.md) | ~1 Gbps | Easy | captures packets using a raw socket |
133+
| [`ndp`](./src/plugins/input/nfb/README.md) | 400 Gbps | Medium | uses CESNET NFB/NDP hardware for packet input |
134+
| [`dpdk`](./src/plugins/input/dpdk/README.md#dpdk-input-plugin) | 400 Gbps | Complex | receives packets via high-performance DPDK |
135+
| [`dpdk-ring`](./src/plugins/input/dpdk/README.md) | 400 Gbps | Complex | receives packets from a shared DPDK memory ring |
136+
137+
---
138+
139+
### Process Plugins
140+
141+
These plugins extract protocol-specific or behavioral information from packets and enrich flow records with metadata.
142+
143+
| Plugin | Description |
144+
|---------------|--------------------------------------------------------------|
145+
| [`basic`](./src/plugins/process/basic/README.md) | extracts basic L3/L4 flow fields (IPs, ports, protocol) |
146+
| [`icmp`](./src/plugins/process/icmp/README.md) | extracts ICMP type/code and related metadata |
147+
| [`http`](./src/plugins/process/http/README.md) | extracts HTTP methods, hosts, URIs, status codes |
148+
| [`tls`](./src/plugins/process/tls/README.md) | extracts TLS handshake info (SNI, version, JA3, etc.) |
149+
| [`ovpn`](./src/plugins/process/ovpn/README.md) | extracts metadata from OpenVPN tunnels |
150+
| [`wg`](./src/plugins/process/wg/README.md) | parses WireGuard handshake and endpoint metadata |
151+
| [`quic`](./src/plugins/process/quic/README.md) | parses QUIC protocol including SNI, versions, ALPN |
152+
| [`basicplus`](./src/plugins/process/basicplus/README.md) | adds common L3/L4 flow fields (e.g., ports, IPs, TCP flags) |
153+
| [`bstats`](./src/plugins/process/bstats/README.md) | basic flow statistics (packet/byte counters, duration, ...) |
154+
| [`dns`](./src/plugins/process/dns/README.md) | extracts DNS queries, responses, and domains |
155+
| [`dnssd`](./src/plugins/process/dnssd/README.md) | parses DNS Service Discovery (mDNS) traffic |
156+
| [`flowHash`](./src/plugins/process/flowHash/README.md) | extracts a flow hash |
157+
| [`idpContent`](./src/plugins/process/idpContent/README.md) | parses IDP content in flows |
158+
| [`mpls`](./src/plugins/process/mpls/README.md) | extracts MPLS labels and encapsulation metadata |
159+
| [`mqtt`](./src/plugins/process/mqtt/README.md) | parses MQTT protocol traffic (IoT messaging) |
160+
| [`netbios`](./src/plugins/process/netbios/README.md) | extracts NetBIOS session and name service info |
161+
| [`nettisa`](./src/plugins/process/nettisa/README.md) | parses NETTISA related metadata (experimental) |
162+
| [`ntp`](./src/plugins/process/ntp/README.md) | extracts NTP timestamps and server info |
163+
| [`osquery`](./src/plugins/process/osquery/README.md) | parses osquery-generated data streams |
164+
| [`passiveDns`](./src/plugins/process/passiveDns/README.md) | generates passive DNS entries from observed DNS traffic |
165+
| [`phists`](./src/plugins/process/phists/README.md) | parses phishing-related signatures (heuristic) |
166+
| [`pstats`](./src/plugins/process/pstats/README.md) | advanced packet statistics (e.g., inter-packet gaps) |
167+
| [`rtsp`](./src/plugins/process/rtsp/README.md) | extracts RTSP stream metadata |
168+
| [`sip`](./src/plugins/process/sip/README.md) | parses SIP call setup, headers, and codecs |
169+
| [`smtp`](./src/plugins/process/smtp/README.md) | extracts SMTP envelope data (from, to, subject, etc.) |
170+
| [`ssaDetector`](./src/plugins/process/ssaDetector/README.md) | performs simple anomaly detection based on traffic patterns |
171+
| [`ssdp`](./src/plugins/process/ssdp/README.md) | parses SSDP (UPnP discovery) protocol |
172+
| [`vlan`](./src/plugins/process/vlan/README.md) | extracts VLAN IDs and QinQ encapsulation |
173+
174+
---
175+
### Output Plugins
176+
177+
These plugins export flow records to various formats and external systems.
178+
179+
| Plugin | Description |
180+
|---------------|-----------------------------------------------------------------------------|
181+
| [`ipfix`](./src/plugins/output/ipfix/README.md) | exports flow records in IPFIX format to a remote collector (UDP/TCP) |
182+
| [`text`](./src/plugins/output/text/README.md) | writes flow records in human-readable text to a file or stdout |
183+
| [`unirec`](./src/plugins/output/unirec/README.md) | exports flow records using the UniRec format for NEMEA/TRAP ecosystem |
184+
185+
---
186+
187+
## 📘 Usage
188+
189+
`ipfixprobed` is a lightweight launcher for the `ipfixprobe` processing pipeline, configured via a YAML configuration file.
190+
This method is the preferred and recommended way to define plugins, queues, and runtime behavior.
191+
192+
There are two ways to run the application:
193+
194+
### ✅ Recommended (YAML-based configuration)
195+
196+
```bash
197+
/usr/bin/ipfixprobed <CONFIG_NAME> [LIBRARY_PATH]
20198
```
21-
dnf install -y dnf-plugins-core &&dnf copr enable @CESNET/ipfixprobe
22-
dnf install ipfixprobe
199+
200+
- `<CONFIG_NAME>` – Name of the YAML configuration file (without the .conf extension).
201+
The full path must be `/etc/ipfixprobe/<CONFIG_NAME>.conf`.
202+
203+
- `[LIBRARY_PATH]` – (Optional) Path to the ipfixprobe plugin library directory.
204+
If not provided, the default `/usr/lib64/ipfixprobe` is used.
205+
206+
#### Example:
207+
```bash
208+
/usr/bin/ipfixprobed example
23209
```
24210

25-
## 🔧 Parameters
26-
### Module specific parameters
27-
- `-i ARGS` Activate input plugin (-h input for help)
28-
- `-s ARGS` Activate storage plugin (-h storage for help)
29-
- `-o ARGS` Activate output plugin (-h output for help)
30-
- `-p ARGS` Activate processing plugin (-h process for help)
211+
This will run `ipfixprobe` using the YAML configuration from `/etc/ipfixprobe/example.conf`.
212+
213+
👉 See the full YAML configuration reference here:
214+
215+
### ⚠️ Legacy usage (CLI parameters)
216+
217+
You may also run the processing pipeline using the `ipfixprobe` binary directly, with CLI parameters.
218+
This method is not recommended for production use, as it lacks flexibility and clarity.
219+
220+
```bash
221+
/usr/bin/ipfixprobe [OPTIONS]
222+
```
223+
224+
#### Available options:
225+
226+
- `-i ARGS` Activate input plugin (`-h input` for help)
227+
- `-s ARGS` Activate storage plugin (`-h storage` for help)
228+
- `-o ARGS` Activate output plugin (`-h output` for help)
229+
- `-p ARGS` Activate processing plugin (`-h process` for help)
31230
- `-q SIZE` Size of queue between input and storage plugins
32231
- `-b SIZE` Size of input queue packet block
33232
- `-Q SIZE` Size of queue between storage and output plugins
34233
- `-B SIZE` Size of packet buffer
35234
- `-f NUM` Export max flows per second
36235
- `-c SIZE` Quit after number of packets are processed on each interface
37-
- `-P FILE` Create pid file
236+
- `-P FILE` Create a PID file
38237
- `-t PATH` Mount point of AppFs telemetry directory
39238
- `-d` Run as a standalone process
40239
- `-h [PLUGIN]` Print help text. Supported help for input, storage, output and process plugins
41240
- `-V` Show version and exit
42241

43-
### ❓ Help
44-
Printing general help is done using the `-h` parameter. To print help for specific plugins, `-h` with parameter is used.
45-
This parameter accepts `input`, `storage`, `process`, `output` or name of a plugin (or path to a .so file with plugin).
242+
## 📖 Examples
243+
Below are practical examples showcasing common plugin configurations using both CLI and YAML formats.
244+
245+
#### **🔹 Basic Interface Capture**
246+
Capture network traffic from the `wlp2s0` interface using a raw socket. Flow records are printed in plain-text format to the console.
46247

47-
## 📖 Example
48-
Here are the examples of various plugins usage:
248+
🧪 Command-line usage:
49249
```bash
50-
# Capture from wlp2s0 interface using raw sockets, print flows to console
51-
./ipfixprobe -i 'raw;ifc=wlp2s0' -o 'text'
250+
/usr/bin/ipfixprobe -i 'raw;ifc=wlp2s0' -o 'text'
251+
```
52252

53-
# Capture from wlp2s0 interface and scale packet processing using 2 instances of plugins, send flow to ifpfix collector using UDP
54-
./ipfixprobe -i 'raw;ifc=wlp2s0;f' -i 'raw;ifc=wlp2s0;f' -o 'ipfix;u;host=collector.example.com;port=4739'
253+
📄 Equivalent YAML configuration:
254+
```yaml
255+
input_plugin:
256+
raw:
257+
interface: 'wlp2s0'
55258

56-
# Capture from a COMBO card using ndp plugin, sends ipfix data to 127.0.0.1:4739 using TCP by default
57-
./ipfixprobe -i 'ndp;dev=/dev/nfb0:0' -i 'ndp;dev=/dev/nfb0:1' -i 'ndp;dev=/dev/nfb0:2'
259+
output_plugin:
260+
text: {}
261+
```
58262
59-
# Capture from eth0 interface using pcap plugin, split biflows into flows and prints them to console without mac addresses, telemetry data are exposed via the appFs library in /var/run/ipfixprobe directory
60-
./ipfixprobe -i 'pcap;ifc=eth0' -s 'cache;split' -o 'text;m' -t /var/run/ipfixprobe
263+
----
61264
62-
# Read packets from pcap file, enable 4 processing plugins, sends L7 HTTP extended biflows to unirec interface named `http` and data from 3 other plugins to the `stats` interface
63-
./ipfixprobe -i 'pcap;file=pcaps/http.pcap' -p http -p pstats -p idpcontent -p phists -o 'unirec;i=u:http:timeout=WAIT,u:stats:timeout=WAIT;p=http,(pstats,phists,idpcontent)'
265+
#### **🔹 PCAP File → IPFIX Collector**
266+
Process packets from a `.pcap` file and export flows via IPFIX over UDP
267+
🧪 Command-line usage:
268+
```bash
269+
/usr/bin/ipfixprobe -i 'pcap;file=/data/capture.pcap' -o 'ipfix;udp;host=collector.example.com;port=4739'
270+
```
64271

65-
# Read packets using DPDK input interface and 1 DPDK queue, enable plugins for basic statistics, http and tls, output to IPFIX on a local machine
66-
# DPDK EAL parameters are passed in `e, eal` parameters
67-
# DPDK plugin configuration has to be specified in the first input interface.
68-
# The following `dpdk` interfaces are given without parameters; their configuration is inherited from the first one.
69-
# Example for the queue of 3 DPDK input plugins (q=3):
70-
`./ipfixprobe -i "dpdk;p=0;q=3;e=-c 0x1 -a <[domain:]bus:devid.func>" -i dpdk -i dpdk -p http "-p" bstats -p tls -o "ipfix;h=127.0.0.1"`
272+
📄 Equivalent YAML configuration:
273+
```yaml
274+
input_plugin:
275+
pcap_file:
276+
file: '/data/capture.pcap'
277+
output_plugin:
278+
ipfix:
279+
collector:
280+
host: collector.example.com
281+
port: 4739
282+
protocol:
283+
udp: {}
284+
```
71285

72-
# Same example for the multiport read from ports 0 and 1, note comma separated ports:
73-
`./ipfixprobe -i "dpdk;p=0,1;q=3;e=-c 0x1 -a <[domain:]bus:devid.func>" -i dpdk -i dpdk -p http "-p" bstats -p tls -o "ipfix;h=127.0.0.1"`
286+
----
74287

288+
#### **🔹Live Capture with Cache Configuration and Telemetry**
289+
Capture from `eth0` interface using libpcap, split biflows into uniflows and use active/inactive timeouts, print flows to console.
290+
Telemetry data are exposed via the appFs library in `/var/run/ipfixprobe` directory.
291+
292+
🧪 Command-line usage:
293+
```bash
294+
/usr/bin/ipfixprobe -i 'pcap;ifc=eth0' -s 'cache;split;active=300;inactive=60' -o 'text' -t /var/run/ipfixprobe
295+
```
296+
297+
📄 Equivalent YAML configuration:
298+
```yaml
299+
input_plugin:
300+
pcap_live:
301+
interface: 'eth0'
302+
303+
storage:
304+
cache: {}
305+
timeouts:
306+
active: 300
307+
inactive: 60
308+
split_biflow: true
309+
310+
output_plugin:
311+
text: {}
312+
313+
telemetry:
314+
appfs:
315+
enabled: true
316+
mount_point: /var/run/ipfixprobe
317+
```
318+
319+
----
320+
321+
#### 🔹 High-speed DPDK Capture with HTTP, TLS, and QUIC Processing
322+
Capture packets using DPDK from port 0 with 2 queues bound to a specific PCI device (`0000:17:00.0`), enable HTTP, TLS, and QUIC process plugins, and export flows via IPFIX to a local collector at 127.0.0.1.
323+
324+
🧪 Command-line usage:
325+
```bash
326+
/usr/bin/ipfixprobe -i "dpdk;p=0;q=2;e=-a 0000:17:00.0" -i dpdk -p http -p tls -p quic -o "ipfix;h=127.0.0.1"
327+
```
75328

76-
# Read packets using DPDK input interface as secondary process with shared memory (DPDK rings) - in this case, 4 DPDK rings are used
77-
`./ipfixprobe -i 'dpdk-ring;r=rx_ipfixprobe_0;e= --proc-type=secondary' -i 'dpdk-ring;r=rx_ipfixprobe_1' -i 'dpdk-ring;r=rx_ipfixprobe_2' -i 'dpdk-ring;r=rx_ipfixprobe_3' -o 'text'`
329+
📄 Equivalent YAML configuration:
330+
```yaml
331+
input_plugin:
332+
dpdk:
333+
allowed_nics: "0000:17:00.0"
334+
rx_queues: 2
335+
336+
process_plugins:
337+
- http
338+
- tls
339+
- quic
340+
341+
output_plugin:
342+
ipfix:
343+
collector:
344+
host: '127.0.0.1'
345+
port: 4739
346+
protocol:
347+
udp: {}
78348
```
79349

80350
## Build

0 commit comments

Comments
 (0)