Skip to content

Commit 63986b9

Browse files
committed
Merge branch 'master' of https://github.com/CESNET/ipfixprobe into rpm-dpdk
2 parents 035a860 + c4019b7 commit 63986b9

File tree

22 files changed

+3129
-25
lines changed

22 files changed

+3129
-25
lines changed

ChangeLog

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,14 @@
1+
2024-10-17 ipfixprobe-4.13.0
2+
* usability: introduce docker/podman container to convert PCAP files to CSV
3+
* IPFIX: fix order of TCP options flags
4+
* basicplus: update TCP options mask across flow packets
5+
* utils: introduce memcpy_le32toh() for ipfix representation
6+
* wg: fix parsing and exporting byte order (IPFIX)
7+
* DPDK-ring: optimization: prefetch; read timestamp from HW metadata if available
8+
* cache: optimization - prefetch
9+
* IPv6: fix header parsing
10+
* DPDK&DPDK-ring: fix use of parse_packet(), skip invalid packets causing crash
11+
112
2024-08-28 ipfixprobe-4.12.0
213
* ipfix plugin: support lz4 compression
314
* ipfixprobe: possibility to set workers affinity

Doxyfile

Lines changed: 2819 additions & 0 deletions
Large diffs are not rendered by default.

Makefile.am

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -283,7 +283,8 @@ EXTRA_DIST=README.md \
283283
pcaps/quic_initial-sample.pcap \
284284
debian/control debian/changelog debian/watch debian/copyright debian/patches debian/patches/series \
285285
debian/source debian/source/format debian/source/local-options debian/source/include-binaries \
286-
debian/rules debian/README.Debian debian/compat
286+
debian/rules debian/README.Debian debian/compat \
287+
docker/Dockerfile docker/ipfixprobe_wrapper.sh docker/process_script.sh docker/README.md
287288

288289
bashcompl_DATA=ipfixprobe.bash
289290

@@ -362,3 +363,7 @@ deb:
362363
else
363364
endif
364365

366+
.PHONY: doc
367+
doc:
368+
doxygen
369+

NEWS

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,23 @@
1+
2024-10-17 (Tomas Cejka): doc: include generated Doxyfile to create documentation
2+
2024-10-17 (Tomas Cejka): doc: add doxygen comment for parse_packet()
3+
2024-10-16 (Jan Sobol): dpdk-ring - fix checking if any packet has actually been parsed
4+
2024-10-16 (Jan Sobol): dpdk - fix checking if any packet has actually been parsed
5+
2024-10-02 (Damir Zainullin): Fix IPv6 header parsing
6+
2024-09-30 (Pavel Šiška): Merge pull request #220 from CESNET/prefetch-optimizations
7+
2024-09-30 (Pavel Šiška): Merge pull request #219 from CESNET/dpdk-ring-metadata-timestamp
8+
2024-09-30 (Pavel Šiška): Merge pull request #215 from CESNET/new-version
9+
2024-09-26 (Jan Sobol): cache - prefetch flow records before checking their expiration
10+
2024-09-26 (Jan Sobol): dpdk-ring - prefetch dequeued packets before processing
11+
2024-09-25 (Jan Sobol): dpdk-ring - read timestamp from hw metadata if available
12+
2024-09-20 (Tomas Cejka): Merge pull request #216 from CESNET/ipfixprobe-docker-wrapper
13+
2024-09-19 (Tomas Cejka): dist: include docker/ files into distribution archive
14+
2024-09-19 (Jan Sobol): wg - fix parsing and exporting byte order
15+
2024-09-19 (Jan Sobol): utils - introduce memcpy_le32toh function
16+
2024-09-18 (Jan Sobol): basicplus test - fix reference values of tcp options
17+
2024-09-18 (Jan Sobol): basicplus - update tcp options mask across flow packets
18+
2024-09-18 (Jan Sobol): parser - fix order of tcp options flags according to ipfix standard https://www.iana.org/assignments/ipfix/ipfix.xhtml, entity 209 - tcpOptions
19+
2024-09-17 (Jaroslav Pesek): process container - introduce docker/podman container wrapper for processing pcaps to csvs
20+
121
2024-08-28 (Pavel Siska): ipfixprobed - add new option LZ4_COMPRESSION to init script and config example
222
2024-08-28 (Pavel Siska): README.md - add LZ4 compression info
323
2024-08-28 (Jakub Antonín Štigler): ipfix plugin: add lz4 compression

configure.ac

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
# Process this file with autoconf to produce a configure script.
33

44
AC_PREREQ([2.69])
5-
AC_INIT([ipfixprobe], [4.12.0], [[email protected]])
5+
AC_INIT([ipfixprobe], [4.13.0], [[email protected]])
66

77
AC_CONFIG_SRCDIR([main.cpp])
88
AC_CONFIG_HEADERS([config.h])

docker/Dockerfile

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
FROM rockylinux:9
2+
3+
RUN dnf install -y dnf-plugins-core && \
4+
dnf copr -y enable @CESNET/NEMEA && \
5+
dnf install -y epel-release && \
6+
dnf install -y ipfixprobe nemea && \
7+
dnf clean all
8+
9+
RUN mkdir -p /output
10+
WORKDIR /output
11+
ENTRYPOINT ["/bin/bash", "-c"]
12+
13+
VOLUME ["/output"]

docker/README.md

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,3 +18,49 @@ To push rpmbuilder image to repository:
1818
podman push rpmbuilder:latest gitlab.liberouter.org:5050/monitoring/ipfixprobe
1919
```
2020

21+
=======
22+
# ipfixprobe Docker wrapper
23+
24+
This repository contains a Docker container that processes network traffic from a pcap file using `ipfixprobe`. It accepts a pcap file and a processing script, runs it inside the container, and outputs the results in CSV format.
25+
26+
## Requirements
27+
* Docker or Podman
28+
* bash
29+
* which, mktemp
30+
31+
## Usage
32+
This container performs the following tasks:
33+
1. Copies a pcap file and processing script into the container.
34+
2. Runs the ipfixprobe tool to export flows.
35+
3. Logs the results in CSV format.
36+
37+
### Build
38+
39+
The script builds the image automatically, but be sure that Dockerfile is in the same directory.
40+
41+
To build the manually image, navigate to the directory containing the Dockerfile and run:
42+
43+
```bash
44+
docker build -t docker_ipfixprobe .
45+
```
46+
47+
### Run
48+
To run, use
49+
50+
```bash
51+
bash ./ipfixprobe_wrapper.sh <process_script.sh> <input_file.pcap> <output_file.csv>
52+
```
53+
54+
To process a file `../pcaps/mixed.pcap` using a processing script `process_script.sh` and output the results to `output.csv`, use the following wrapper script:
55+
56+
```bash
57+
bash ./ipfixprobe_wrapper.sh ./process_script.sh ../pcaps/mixed.pcap ./output.csv
58+
```
59+
60+
* `process_script.sh` Script for processing the pcap file inside the container.
61+
* `input_file.pcap` Path to the input pcap file
62+
* `output_file.csv` Path to the output CSV file
63+
64+
### Volumes
65+
66+
The container uses `/output` as a volume to share files between your host system temporary dir (with `mktemp`) and the container.

docker/ipfixprobe_wrapper.sh

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
#!/bin/bash
2+
IMAGE_NAME="docker_ipfixprobe"
3+
4+
# Run the ipfixprobe on the input pcap file with defined script, and save the output CSV file to the output path.
5+
PROCESS_SCRIPT_PATH=$1
6+
INPUT_FILE_PATH=$2
7+
OUTPUT_CSV_PATH=$3
8+
9+
if [ -z "$PROCESS_SCRIPT_PATH" ] || [ -z "$INPUT_FILE_PATH" ] || [ -z "$OUTPUT_CSV_PATH" ] ; then
10+
echo "Usage: $0 <process_script> <input_file_path> <output_csv_path>"
11+
exit 1
12+
fi
13+
14+
CONT_BIN="$(which podman 2>/dev/null)"
15+
if [ -z "$CONT_BIN" ]; then
16+
CONT_BIN="$(which docker 2>/dev/null)"
17+
fi
18+
if [ -z "$CONT_BIN" ]; then
19+
echo "Missing podman or docker."
20+
exit 2
21+
fi
22+
23+
# Check if the Docker image exists
24+
if ! $CONT_BIN image inspect "$IMAGE_NAME" >/dev/null 2>&1; then
25+
echo "Docker image '$IMAGE_NAME' not found. Attempting to build it..."
26+
27+
# Determine the script directory
28+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
29+
DOCKERFILE_PATH="$SCRIPT_DIR/Dockerfile"
30+
31+
if [ ! -f "$DOCKERFILE_PATH" ]; then
32+
echo "Dockerfile not found at $DOCKERFILE_PATH"
33+
exit 3
34+
fi
35+
36+
# Build the Docker image
37+
echo "Building Docker image '$IMAGE_NAME'..."
38+
$CONT_BIN build -t "$IMAGE_NAME" -f "$DOCKERFILE_PATH" "$SCRIPT_DIR"
39+
40+
if [ $? -ne 0 ]; then
41+
echo "Failed to build Docker image."
42+
exit 4
43+
fi
44+
fi
45+
46+
47+
INPUT_FILE=$(basename "$INPUT_FILE_PATH")
48+
PROCESS_SCRIPT=$(basename "$PROCESS_SCRIPT_PATH")
49+
TMP_FOLDER="$(mktemp -d)"
50+
51+
cp "$INPUT_FILE_PATH" "$TMP_FOLDER/$INPUT_FILE"
52+
cp "$PROCESS_SCRIPT_PATH" "$TMP_FOLDER/$PROCESS_SCRIPT"
53+
chmod +x "$TMP_FOLDER/$PROCESS_SCRIPT"
54+
55+
"$CONT_BIN" run --privileged --rm -v $TMP_FOLDER:/output "$IMAGE_NAME" "/output/$PROCESS_SCRIPT \"$INPUT_FILE\""
56+
[ -f "$TMP_FOLDER/$INPUT_FILE.csv" ] && cp "$TMP_FOLDER/$INPUT_FILE.csv" "$OUTPUT_CSV_PATH" || echo "No output CSV file found."
57+
58+
# Clean up
59+
rm "$TMP_FOLDER/$INPUT_FILE"
60+
rm "$TMP_FOLDER/$PROCESS_SCRIPT"
61+
[ -f "$TMP_FOLDER/$INPUT_FILE.csv" ] && rm "$TMP_FOLDER/$INPUT_FILE.csv"
62+
rm -rf "$TMP_FOLDER"

docker/process_script.sh

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
#!/bin/bash
2+
3+
FILE=$1 # input file
4+
cd /output # workdir
5+
6+
7+
ipfixprobe -i "pcap;file=$FILE" -p "pstats" -p "nettisa" -o "unirec;i=f:$FILE.trapcap:timeout=WAIT;p=(pstats,nettisa)"
8+
/usr/bin/nemea/logger -t -i "f:$FILE.trapcap" -w "$FILE.csv"
9+
rm $FILE.trapcap

include/ipfixprobe/utils.hpp

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,11 @@ std::string vec2str(const std::vector<T> &vec) {
171171
return ss.str();
172172
}
173173

174+
/**
175+
* @brief Copy uint32 in little endian byte order to destination in host byte order
176+
*/
177+
void memcpy_le32toh(uint32_t* dest, const uint32_t* src);
178+
174179
}
175180

176181
#endif /* IPXP_UTILS_HPP */

0 commit comments

Comments
 (0)