Skip to content

Commit ae7f97e

Browse files
Merge pull request #1403 from pdsouza01/main
Scaling Snort3 - How To Leverage Multithreading For Better Performance
2 parents c1505fb + 4113285 commit ae7f97e

File tree

5 files changed

+519
-0
lines changed

5 files changed

+519
-0
lines changed
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
---
2+
title: Scaling Snort3 - How To Leverage Multithreading For Better Performance
3+
4+
minutes_to_complete: 45
5+
6+
who_is_this_for: This blog is for engineers familiar with Snort who want to enhance its performance by leveraging the benefits of multithreading.
7+
8+
learning_objectives:
9+
- Install Snort with all of its dependencies
10+
- Configure Snort Lua files to enable multithreading
11+
- Use multithreading to process capture files and measure performance
12+
13+
prerequisites:
14+
- An Arm-based instance from a cloud provider or an Arm server running Ubuntu 20.04 or 22.04.
15+
- A basic understanding of Snort's operation and configuration.
16+
17+
18+
author_primary: Preema Merlin Dsouza
19+
20+
### Tags
21+
skilllevels: Introductory
22+
subjects: Libraries
23+
armips:
24+
- Neoverse
25+
tools_software_languages:
26+
- AWS EC2
27+
- Snort3
28+
- Bash
29+
- GCC
30+
operatingsystems:
31+
- Linux
32+
33+
### FIXED, DO NOT MODIFY
34+
# ================================================================================
35+
weight: 1 # _index.md always has weight of 1 to order correctly
36+
layout: "learningpathall" # All files under learning paths have this same wrapper
37+
learning_path_main_page: "yes" # This should be surfaced when looking for related content. Only set for _index.md of learning path content.
38+
---
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
---
2+
next_step_guidance: To continue learning about enabling hyperscan on arm,please refer to the learning path provided below.
3+
4+
recommended_path: /learning-paths/servers-and-cloud-computing/vectorscan/
5+
6+
further_reading:
7+
- resource:
8+
title: Snort3 Documentation
9+
link: https://docs.snort.org/start/
10+
type: documentation
11+
- resource:
12+
title: Performance Optimization for NGFW Whitepaper
13+
link: https://files.techmahindra.com/static/img/pdf/next-generation-firewall.pdf
14+
type: blog
15+
16+
# ================================================================================
17+
# FIXED, DO NOT MODIFY
18+
# ================================================================================
19+
weight: 21 # set to always be larger than the content in this path, and one more than 'review'
20+
title: "Next Steps" # Always the same
21+
layout: "learningpathall" # All files under learning paths have this same wrapper
22+
---
Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
---
2+
review:
3+
- questions:
4+
question: >
5+
Which of the following is a key benefit of Snort3's multithreading support?
6+
answers:
7+
- It allows Snort to detect encrypted traffic.
8+
- It improves packet processing performance
9+
- It enables Snort to be run on legacy hardware
10+
- It support multiple rule sets at the same time.
11+
correct_answer: 2
12+
explanation: >
13+
It improves packet processing performance by parallelizing tasks.
14+
15+
- questions:
16+
question: >
17+
Which parameter is used to enable multithreading in Snort3?
18+
answers:
19+
- --max-packet-threads
20+
- --enable-threads
21+
- --enable-multithreading
22+
- --packet-loop
23+
correct_answer: 1
24+
explanation: >
25+
--max-packet-threads parameter is used to enable and configure multithreading.
26+
27+
- questions:
28+
question: >
29+
In Snort 3, which DAQ (Data Acquisition) module is used to read capture files for packet processing?
30+
answers:
31+
- afpacket
32+
- vpp
33+
- dump
34+
- pcap
35+
correct_answer: 3
36+
explanation: >
37+
The dump module in Snort3 is used to read capture files (such as .pcap or .pcapng files) for offline packet analysis.
38+
39+
40+
41+
# ================================================================================
42+
# FIXED, DO NOT MODIFY
43+
# ================================================================================
44+
title: "Review" # Always the same title
45+
weight: 20 # Set to always be larger than the content in this path
46+
layout: "learningpathall" # All files under learning paths have this same wrapper
47+
---
Lines changed: 212 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,212 @@
1+
---
2+
title: Install Snort3 along with all its required dependencies.
3+
weight: 2
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
Multithreading in Snort 3 refers to the ability to associate multiple threads with a single Snort instance enabling the concurrent processing of multiple packet files. This optimization frees up additional memory for further packet processing.
10+
11+
In order to enable multithreading in Snort3, specify the quantity of threads designated for processing network traffic using either the '--max-packet-threads' or '-z' option.
12+
13+
{{%notice Note%}}
14+
The instruction provided have been tested on AWS EC2 Graviton4 metal instance (Neoverse V2)
15+
{{%/notice%}}
16+
17+
## Compile and build Snort3
18+
Run the script to download and install Snort3 and its dependent libraries.
19+
Skip this step if Snort3 is already installed.
20+
21+
<!-- add github link for the below file [build_snort3.sh]() -->
22+
``` bash
23+
#!/usr/bin/env bash
24+
25+
# Copyright (c) 2022-2024, Arm Limited.
26+
#
27+
# SPDX-License-Identifier: Apache-2.0
28+
29+
30+
# Define a list of dependency package URLs
31+
declare -a PACKAGE_URLS=(
32+
"https://github.com/snort3/snort3/archive/refs/tags/3.3.5.0.tar.gz"
33+
"https://sourceforge.net/projects/pcre/files/pcre/8.45/pcre-8.45.tar.gz"
34+
"https://github.com/VectorCamp/vectorscan/archive/refs/tags/vectorscan/5.4.11.tar.gz"
35+
"https://github.com/snort3/libdaq/archive/refs/tags/v3.0.16.tar.gz"
36+
"https://boostorg.jfrog.io/artifactory/main/release/1.86.0/source/boost_1_86_0.tar.gz"
37+
"https://github.com/rurban/safeclib/releases/download/v3.8.1/safeclib-3.8.1.tar.gz"
38+
"https://github.com/gperftools/gperftools/releases/download/gperftools-2.13/gperftools-2.13.tar.gz"
39+
)
40+
41+
downlaodPackages()
42+
{
43+
for url in "${PACKAGE_URLS[@]}"; do
44+
# Extract the file name from the URL
45+
fname=$(basename "$url")
46+
fpath="${ROOT_DIR}/${fname}"
47+
# Check if the file already exists
48+
if [[ -f "$fpath" ]]; then
49+
echo "File $fname already exists. Skipping download."
50+
else
51+
# Download the file using wget
52+
53+
echo "File $fname not found. Downloading..."
54+
55+
wget -O "$fpath" "$url"
56+
if [[ $? -eq 0 ]]; then
57+
echo "$fname download complete"
58+
else
59+
echo "ERROR:$fname download Fail."
60+
fi
61+
fi
62+
done
63+
}
64+
65+
installPackages()
66+
{
67+
echo "@@@@@@@@@@@@@@@@@@ Installing packages ... @@@@@@@@@@@@@@@@@@@@"
68+
if [[ -r /etc/os-release ]]; then
69+
OS_NAME=$(grep -w "NAME" /etc/os-release | cut -d= -f2 | tr -d '"')
70+
OS_VERSION_ID=$(grep -w "VERSION_ID" /etc/os-release | cut -d= -f2 | tr -d '"')
71+
if [[ "${OS_NAME}" == "Ubuntu" ]]; then
72+
echo "OS: ${OS_NAME} ${OS_VERSION_ID}"
73+
else
74+
echo "Error: This script is only for ubuntu"
75+
exit 1
76+
fi
77+
if [[ "${OS_VERSION_ID}" != "22.04" ] || [ "${OS_VERSION_ID}" != "20.04" ]];then
78+
echo "Warning: OS: ${OS_NAME} ${OS_VERSION_ID}"
79+
echo "Warning: Ubuntu 20.04 or 22.04 is recommended"
80+
fi
81+
else
82+
echo "Error: OS information detection failed"
83+
exit 1
84+
fi
85+
86+
apt-get update
87+
apt-get install -y $LIST_OF_APPS
88+
89+
# required to get optimised result from Snort3
90+
downlaodPackages
91+
mkdir -p ${ROOT_DIR}/snort3
92+
tar -xzf 3.3.5.0.tar.gz --directory ${ROOT_DIR}/snort3 --strip-components=1
93+
echo "@@@@@@@@@@@@@@@@@@ Installing Snort3 Dependencies ... @@@@@@@@@@@@@@@@@@@@"
94+
mkdir -p ${SNORT_DIR}
95+
mkdir -p $SNORT_DIR/pcre
96+
tar -xvf pcre-8.45.tar.gz --directory $SNORT_DIR/pcre --strip-components=1
97+
#vector scan
98+
mkdir -p $SNORT_DIR/vectorscan
99+
tar -xzvf 5.4.11.tar.gz --directory $SNORT_DIR/vectorscan --strip-components=1
100+
101+
#libdaq
102+
mkdir -p $SNORT_DIR/libdaq
103+
tar -xvzf v3.0.16.tar.gz --directory $SNORT_DIR/libdaq --strip-components=1
104+
105+
#required to get optimized result from vectorscan
106+
mkdir -p $SNORT_DIR/boost
107+
tar -xvf boost_1_86_0.tar.gz -C $SNORT_DIR/boost --strip-components=1
108+
109+
#safeclib
110+
mkdir -p $SNORT_DIR/safeclib
111+
tar -xzvf safeclib-3.8.1.tar.gz --directory $SNORT_DIR/safeclib --strip-components=1
112+
113+
#gperftools
114+
mkdir -p $SNORT_DIR/gperftools
115+
tar -xzvf gperftools-2.13.tar.gz --directory $SNORT_DIR/gperftools --strip-components=1
116+
117+
echo "@@@@@@@@@@@@@@@@@@ Packages installed @@@@@@@@@@@@@@@@@@@@"
118+
}
119+
120+
buildInstall()
121+
{
122+
echo "@@@@@@@@@@@@@@@@@@ Build & Installation ... Start @@@@@@@@@@@@@@@@@@@@"
123+
cd $SNORT_DIR/libdaq
124+
mkdir -p ${SNORT_DIR}/libdaq/install
125+
./bootstrap
126+
./configure --prefix=${SNORT_DIR}/libdaq/install
127+
make -j${NUM_JOBS}
128+
make install
129+
130+
cd ${SNORT_DIR}/safeclib
131+
./configure
132+
make -j${NUM_JOBS}
133+
make -j${NUM_JOBS} install
134+
135+
cd $SNORT_DIR/gperftools
136+
./configure --with-tcmalloc-pagesize=64
137+
make -j${NUM_JOBS}
138+
make -j${NUM_JOBS}
139+
140+
cd $SNORT_DIR/pcre
141+
./configure
142+
make -j${NUM_JOBS}
143+
make -j${NUM_JOBS}
144+
145+
cd ${SNORT_DIR}/vectorscan
146+
cmake -DBOOST_ROOT=$(SNORT_DIR)/boost -DCMAKE_BUILD_TYPE=Release .
147+
make -j${NUM_JOBS}
148+
make -j${NUM_JOBS}
149+
150+
cd ${ROOT_DIR}/snort3
151+
./configure_cmake.sh --build-type=Release --with-daq-includes=${SNORT_DIR}/libdaq/install/include/ --with-daq-libraries=${SNORT_DIR}/libdaq/install/lib/ --enable-unit-tests --enable-tcmalloc
152+
cd ${ROOT_DIR}/snort3/build
153+
make -j$NUM_JOBS
154+
make -j$NUM_JOBS install
155+
echo "@@@@@@@@@@@@@@@@@@ Build & Installation ... Done @@@@@@@@@@@@@@@@@@@@"
156+
}
157+
158+
#------ Execution Start ----------#
159+
# provide nproc count to the scripts , it will be used as -j for make
160+
if [[ $# -ne 2 ]]; then
161+
echo "Usage: $0 <current_working_directory> <nprc>"
162+
exit 1
163+
fi
164+
165+
ROOT_DIR=$(pwd)/"$1"
166+
NUM_JOBS="$2"
167+
SNORT_DIR=${ROOT_DIR}/snort3/dependencies
168+
set -e
169+
170+
LIST_OF_APPS="sudo net-tools build-essential manpages-dev libnuma-dev python3
171+
python3-venv cmake meson pkg-config python3-pyelftools lshw
172+
util-linux iperf3 nginx libboost-all-dev ragel libsqlite3-dev
173+
libpcap-dev libdumbnet-dev libluajit-5.1-dev zlib1g-dev
174+
libhwloc-dev liblzma-dev libssl-dev libgoogle-perftools-dev
175+
libpcre++-dev flex openssl libunwind-dev autotools-dev
176+
libhugetlbfs-bin autoconf libmnl-dev bats wget unzip iproute2
177+
git pkg-config cpputest libtool bison libcmocka-dev
178+
libnetfilter-queue-dev ethtool"
179+
180+
# nprc should be a positive integer)
181+
if ! [[ "$NUM_JOBS" =~ ^[0-9]+$ ]] || [[ "$NUM_JOBS" -le 0 ]]; then
182+
echo "Error: nprc should be a positive integer."
183+
exit 1
184+
fi
185+
186+
mkdir -p ${ROOT_DIR}
187+
cd ${ROOT_DIR}
188+
installPackages
189+
buildInstall
190+
```
191+
192+
To check if the installation is complete, run the command below.
193+
```bash{ output_lines = "2-20" }
194+
snort -V
195+
,,_ -*> Snort++ <*-
196+
o" )~ Version 3.3.5.0
197+
'''' By Martin Roesch & The Snort Team
198+
http://snort.org/contact#team
199+
Copyright (C) 2014-2024 Cisco and/or its affiliates. All rights reserved.
200+
Copyright (C) 1998-2013 Sourcefire, Inc., et al.
201+
Using DAQ version 3.0.16
202+
Using Hyperscan version 5.4.11 2024-09-12
203+
Using libpcap version 1.10.1 (with TPACKET_V3)
204+
Using LuaJIT version 2.1.0-beta3
205+
Using LZMA version 5.2.5
206+
Using OpenSSL 3.0.2 15 Mar 2022
207+
Using PCRE version 8.45 2021-06-15
208+
Using ZLIB version 1.2.11
209+
210+
```
211+
212+

0 commit comments

Comments
 (0)