Userspace Networking in gem5

This documentation contains instructions, benchmarks, and files for running unmodified Intel® Data Plane Development Kit (DPDK) in gem5 simulator.

There are three shortcomings in current architectural simulators with respect to evaluating future networked systesm:

1. Existing simulations have outdated networking subsystem that can at best model a few tens of Gbps network throughput.
2. Existing simulators use slow and sometimes inaccurate load generator applications running on different simulated nodes.
3. There are limited standardized networking benchmark suite tailored for running in simulators with standardized metrics and evaluation methodology.

In this tutorial, we provide detailed steps on how to use a popular kernel-bypass framework Data Plane Development Kit DPDK in a state-of-the-art computer architecture simulator gem5.

System Requirements

• gem5 is supported on Intel, ARM, AMD, Apple M1 architectures
• Test Node (DUT) running DPDK (real system ARM Neoverse N1 CPU)
• Drive Node running pktgen (real system Intel® Sapphire Rapids)
• 100Gbps Mellanox Bluefield ConnectX-5 NIC (Test Node) or any DPDK-enabled NIC
• 100Gbps Mellanox Bluefield ConnectX-6 DX NIC (Drive Node) or any DPDK-enabled NIC
• If not using devcontainers
  • Install gem5 dependencies. Find here
  • export GIT_ROOT=/path/to/gem5-dpdk-setup

Software Information

• gem5 v21.1.0.2
• dpdk v20.11.03 (gem5-dpdk)
• dpdk v21.11.0 (ARM Neoverse N1)
• dpdk v23.03 (Intel® Sapphire Rapids CPU)
• pktgen v23.03.0 (Intel® Sapphire Rapids CPU)

Getting Started

If you are new to gem5 or DPDK, you can visit the gem5 bootcamp website or DPDK documentation to learn more. Also, check out the Wiki for more info on how to use the tools in this repository.

Installing and setting up gem5

• clone the gem5 respository

git clone https://github.com/architecture-research-group/gem5-dpdk-setup

git lfs install

• download the arm bootloader boot.arm64, kernel image vmlinux, and disk image rootfs.ext2 (Find here) to resources-dpdk directory.
• build gem5.opt and gem5.fast

cd gem5
scons build/<ISA>/gem5.fast -j $(nproc)
scons build/<ISA>/gem5.opt -j $(nproc)

Installing DPDK

• Modified DPDK v20.11.3 (with new benchmarks) can be found in /path/to/gem5-dpdk-setup/buildroot/package/dpdk/dpdk-source
• DPDK v21.11 needed on ARM Neoverse N1

  cd dpdk-pktgen-build
  

• Modify the DPDK_VER environment variable in dpdk_pktgen.sh script (optional)
• source the DPDK installation script (applies a patch that adds the changes we made to DPDK)

  source dpdk_pktgen.sh
  

• build DPDK (on real system)

  install_dpdk
  

Installing Pktgen

• Follow the steps above to install dpdk first before you install pktgen
• Modify PKT_VER environment variable in dpdk_pktgen.sh script (optional)
• source the pktgen installation script

  source dpdk_pktgen.sh
  

• download, install, and build pktgen

  install_pktgen
  

We also developed a highly parameterizable load generator model that can run in gem5, sending requests at configurable rates to a server node running in the same simulation instance.

Please visit the FAQs in our wiki page if you are having issues.

Check out the wiki page for details on how to run:

• DPDK in gem5
• Memcached in gem5
• DPDK in real system
• Memcached in real system

Authors

• Johnson Umeike
• Siddharth Agarwal
• Nikita Lazarev

Citation

Please cite our paper if you are using any part of the code for your project

@INPROCEEDINGS{10590060,
  author={Umeike, Johnson and Agarwal, Siddharth and Lazarev, Nikita and Alian, Mohammad},
  booktitle={2024 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)}, 
  title={Userspace Networking in gem5}, 
  year={2024},
  volume={},
  number={},
  pages={179-191},
  keywords={Sensitivity;Linux;Full stack;Benchmark testing;Hardware;Software;Generators},
  doi={10.1109/ISPASS61541.2024.00026}}