Skip to content

Configuration.md

Latest commit

 

History

History
190 lines (151 loc) · 10.4 KB

Configuration.md

File metadata and controls

190 lines (151 loc) · 10.4 KB
Raw

Experiment Configuration (YAML)

Every experiment is driven by a single YAML config under config/, grouped by the RoCEv2 role under test: config/opensource-np/ (Notification Point), config/opensource-rp/ (Reaction Point), and config/opensource-cp/ (Congestion Point). The config is passed with --config / -c.

Who reads the config

A config file is consumed by two different programs, and it helps to keep straight which section is read by which:

Section Read by Notes
servers.peer_id run.py only Resolved to data-plane IPs (peer_ip) from HOST_CONFIGS; ignored by the C app.
servers.peer_ip, cp, np, rp C app (arg_parser.c) Defines the peer list and roles. run.py also reads peer_ip to pick which hosts to launch on.
experiment C app All runtime knobs (traffic shape, modes, stats).
cnp_state_machine C app Receiver/NP-side CNP send timeline.
qp_state_machines C app Sender/RP-side per-QP RDMA send timeline.
pre_launch, post_launch run.py only Shell commands run on a host before/after the experiment; the C app never sees them.

The C-side parser lives in parse_yaml_config() and only recognizes the four sections servers, experiment, cnp_state_machine, and qp_state_machines; any other top-level key is reported as an unknown section warning and skipped (which is why pre_launch / post_launch are harmless to the binary).

How roles are resolved

Each host launches the same binary with the same config but a different --local-ip / -l. On startup the app looks up its own --local-ip in the peer_ip array to find its peer index, then compares that index against np / cp / rp to decide which of the NP / CP / RP roles it plays (a host can hold several). See anytest_parse_args() for the matching logic.

servers

Defines the set of peers and assigns DCQCN roles to them.

servers:
  peer_id: ["33.255.69.193:bond1", "33.255.69.194:bond1"]
  peer_ip: ["26.248.37.98", "26.248.37.99"]
  cp: 0
  np: 0
  rp: [1]
Key Type Description
peer_id list of "<control_ip>:<bond>" Logical host identifiers matched against HOST_CONFIGS in run.py. When present, run.py auto-fills peer_ip from the matching data_ip before distributing the config (see augment_config_with_peer_ips()). The C app never reads this key.
peer_ip list of IPv4 strings The data-plane IP of each peer. The position in this list is the peer index used everywhere else (cp / np / rp, custom_sequence.rp_idx, …). The length of this list defines peer_cnt.
cp index or IP string The single peer acting as the Congestion Point (the switch under test).
np index or IP string The single peer acting as the Notification Point (replies with CNP).
rp list of indices or IP strings The peers acting as Reaction Points (the senders that react to CNP). The list length defines rp_cnt.

cp / np / rp entries may be written either as a numeric peer index or as an IP string that exists in peer_ip; both are resolved by resolve_peer_idx_from_scalar().

experiment

The main block of runtime knobs. Unknown keys are silently ignored, so a key only takes effect if it is spelled exactly as the parser expects.

Mode selection

Key Type / Values Default Description
exp_mode SENDER | RECEIVER | SWITCH | ROCE | RECEIVER_CUSTOM_SEQ | RP_EXP_CAPABILITY_TEST RECEIVER Which experiment entry point main.c dispatches to. SENDER tests the RX/sink path (DPDK receives forged RoCE), RECEIVER forges RoCE traffic to drive a NIC's CNP, SWITCH observes ECN marking, ROCE runs real RDMA ops, RECEIVER_CUSTOM_SEQ replays a hand-written packet sequence, RP_EXP_CAPABILITY_TEST is the RoCEv2-receive sanity check. See the exp_mode enum in common.h.

Traffic shape

Key Type Default Description
qp_cnt int 1 Number of QPs to set up.
qp_mod int 1 Stride for which QPs actually generate traffic: only QPs whose index is a multiple of qp_mod send (so 1 = every QP). Used by the round-robin sender in rdma_op.c.
customed_mtu int (IBV MTU enum) 3 RoCE path MTU as the standard IBV enum: 1=256B, 2=512B, 3=1024B, 4=2048B, 5=4096B. Applied as the QP's path_mtu.
opcode WRITE | WRITE_ONLY WRITE RoCE opcode used by the generated flow.
send_interval_ns int (ns) 1 Gap between consecutive sends; 0 sends as fast as possible.
ackreq_interval int (packets) 10 How often the AckReq bit is set in the BTH (every Nth packet).
send_times int 0 Number of send rounds; 0 = send indefinitely (until stop).
auto_stop_seconds int (s) 0 Auto-stop the experiment after N seconds; 0 disables.
padding_pkt_size int (bytes) 0 Size of padding packets inserted into the flow.
padding_pkt_cnt int 0 Number of padding packets per unit.
tail_padding_size int (bytes) 0 Extra padding appended at the tail of the flow.
tx_batch_size int 64 Burst size for the batched-prepared TX path (rte_eth_tx_burst).
tx_mode default | dynpsn | dynpsn_mt default TX path: default (prepared / prepared-batched), dynpsn (single-thread clone + incrementing PSN/ICRC), dynpsn_mt (two-thread prepare+tx via rte_ring). See the tx_mode enum in common.h.
customed_qp_dscp list of int (≤ 8) unset Per-QP DSCP values (one per QP, in QP-index order). Setting this enables custom DSCP; e.g. [34, 34, 06, 06] maps the four QPs onto two traffic classes.

Statistics

Key Type Default Description
detailed_ts_enabled bool false Enable detailed per-packet timestamp / counting.
detailed_lat_enabled bool false Enable RDMA completion-latency statistics.
detailed_enabled_qps list of int unset (= all) Restrict detailed timestamp stats to these QP indices; empty/unset means all QPs (see is_qp_detailed_enabled() in common.h).
cnp_response_min_interval_us int (µs) 0 Minimum spacing the sink enforces between CNP responses; 0 disables rate limiting.

Connection / generator options

Key Type Default Description
dummy_rdma_conn_enabled bool false Skip real connection setup with the peer and fill in fake QP metadata locally. Required by RECEIVER_CUSTOM_SEQ when there is no live peer.
fixed_prepared_pkt_num_enabled bool false Pin the prepared-packet count to a constant derived from MTU=1024, to keep working-set size comparable across MTUs.
gen_bench_enabled bool false Enable the generate-only PPS micro-benchmark (no rte_ring, no NIC TX).
gen_bench_target multi | dynpsn dynpsn Which generator path the bench exercises.
gen_bench_duration_s int (s) 5 Bench measurement duration.
gen_bench_warmup_s int (s) 1 Bench warm-up duration.
gen_bench_wqe_len int 0 WQE length used by the bench (0 = default).

custom_sequence (for RECEIVER_CUSTOM_SEQ)

A hand-written list of packets to emit, one mapping per packet. Used to reproduce specific out-of-order / ECN patterns (see config/opensource-np/cnp-custom-seq-0.yaml).

custom_sequence:
  - {rp_idx: 0, qp_idx: 0, psn: 0x1000, ack_req: 1, ecn: 3, opcode: "WRITE_FIRST"}
  - {rp_idx: 0, qp_idx: 0, psn: 0x1001, ack_req: 1, ecn: 3, opcode: "WRITE_LAST"}
Field Type Description
rp_idx int Index into the RP array (0 … rp_cnt-1).
qp_idx int QP index within that RP (0 … qp_cnt-1).
psn int (decimal or 0x hex) Packet Sequence Number.
ack_req 0 | 1 AckReq flag.
ecn 0…3 ECN bits (e.g. 3 = 0b11).
opcode string BTH opcode: WRITE_FIRST / WRITE_MIDDLE / WRITE_LAST / WRITE_LAST_WITH_IMM / WRITE_ONLY / WRITE_ONLY_WITH_IMM.

cnp_state_machine and qp_state_machines

These two sections describe time-sequenced send timelines built on the shared state-machine engine — cnp_state_machine is a single global timeline that drives CNP sends on the receiver/NP side, and qp_state_machines is a set of per-QP timelines that drive RDMA writes on the sender/RP side.

cnp_state_machine:
  enabled: true
  states:
    - {duration_us: 2500000, send_cnp: "NONE", cnp_interval_us: 0}
    - {duration_us: 1,       send_cnp: "ONCE", cnp_interval_us: 0}

qp_state_machines:
  enabled: true
  qps:
    - qp_indices: [0]
      max_depth: 0
      states:
        - {duration_us: 1000000, send_rdma: NONE}
        - {duration_us: 3000000, send_rdma: MANY, rdma_interval_us: 5, rdma_len: 16777216}

Each state carries a duration_us, an execute mode (NONE / ONCE / MANY), and an interval used in MANY mode (cnp_interval_us / rdma_interval_us); QP states additionally take an optional rdma_len, and each QP group an optional max_depth. The full semantics of the engine, the meaning of every per-state field, and how the timelines advance are documented separately in State_machine.md — refer to it for these two sections.

pre_launch / post_launch

These sections are executed by run.py (via execute_launch_commands()), not by the C binary. They run a list of shell commands on a target host before the experiment starts (pre_launch) and after it finishes (post_launch) — typically to set up NIC QoS / DCQCN parameters and then restore them.

pre_launch:
  enable: true
  host: "33.255.69.194"
  pre_launch_cmd:
    - "sudo mlnx_qos -i eth0 --dscp2prio=set,06,5"
    - "sudo ccmanage switch dcqcn"
Key Type Description
enable bool Whether to run this block.
host IP string The host on which the commands run (matched against the cluster host list).
pre_launch_cmd / post_launch_cmd list of strings Shell commands run in order over SSH.