Skip to content

Commit 05b769a

Browse files
randomizedcoderrandomizedcoder
committed
working
1 parent d6dd317 commit 05b769a

File tree

9 files changed

+1630
-18
lines changed

9 files changed

+1630
-18
lines changed

‎desktop/l2/CPU_and_IRQ_optimization.md‎

Lines changed: 427 additions & 0 deletions
Large diffs are not rendered by default.

‎desktop/l2/README.md‎

Lines changed: 318 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,318 @@
1+
# L2 WiFi Access Point Configuration
2+
3+
This directory contains the NixOS configuration for a high-performance WiFi access point with 4x WiFi NICs, designed to handle multiple concurrent clients with optimized network performance.
4+
5+
## Overview
6+
7+
The L2 system is configured as a dedicated WiFi access point with:
8+
- **4x WiFi NICs** for high-capacity wireless networking
9+
- **Custom hostapd 2.10** with LAR (License Assisted Radar) patch
10+
- **Advanced network services** (DHCP, DNS, IPv6 RA)
11+
- **nftables firewall** with connection tracking and NAT
12+
- **Network interface optimizations** for maximum performance
13+
- **CPU and IRQ optimizations** for dedicated network processing
14+
15+
## WiFi Configuration
16+
17+
### Hostapd 2.10 with LAR Patch
18+
19+
The system uses a custom hostapd 2.10 build with the LAR (License Assisted Radar) patch applied. This is configured in `flake.nix`:
20+
21+
```nix
22+
overlays = {
23+
default = final: prev: {
24+
hostapd = prev.hostapd.overrideDerivation (old: {
25+
version = "2.10";
26+
src = final.fetchurl {
27+
url = "https://w1.fi/releases/hostapd-2.10.tar.gz";
28+
sha256 = "0pcik0a6yin9nib02frjhaglmg44hwik086iwg1751b7kdwpqvi0";
29+
};
30+
# Replace all patches with just the LAR patch
31+
patches = [
32+
(final.fetchpatch {
33+
url = "https://tildearrow.org/storage/hostapd-2.10-lar.patch";
34+
sha256 = "USiHBZH5QcUJfZSxGoFwUefq3ARc4S/KliwUm8SqvoI=";
35+
})
36+
];
37+
});
38+
};
39+
};
40+
```
41+
42+
**Why hostapd 2.10?**
43+
- Better support for modern WiFi features
44+
- Improved performance and stability
45+
- LAR patch enables License Assisted Radar functionality
46+
- **Note**: Default nixpkgs hostapd version is 2.11, but the LAR patch can only be applied to 2.10
47+
48+
**LAR Patch Benefits:**
49+
- Enables radar detection and avoidance
50+
- Required for certain regulatory domains
51+
- Improves coexistence with radar systems
52+
- **Reference**: [Making hostapd LAR-friendly on Intel 5GHz wireless cards](https://tildearrow.org/?p=post&month=7&year=2022&item=lar)
53+
54+
The LAR patch addresses issues with Intel wireless cards that use Location-Aware Regulatory (LAR) to automatically detect the country/regulatory domain. The patch modifies hostapd to scan for nearby networks before setting up the access point, which helps the card properly detect the regulatory domain and enable 5GHz channels.
55+
56+
### WiFi Interface Configuration
57+
58+
The system manages 4 WiFi interfaces:
59+
- `wlp35s0` - Channel 36 (non-DFS)
60+
- `wlp65s0` - Channel 40 (non-DFS)
61+
- `wlp66s0` - Channel 44 (non-DFS)
62+
- `wlp97s0` - Channel 48 (non-DFS)
63+
64+
All interfaces operate in 5GHz band with WPA3-SAE authentication.
65+
66+
## CPU and IRQ Optimization
67+
68+
### System Architecture
69+
70+
The L2 system features an **AMD Ryzen Threadripper PRO 3945WX** with:
71+
- **12 physical cores** (24 logical threads with SMT)
72+
- **4 NUMA nodes** with **64 MiB L3 cache**
73+
- **6 MiB L2 cache** (12 instances)
74+
- **384 KiB L1 cache** per core
75+
76+
### Core Dedication Strategy
77+
78+
#### Network Processing Cores (0-7)
79+
- **Dedicated cores** for network interrupts and processing
80+
- **Isolated from scheduler** (`isolcpus=0-7`)
81+
- **No tick processing** (`nohz_full=0-7`)
82+
- **Disabled RCU callbacks** (`rcu_nocbs=0-7`)
83+
- **Performance governor** with maximum frequency
84+
- **Benefits**:
85+
- Dedicated L1/L2 cache for network processing
86+
- No competition with userland workloads
87+
- Better cache locality for network data structures
88+
- Reduced interrupt latency
89+
90+
#### Userland Processing Cores (8-23)
91+
- **Remaining cores** for system services and userland
92+
- **Normal scheduling** and power management
93+
- **Benefits**:
94+
- Isolated from network interrupt processing
95+
- Dedicated resources for DHCP, DNS, firewall processing
96+
- Better performance for non-network workloads
97+
98+
### IRQ Affinity Configuration
99+
100+
#### Ethernet Interface (enp1s0)
101+
- **8 MSI-X vectors** distributed across cores 0-7
102+
- **Atlantic driver** with optimized interrupt handling
103+
104+
#### WiFi Interfaces
105+
- **wlp35s0**: 16 MSI-X vectors → cores 0-3
106+
- **wlp65s0**: 16 MSI-X vectors → cores 4-7
107+
- **wlp66s0**: 16 MSI-X vectors → cores 0-3 (alternating)
108+
- **wlp97s0**: 16 MSI-X vectors → cores 4-7 (alternating)
109+
110+
### Service CPU Affinity
111+
112+
#### Network Processing Services (Cores 0-7)
113+
- **hostapd**: High priority (-10), real-time I/O, network-processing slice
114+
- **nftables**: High priority (-5), network-processing slice
115+
- **network-optimization**: High priority (-5), network-processing slice
116+
117+
#### Network Services (Cores 8-15)
118+
- **kea-dhcp4-server**: High priority (-5), network-services slice
119+
- **pdns-recursor**: High priority (-5), network-services slice
120+
- **radvd**: High priority (-5), network-services slice
121+
122+
#### Userland Services (Cores 16-23)
123+
- **Monitoring services**: Normal priority, userland-processing slice
124+
- **System services**: Normal priority, userland-processing slice
125+
- **User processes**: Normal priority, userland-processing slice
126+
127+
## Network Services (`hostapd-multi.nix`)
128+
129+
### DHCP Server (Kea)
130+
- **Purpose**: Provides IPv4 addresses to WiFi clients
131+
- **Subnet**: 192.168.1.0/24
132+
- **Range**: 192.168.1.100 - 192.168.1.200
133+
- **Gateway**: 192.168.1.1
134+
- **DNS**: 192.168.1.1
135+
136+
### DNS Resolver (PowerDNS Recursor)
137+
- **Purpose**: Local DNS resolution for WiFi clients
138+
- **Listen addresses**: 127.0.0.1, ::1, 192.168.1.1, fd00::1
139+
- **Features**: RFC1918 support, recursive resolution
140+
- **Fallback**: Cloudflare DNS (1.1.1.1, 2606:4700:4700::1111)
141+
142+
### IPv6 Router Advertisement (radvd)
143+
- **Purpose**: IPv6 SLAAC for WiFi clients
144+
- **Prefix**: fd00::/64
145+
- **Features**: Autonomous address configuration
146+
- **DNS**: fd00::1
147+
148+
### Network Bridge (br0)
149+
- **Purpose**: Bridges all WiFi interfaces
150+
- **IPv4**: 192.168.1.1/24
151+
- **IPv6**: fd00::1/64
152+
- **QoS**: CAKE (Common Applications Kept Enhanced) for traffic shaping
153+
154+
## Firewall and NAT (`firewall.nix`)
155+
156+
### nftables Configuration
157+
The system uses nftables with connection tracking for maximum security and performance:
158+
159+
#### Filter Table (inet)
160+
- **Input Chain**: Handles incoming traffic to the router
161+
- SSH (port 22)
162+
- DNS (port 53)
163+
- DHCP (ports 67 for DHCPv4, 547 for DHCPv6)
164+
- ICMP (ping)
165+
- IPv6 RA
166+
- **Forward Chain**: Handles traffic between networks
167+
- Allow internal to external (br0 → enp1s0)
168+
- Allow return traffic for established connections
169+
- **Output Chain**: Allow all outgoing traffic
170+
171+
#### NAT Tables
172+
- **IPv4 NAT**: Masquerades traffic from br0 to enp1s0
173+
- **IPv6 NAT**: Masquerades IPv6 traffic from br0 to enp1s0
174+
175+
### Connection Tracking
176+
- **Purpose**: Stateful packet filtering
177+
- **Benefits**:
178+
- Only legitimate return traffic is allowed
179+
- Better security than stateless filtering
180+
- Improved performance for established connections
181+
182+
## Network Optimizations (`network-optimization.nix`)
183+
184+
### Hardware Optimizations
185+
Applied via ethtool during boot:
186+
187+
#### Ring Buffers
188+
- **RX/TX**: Increased to maximum (8184)
189+
- **Benefit**: Higher throughput, better burst handling
190+
191+
#### Feature Enables
192+
- **LRO (Large Receive Offload)**: Combines packets for CPU efficiency
193+
- **IPv4 Checksum Offload**: Hardware handles checksum calculation
194+
- **TCP ECN Segmentation**: Better ECN packet handling
195+
- **GRO List**: Generic Receive Offload with list support
196+
197+
#### Interrupt Coalescing
198+
- **RX**: 512μs, 32 frames (was 256μs, 0 frames)
199+
- **TX**: 1024μs, 32 frames (was 1022μs, 0 frames)
200+
- **Benefits**: Fewer interrupts, better batch processing
201+
202+
### Kernel Optimizations (`sysctl.nix`)
203+
- **TCP buffers**: Optimized for high throughput
204+
- **Connection tracking**: 262K entries for multiple clients
205+
- **Network backlog**: Increased for burst traffic handling
206+
- **Congestion control**: BBR for better performance
207+
208+
### Verification
209+
Network optimization results are logged to `/tmp/network-optimization.log` and include:
210+
- Ring buffer settings
211+
- Feature status
212+
- Interrupt coalescing configuration
213+
- Driver information
214+
215+
## Performance Monitoring (`monitoring.nix`)
216+
217+
### Automated Monitoring
218+
- **IRQ distribution**: Tracks interrupt distribution across cores
219+
- **CPU utilization**: Monitors per-core usage patterns
220+
- **Network statistics**: Tracks interface performance
221+
- **Cache performance**: Monitors cache misses for network processes
222+
- **System load**: Tracks overall system performance
223+
224+
### Performance Testing
225+
- **Throughput testing**: Automated iperf3 testing
226+
- **Latency testing**: Ping latency measurements
227+
- **IRQ distribution testing**: Validates interrupt affinity
228+
- **CPU utilization testing**: Monitors during network activity
229+
230+
### Logging and Analysis
231+
- **Log directory**: `/var/log/network-performance/`
232+
- **Real-time monitoring**: Continuous performance tracking
233+
- **Historical data**: sysstat integration for trend analysis
234+
- **Log rotation**: Automated log management
235+
236+
## System Architecture
237+
238+
```
239+
Internet (enp1s0)
240+
↓
241+
[NAT/Firewall] ← nftables with connection tracking (cores 0-7, network-processing slice)
242+
↓
243+
[Bridge (br0)] ← 192.168.1.1/24, fd00::1/64
244+
↓
245+
[WiFi Clients] ← 4x WiFi interfaces with hostapd 2.10 (cores 0-7, network-processing slice)
246+
↓
247+
[Network Services] ← DHCP, DNS, RA (cores 8-15, network-services slice)
248+
↓
249+
[Userland Services] ← Monitoring, system services (cores 16-23, userland-processing slice)
250+
```
251+
252+
## Services Overview
253+
254+
| Service | Purpose | CPU Cores | Priority | Slice |
255+
|---------|---------|-----------|----------|-------|
256+
| hostapd | WiFi access point | 0-7 | -10 (RT) | network-processing |
257+
| nftables | Firewall/NAT | 0-7 | -5 | network-processing |
258+
| Kea | DHCP server | 8-15 | -5 | network-services |
259+
| PowerDNS | DNS resolver | 8-15 | -5 | network-services |
260+
| radvd | IPv6 RA | 8-15 | -5 | network-services |
261+
| CAKE | QoS | 0-7 | -5 | network-processing |
262+
| Monitoring | Performance tracking | 16-23 | 0 | userland-processing |
263+
264+
## Performance Features
265+
266+
- **Multi-interface WiFi**: 4x concurrent access points
267+
- **Hardware offloading**: Checksums, segmentation, GRO
268+
- **Connection tracking**: Stateful firewall with 262K entries
269+
- **Optimized buffers**: Maximum ring buffers and TCP windows
270+
- **Interrupt coalescing**: Reduced CPU overhead
271+
- **BBR congestion control**: Better throughput and latency
272+
- **CPU isolation**: Dedicated network processing cores
273+
- **IRQ affinity**: Optimized interrupt distribution
274+
- **Cache optimization**: Dedicated L1/L2 cache for network processing
275+
276+
## Expected Performance Improvements
277+
278+
### 1. **Reduced Interrupt Latency**
279+
- Dedicated cores eliminate competition for CPU resources
280+
- Better cache locality reduces memory access latency
281+
- SMT isolation prevents cache pollution
282+
283+
### 2. **Improved Throughput**
284+
- Parallel processing across 8 dedicated network cores
285+
- Better interrupt distribution reduces bottlenecks
286+
- Optimized cache utilization for network data structures
287+
288+
### 3. **Lower CPU Overhead**
289+
- Reduced context switching on network cores
290+
- Better interrupt coalescing effectiveness
291+
- Optimized memory allocation patterns
292+
293+
### 4. **Enhanced Scalability**
294+
- Better support for multiple concurrent WiFi clients
295+
- Improved handling of burst traffic
296+
- More predictable performance under load
297+
298+
## Monitoring
299+
300+
- **Network optimization log**: `/tmp/network-optimization.log`
301+
- **Performance monitoring**: `/var/log/network-performance/`
302+
- **nftables rules**: `sudo nft list ruleset`
303+
- **Service status**: `systemctl status hostapd kea-dhcp4-server pdns-recursor radvd nftables`
304+
- **IRQ distribution**: `cat /proc/interrupts | grep -E "(enp1s0|iwlwifi)"`
305+
- **CPU utilization**: `mpstat -P ALL 1`
306+
307+
## Files Overview
308+
309+
- `flake.nix` - Hostapd 2.10 overlay and flake configuration
310+
- `hostapd-multi.nix` - WiFi, DHCP, DNS, and IPv6 services
311+
- `firewall.nix` - nftables firewall and NAT configuration
312+
- `network-optimization.nix` - Hardware and kernel optimizations
313+
- `irq-affinity.nix` - IRQ affinity and CPU dedication configuration
314+
- `kernel-params.nix` - Kernel boot parameters and runtime optimizations
315+
- `monitoring.nix` - Performance monitoring and testing services
316+
- `sysctl.nix` - Kernel network parameters
317+
- `configuration.nix` - Main system configuration
318+
- `CPU_and_IRQ_optimization.md` - Detailed optimization documentation

‎desktop/l2/configuration.nix‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,10 @@
3939
#./hostapd.nix
4040
./hostapd-multi.nix
4141
./network-optimization.nix
42+
# CPU and IRQ optimization modules
43+
./irq-affinity.nix
44+
./kernel-params.nix
45+
./monitoring.nix
4246
];
4347

4448
boot = {

‎desktop/l2/firewall.nix‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,41 @@
1+
#
2+
# l2/firewall.nix
3+
#
4+
# Firewall configuration for WiFi access point
5+
#
6+
# # List all rules (most common)
7+
# sudo nft list ruleset
8+
9+
# # List specific table
10+
# sudo nft list table inet filter
11+
12+
# # List specific chain
13+
# sudo nft list chain inet filter input
14+
15+
# # Show rules with handles (useful for deleting specific rules)
16+
# sudo nft list ruleset -a
17+
18+
# # Monitor nftables events in real-time
19+
# sudo nft monitor
20+
21+
# # Monitor specific events (new rules, deleted rules, etc.)
22+
# sudo nft monitor new rules
23+
# sudo nft monitor destroy rules
24+
25+
# # Show packet counters
26+
# sudo nft list ruleset -n
27+
28+
# # Show rules with statistics
29+
# sudo nft list ruleset -s
30+
# #
31+
# # See all filter rules (input, forward, output chains)
32+
# sudo nft list table inet filter
33+
34+
# # See NAT rules
35+
# sudo nft list table ip nat
36+
# sudo nft list table ip6 nat
37+
#
38+
139
{ config, pkgs, ... }:
240

341
{

0 commit comments

Comments
 (0)