You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This document explains the profiling infrastructure set up for our indexer-service and tap-agent services. The profiling setup enables developers to diagnose performance issues, memory leaks, and analyze runtime behavior in both development and production environments.
1
+
# Load Testing and Profiling Guide for Indexer Service
4
2
5
3
## Overview
6
4
7
-
Our project includes an integrated profiling system for the indexer services. The system supports multiple profiling methods through:
5
+
This guide explains how to perform load testing and profiling on the indexer service and tap-agent components. The system includes built-in load testing tools and various profiling methods to help identify performance bottlenecks and optimize the services.
6
+
7
+
Our project includes an integrated profiling system that supports multiple profiling methods through:
8
8
9
9
1. A custom `profiler` library (included in the workspace)
10
10
2. Docker-based profiling environments
11
11
3. Various third-party profiling tools
12
12
13
-
## Available Profiling Methods
13
+
## Important Note About Load Limits
14
14
15
-
### Built-in Profiler (pprof-based Flamegraphs)
15
+
⚠️ **Important**: The current indexer-service implementation has a built-in protection mechanism against high load from a single sender. When receiving too many receipts from the same server (approximately 1000 receipts), the sender will be marked as denied. This is a security feature to prevent abuse.
16
16
17
-
A Rust library that uses [pprof](https://crates.io/crates/pprof) to continuously profile the application and generate flamegraphs at specified intervals.
18
-
This solution was particularly suitable because tools like `perf`, while powerful, often pose configuration challenges or require specific capabilities (like CAP_SYS_ADMIN) that complicate their deployment within standard Docker containers.
17
+
## Load Testing
18
+
19
+
### Prerequisites
19
20
20
-
-**Configuration**: Set in code with the `setup_profiling` function
21
-
-**Activation**: Enabled via the `profiling` feature flag
22
-
-**Output**: Flamegraphs (SVG) and protobuf profiles in `/opt/profiling/{service-name}/`
21
+
1. Set up the local test network:
23
22
24
-
### External Profiling Tools
23
+
```bash
24
+
just setup
25
+
```
25
26
26
-
The profiling environment also supports the following tools:
27
+
2. Fund the escrow account (required for testing):
|**strace**| Traces system calls with detailed timing |`/opt/profiling/{service-name}/strace.log`|
31
-
|**valgrind**| Memory profiling with Massif |`/opt/profiling/{service-name}/massif.out`|
32
-
|**callgrind**| CPU profiling (part of valgrind) |`/opt/profiling/{service-name}/callgrind.out`|
29
+
```bash
30
+
cd integration-tests
31
+
./fund_escrow.sh
32
+
```
33
33
34
-
##How to Use
34
+
### Running Load Tests
35
35
36
-
### Prerequisites
36
+
The integration tests include a dedicated load testing tool that communicates directly with the indexer-service (bypassing the gateway). This allows for more accurate performance measurements.
37
37
38
-
Run the setup command first to prepare the testing environment:
38
+
To run a load test:
39
39
40
40
```bash
41
-
just setup
41
+
# Run with 1000 receipts
42
+
cargo run -- load --num-receipts 1000
43
+
44
+
# Run with custom number of receipts
45
+
cargo run -- load --num-receipts <number>
42
46
```
43
47
44
-
### Profiling Commands
48
+
The load test will:
49
+
50
+
- Send the specified number of receipts concurrently
51
+
- Use all available CPU cores for concurrency
52
+
- Report success/failure rates
53
+
- Show average processing time per request
54
+
- Display total duration
55
+
56
+
### Understanding Load Test Results
57
+
58
+
The load test output includes:
59
+
60
+
- Total number of receipts processed
61
+
- Processing duration
62
+
- Average time per request
63
+
- Number of successful receipts
64
+
- Number of failed receipts
65
+
66
+
Example output:
67
+
68
+
```
69
+
Completed processing 1000 requests in 2.5s
70
+
Average time per request: 2.5ms
71
+
Successfully sent receipts: 998
72
+
Failed receipts: 2
73
+
```
74
+
75
+
## Profiling
76
+
77
+
### Available Profiling Methods
78
+
79
+
The system supports multiple profiling tools:
80
+
81
+
1.**Flamegraph** (Default)
82
+
83
+
- Visual representation of CPU usage
84
+
- Shows function call stacks
85
+
- Based on [pprof](https://crates.io/crates/pprof)
86
+
- Output: SVG files and protobuf profiles in `/opt/profiling/{service-name}/`
87
+
88
+
2.**Valgrind Massif**
89
+
90
+
- Memory profiling
91
+
- Tracks heap usage
92
+
- Output: `massif.out` files
93
+
94
+
3.**Callgrind**
95
+
96
+
- Detailed CPU profiling
97
+
- Cache and branch prediction analysis
98
+
- Output: `callgrind.out` files
99
+
100
+
4.**Strace**
101
+
- System call tracing
102
+
- Detailed timing information
103
+
- Output: `strace.log` files
104
+
105
+
### Running Profiling
45
106
46
-
Use the following commands to profile specific services:
107
+
1. Start profiling with your chosen tool:
47
108
48
109
```bash
49
-
#Profile with flamegraph (default)
110
+
#For flamegraph (default)
50
111
just profile-flamegraph
51
112
52
-
#Profile with valgrind
113
+
#For memory profiling
53
114
just profile-valgrind
54
115
55
-
#Profile with strace
116
+
#For system call tracing
56
117
just profile-strace
57
118
58
-
#Profile with callgrind
119
+
#For CPU profiling
59
120
just profile-callgrind
121
+
```
122
+
123
+
2. Run your load test while profiling:
124
+
125
+
```bash
126
+
cargo run -- load --num-receipts 1000
127
+
```
128
+
129
+
3. Stop profiling to generate results:
60
130
61
-
# Stop profiling (gracefully terminate to generate output)
- For production use, prefer the built-in profiler over external tools to minimize performance impact
197
+
198
+
3.**Environment**
199
+
- Ensure clean state before testing
200
+
- Monitor system resources
201
+
- Check logs for errors
202
+
- Verify escrow funding before testing
203
+
204
+
## Troubleshooting
205
+
206
+
1. If load tests fail:
207
+
208
+
- Check escrow funding
209
+
- Verify service health
210
+
- Check logs for errors
211
+
- Ensure proper network setup
212
+
213
+
2. If profiling fails:
214
+
- Check container permissions
215
+
- Verify profiling tool installation
216
+
- Check available disk space
217
+
- Ensure proper service shutdown
218
+
103
219
## Implementation Details
104
220
105
221
### Profiler Integration
@@ -135,11 +251,16 @@ security_opt:
135
251
## Notes
136
252
137
253
- The flamegraph profiling is enabled whenever using any of the profiling commands through the Justfile, as the binaries are compiled with the `profiling` feature flag.
138
-
- For production use, prefer the built-in profiler over the external tools to minimize performance impact.
254
+
- Built-in profiler was chosen because tools like `perf`, while powerful, often pose configuration challenges or require specific capabilities (like CAP_SYS_ADMIN) that complicate their deployment within standard Docker containers.
139
255
- When using callgrind, consider enabling debug information and frame pointers in your Cargo.toml for better output:
0 commit comments