Skip to content

Commit f232735

Browse files
Copilothsluoyz
andcommitted
Add comprehensive benchmark documentation README
Co-authored-by: hsluoyz <3787410+hsluoyz@users.noreply.github.com>
1 parent f572ce5 commit f232735

File tree

1 file changed

+207
-0
lines changed
  • src/test/java/org/casbin/jcasbin/main/benchmark

1 file changed

+207
-0
lines changed
Lines changed: 207 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,207 @@
1+
# JCasbin Benchmark Suite
2+
3+
This directory contains the official JCasbin benchmark suite aligned with [go-casbin](https://github.com/casbin/casbin) standard benchmark scenarios.
4+
5+
## Overview
6+
7+
The benchmark suite tests authorization performance across different Casbin models and data scales. All benchmarks use deterministic data generation to ensure reproducible results across runs.
8+
9+
## Benchmark Scenarios
10+
11+
The following standard scenarios are implemented:
12+
13+
| Scenario | Rules | Users | Roles | Resources/Domains | Description |
14+
|----------|-------|-------|-------|------------------|-------------|
15+
| **ACL** | 2 | 2 | - | - | Basic Access Control List |
16+
| **RBAC** | 5 | 2 | 1 | - | Basic Role-Based Access Control |
17+
| **RBAC Small** | 1,100 | 1,000 | 100 | 10 | Small-scale RBAC |
18+
| **RBAC Medium** | 11,000 | 10,000 | 1,000 | 100 | Medium-scale RBAC |
19+
| **RBAC Large** | 110,000 | 100,000 | 10,000 | 1,000 | Large-scale RBAC |
20+
| **RBAC with Resource Roles** | 6 | 2 | 2 | - | RBAC with resource grouping |
21+
| **RBAC with Domains** | 6 | 2 | 1 | 2 domains | Multi-tenant RBAC |
22+
| **ABAC** | 0 | 0 | - | - | Attribute-Based Access Control |
23+
| **RESTful/KeyMatch** | 5 | 3 | - | - | REST API pattern matching |
24+
| **Deny-override** | 6 | 2 | 1 | - | RBAC with explicit deny |
25+
| **Priority** | 9 | 2 | 2 | - | Priority-based authorization |
26+
27+
## Running Benchmarks
28+
29+
### Prerequisites
30+
31+
- JDK 8 or higher
32+
- Maven 3.x
33+
34+
### Running All Benchmarks
35+
36+
```bash
37+
# Using Maven
38+
mvn clean test-compile exec:java -Dexec.mainClass="org.openjdk.jmh.Main" -Dexec.classpathScope=test
39+
40+
# Or build and run JMH benchmark JAR
41+
mvn clean package
42+
java -jar target/benchmarks.jar
43+
```
44+
45+
### Running a Specific Benchmark
46+
47+
```bash
48+
# Run a single benchmark class
49+
mvn clean test-compile exec:java -Dexec.mainClass="org.casbin.jcasbin.main.benchmark.BenchmarkBasicModel" -Dexec.classpathScope=test
50+
51+
# Or using JMH pattern matching
52+
java -jar target/benchmarks.jar BenchmarkBasicModel
53+
```
54+
55+
### Recommended JMH Parameters
56+
57+
All benchmarks are configured with the following recommended parameters to match go-casbin benchmark behavior:
58+
59+
```
60+
-f 2 # 2 forks
61+
-wi 3 # 3 warmup iterations
62+
-i 5 # 5 measurement iterations
63+
-t 1 # 1 thread
64+
```
65+
66+
These can be overridden when running benchmarks:
67+
68+
```bash
69+
java -jar target/benchmarks.jar -f 1 -wi 5 -i 10
70+
```
71+
72+
## Benchmark Classes
73+
74+
### BenchmarkBasicModel (ACL)
75+
- **File**: `BenchmarkBasicModel.java`
76+
- **Model**: ACL (Access Control List)
77+
- **Scale**: 2 rules, 2 users
78+
- **Test**: `enforce("alice", "data1", "read")`
79+
80+
### BenchmarkRBACModelSingle (RBAC)
81+
- **File**: `BenchmarkRBACModelSingle.java`
82+
- **Model**: Basic RBAC
83+
- **Scale**: 5 rules (4 policies + 1 role assignment), 2 users, 1 role
84+
- **Test**: `enforce("alice", "data2", "read")`
85+
86+
### BenchmarkRBACModelSmall
87+
- **File**: `BenchmarkRBACModelSmall.java`
88+
- **Model**: RBAC with small dataset
89+
- **Scale**: 1,100 rules, 1,000 users, 100 roles, 10 resources
90+
- **Test**: `enforce("user501", "data9", "read")`
91+
- **Generation**: Every 10 roles → 1 resource, Every 10 users → 1 role
92+
93+
### BenchmarkRBACModelMedium
94+
- **File**: `BenchmarkRBACModelMedium.java`
95+
- **Model**: RBAC with medium dataset
96+
- **Scale**: 11,000 rules, 10,000 users, 1,000 roles, 100 resources
97+
- **Test**: `enforce("user5001", "data150", "read")`
98+
- **Generation**: Every 10 roles → 1 resource, Every 10 users → 1 role
99+
100+
### BenchmarkRBACModelLarge
101+
- **File**: `BenchmarkRBACModelLarge.java`
102+
- **Model**: RBAC with large dataset
103+
- **Scale**: 110,000 rules, 100,000 users, 10,000 roles, 1,000 resources
104+
- **Test**: `enforce("user50001", "data1500", "read")`
105+
- **Generation**: Every 10 roles → 1 resource, Every 10 users → 1 role
106+
107+
### BenchmarkRBACModelWithResourceRoles
108+
- **File**: `BenchmarkRBACModelWithResourceRoles.java`
109+
- **Model**: RBAC with resource roles
110+
- **Scale**: 6 rules, 2 users, 2 roles
111+
- **Test**: `enforce("alice", "data1", "read")`
112+
113+
### BenchmarkRBACModelWithDomains
114+
- **File**: `BenchmarkRBACModelWithDomains.java`
115+
- **Model**: RBAC with multi-tenancy
116+
- **Scale**: 6 rules, 2 users, 1 role, 2 domains
117+
- **Test**: `enforce("alice", "domain1", "data1", "read")`
118+
119+
### BenchmarkABACModel
120+
- **File**: `BenchmarkABACModel.java`
121+
- **Model**: ABAC (Attribute-Based Access Control)
122+
- **Scale**: 0 rules (attribute-based logic in model)
123+
- **Test**: `enforce("alice", data1, "read")`
124+
125+
### BenchmarkKeyMatchModel (RESTful)
126+
- **File**: `BenchmarkKeyMatchModel.java`
127+
- **Model**: RESTful with pattern matching
128+
- **Scale**: 5 rules, 3 users
129+
- **Test**: `enforce("alice", "/alice_data/resource1", "GET")`
130+
131+
### BenchmarkRBACModelWithDeny (Deny-override)
132+
- **File**: `BenchmarkRBACModelWithDeny.java`
133+
- **Model**: RBAC with explicit deny
134+
- **Scale**: 6 rules (5 policies + 1 role assignment), 2 users, 1 role
135+
- **Test**: `enforce("alice", "data1", "read")`
136+
137+
### BenchmarkPriorityModel
138+
- **File**: `BenchmarkPriorityModel.java`
139+
- **Model**: Priority-based authorization
140+
- **Scale**: 9 rules (7 policies + 2 role assignments), 2 users, 2 roles
141+
- **Test**: `enforce("alice", "data1", "read")`
142+
143+
## Deterministic Data Generation
144+
145+
All benchmarks use deterministic policy generation to ensure identical results across runs:
146+
147+
- No randomness in data generation
148+
- Loop counters use integer division for predictable patterns
149+
- Static initialization blocks generate policies before benchmarking
150+
- User/role/resource names follow consistent patterns: `user{i}`, `group{i}`, `data{i}`
151+
152+
Example from RBAC Small:
153+
```java
154+
// 100 roles, 10 resources
155+
for (int i = 0; i < 100; i++) {
156+
e.addPolicy(String.format("group%d", i), String.format("data%d", i/10), "read");
157+
}
158+
// 1000 users
159+
for (int i = 0; i < 1000; i++) {
160+
e.addGroupingPolicy(String.format("user%d", i), String.format("group%d", i/10));
161+
}
162+
```
163+
164+
## JMH Configuration
165+
166+
All benchmarks use the following JMH annotations:
167+
168+
```java
169+
@OutputTimeUnit(TimeUnit.MILLISECONDS)
170+
@BenchmarkMode(Mode.AverageTime)
171+
@Threads(1)
172+
```
173+
174+
This configuration measures average execution time per operation in milliseconds, similar to Go's `ns/op` metric.
175+
176+
## Comparison with go-casbin
177+
178+
These benchmarks are designed to be directly comparable with [go-casbin benchmarks](https://github.com/casbin/casbin/tree/master/benchmarks):
179+
180+
- Identical data scales
181+
- Same policy generation logic
182+
- Same test cases (user, resource, action)
183+
- Consistent naming conventions
184+
185+
This allows for fair performance comparisons between JCasbin and go-casbin implementations.
186+
187+
## Contributing
188+
189+
When adding new benchmarks:
190+
191+
1. Follow the naming convention: `Benchmark<Scenario>.java`
192+
2. Add comprehensive Javadoc with:
193+
- Scenario description
194+
- Data scale details
195+
- Policy structure
196+
- Test case
197+
- Recommended JMH options
198+
3. Use deterministic data generation (no randomness)
199+
4. Use standard JMH parameters: `-f 2 -wi 3 -i 5 -t 1`
200+
5. Update this README with the new benchmark details
201+
202+
## References
203+
204+
- [Casbin Official Documentation](https://casbin.org/docs/en/overview)
205+
- [go-casbin Benchmarks](https://github.com/casbin/casbin/tree/master/benchmarks)
206+
- [JMH Documentation](https://github.com/openjdk/jmh)
207+
- [Casbin Performance Monitor](https://casbin.org/docs/en/benchmark)

0 commit comments

Comments
 (0)