|
| 1 | +# JCasbin Benchmark Suite |
| 2 | + |
| 3 | +This directory contains the official JCasbin benchmark suite aligned with [go-casbin](https://github.com/casbin/casbin) standard benchmark scenarios. |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +The benchmark suite tests authorization performance across different Casbin models and data scales. All benchmarks use deterministic data generation to ensure reproducible results across runs. |
| 8 | + |
| 9 | +## Benchmark Scenarios |
| 10 | + |
| 11 | +The following standard scenarios are implemented: |
| 12 | + |
| 13 | +| Scenario | Rules | Users | Roles | Resources/Domains | Description | |
| 14 | +|----------|-------|-------|-------|------------------|-------------| |
| 15 | +| **ACL** | 2 | 2 | - | - | Basic Access Control List | |
| 16 | +| **RBAC** | 5 | 2 | 1 | - | Basic Role-Based Access Control | |
| 17 | +| **RBAC Small** | 1,100 | 1,000 | 100 | 10 | Small-scale RBAC | |
| 18 | +| **RBAC Medium** | 11,000 | 10,000 | 1,000 | 100 | Medium-scale RBAC | |
| 19 | +| **RBAC Large** | 110,000 | 100,000 | 10,000 | 1,000 | Large-scale RBAC | |
| 20 | +| **RBAC with Resource Roles** | 6 | 2 | 2 | - | RBAC with resource grouping | |
| 21 | +| **RBAC with Domains** | 6 | 2 | 1 | 2 domains | Multi-tenant RBAC | |
| 22 | +| **ABAC** | 0 | 0 | - | - | Attribute-Based Access Control | |
| 23 | +| **RESTful/KeyMatch** | 5 | 3 | - | - | REST API pattern matching | |
| 24 | +| **Deny-override** | 6 | 2 | 1 | - | RBAC with explicit deny | |
| 25 | +| **Priority** | 9 | 2 | 2 | - | Priority-based authorization | |
| 26 | + |
| 27 | +## Running Benchmarks |
| 28 | + |
| 29 | +### Prerequisites |
| 30 | + |
| 31 | +- JDK 8 or higher |
| 32 | +- Maven 3.x |
| 33 | + |
| 34 | +### Running All Benchmarks |
| 35 | + |
| 36 | +```bash |
| 37 | +# Using Maven |
| 38 | +mvn clean test-compile exec:java -Dexec.mainClass="org.openjdk.jmh.Main" -Dexec.classpathScope=test |
| 39 | + |
| 40 | +# Or build and run JMH benchmark JAR |
| 41 | +mvn clean package |
| 42 | +java -jar target/benchmarks.jar |
| 43 | +``` |
| 44 | + |
| 45 | +### Running a Specific Benchmark |
| 46 | + |
| 47 | +```bash |
| 48 | +# Run a single benchmark class |
| 49 | +mvn clean test-compile exec:java -Dexec.mainClass="org.casbin.jcasbin.main.benchmark.BenchmarkBasicModel" -Dexec.classpathScope=test |
| 50 | + |
| 51 | +# Or using JMH pattern matching |
| 52 | +java -jar target/benchmarks.jar BenchmarkBasicModel |
| 53 | +``` |
| 54 | + |
| 55 | +### Recommended JMH Parameters |
| 56 | + |
| 57 | +All benchmarks are configured with the following recommended parameters to match go-casbin benchmark behavior: |
| 58 | + |
| 59 | +``` |
| 60 | +-f 2 # 2 forks |
| 61 | +-wi 3 # 3 warmup iterations |
| 62 | +-i 5 # 5 measurement iterations |
| 63 | +-t 1 # 1 thread |
| 64 | +``` |
| 65 | + |
| 66 | +These can be overridden when running benchmarks: |
| 67 | + |
| 68 | +```bash |
| 69 | +java -jar target/benchmarks.jar -f 1 -wi 5 -i 10 |
| 70 | +``` |
| 71 | + |
| 72 | +## Benchmark Classes |
| 73 | + |
| 74 | +### BenchmarkBasicModel (ACL) |
| 75 | +- **File**: `BenchmarkBasicModel.java` |
| 76 | +- **Model**: ACL (Access Control List) |
| 77 | +- **Scale**: 2 rules, 2 users |
| 78 | +- **Test**: `enforce("alice", "data1", "read")` |
| 79 | + |
| 80 | +### BenchmarkRBACModelSingle (RBAC) |
| 81 | +- **File**: `BenchmarkRBACModelSingle.java` |
| 82 | +- **Model**: Basic RBAC |
| 83 | +- **Scale**: 5 rules (4 policies + 1 role assignment), 2 users, 1 role |
| 84 | +- **Test**: `enforce("alice", "data2", "read")` |
| 85 | + |
| 86 | +### BenchmarkRBACModelSmall |
| 87 | +- **File**: `BenchmarkRBACModelSmall.java` |
| 88 | +- **Model**: RBAC with small dataset |
| 89 | +- **Scale**: 1,100 rules, 1,000 users, 100 roles, 10 resources |
| 90 | +- **Test**: `enforce("user501", "data9", "read")` |
| 91 | +- **Generation**: Every 10 roles → 1 resource, Every 10 users → 1 role |
| 92 | + |
| 93 | +### BenchmarkRBACModelMedium |
| 94 | +- **File**: `BenchmarkRBACModelMedium.java` |
| 95 | +- **Model**: RBAC with medium dataset |
| 96 | +- **Scale**: 11,000 rules, 10,000 users, 1,000 roles, 100 resources |
| 97 | +- **Test**: `enforce("user5001", "data150", "read")` |
| 98 | +- **Generation**: Every 10 roles → 1 resource, Every 10 users → 1 role |
| 99 | + |
| 100 | +### BenchmarkRBACModelLarge |
| 101 | +- **File**: `BenchmarkRBACModelLarge.java` |
| 102 | +- **Model**: RBAC with large dataset |
| 103 | +- **Scale**: 110,000 rules, 100,000 users, 10,000 roles, 1,000 resources |
| 104 | +- **Test**: `enforce("user50001", "data1500", "read")` |
| 105 | +- **Generation**: Every 10 roles → 1 resource, Every 10 users → 1 role |
| 106 | + |
| 107 | +### BenchmarkRBACModelWithResourceRoles |
| 108 | +- **File**: `BenchmarkRBACModelWithResourceRoles.java` |
| 109 | +- **Model**: RBAC with resource roles |
| 110 | +- **Scale**: 6 rules, 2 users, 2 roles |
| 111 | +- **Test**: `enforce("alice", "data1", "read")` |
| 112 | + |
| 113 | +### BenchmarkRBACModelWithDomains |
| 114 | +- **File**: `BenchmarkRBACModelWithDomains.java` |
| 115 | +- **Model**: RBAC with multi-tenancy |
| 116 | +- **Scale**: 6 rules, 2 users, 1 role, 2 domains |
| 117 | +- **Test**: `enforce("alice", "domain1", "data1", "read")` |
| 118 | + |
| 119 | +### BenchmarkABACModel |
| 120 | +- **File**: `BenchmarkABACModel.java` |
| 121 | +- **Model**: ABAC (Attribute-Based Access Control) |
| 122 | +- **Scale**: 0 rules (attribute-based logic in model) |
| 123 | +- **Test**: `enforce("alice", data1, "read")` |
| 124 | + |
| 125 | +### BenchmarkKeyMatchModel (RESTful) |
| 126 | +- **File**: `BenchmarkKeyMatchModel.java` |
| 127 | +- **Model**: RESTful with pattern matching |
| 128 | +- **Scale**: 5 rules, 3 users |
| 129 | +- **Test**: `enforce("alice", "/alice_data/resource1", "GET")` |
| 130 | + |
| 131 | +### BenchmarkRBACModelWithDeny (Deny-override) |
| 132 | +- **File**: `BenchmarkRBACModelWithDeny.java` |
| 133 | +- **Model**: RBAC with explicit deny |
| 134 | +- **Scale**: 6 rules (5 policies + 1 role assignment), 2 users, 1 role |
| 135 | +- **Test**: `enforce("alice", "data1", "read")` |
| 136 | + |
| 137 | +### BenchmarkPriorityModel |
| 138 | +- **File**: `BenchmarkPriorityModel.java` |
| 139 | +- **Model**: Priority-based authorization |
| 140 | +- **Scale**: 9 rules (7 policies + 2 role assignments), 2 users, 2 roles |
| 141 | +- **Test**: `enforce("alice", "data1", "read")` |
| 142 | + |
| 143 | +## Deterministic Data Generation |
| 144 | + |
| 145 | +All benchmarks use deterministic policy generation to ensure identical results across runs: |
| 146 | + |
| 147 | +- No randomness in data generation |
| 148 | +- Loop counters use integer division for predictable patterns |
| 149 | +- Static initialization blocks generate policies before benchmarking |
| 150 | +- User/role/resource names follow consistent patterns: `user{i}`, `group{i}`, `data{i}` |
| 151 | + |
| 152 | +Example from RBAC Small: |
| 153 | +```java |
| 154 | +// 100 roles, 10 resources |
| 155 | +for (int i = 0; i < 100; i++) { |
| 156 | + e.addPolicy(String.format("group%d", i), String.format("data%d", i/10), "read"); |
| 157 | +} |
| 158 | +// 1000 users |
| 159 | +for (int i = 0; i < 1000; i++) { |
| 160 | + e.addGroupingPolicy(String.format("user%d", i), String.format("group%d", i/10)); |
| 161 | +} |
| 162 | +``` |
| 163 | + |
| 164 | +## JMH Configuration |
| 165 | + |
| 166 | +All benchmarks use the following JMH annotations: |
| 167 | + |
| 168 | +```java |
| 169 | +@OutputTimeUnit(TimeUnit.MILLISECONDS) |
| 170 | +@BenchmarkMode(Mode.AverageTime) |
| 171 | +@Threads(1) |
| 172 | +``` |
| 173 | + |
| 174 | +This configuration measures average execution time per operation in milliseconds, similar to Go's `ns/op` metric. |
| 175 | + |
| 176 | +## Comparison with go-casbin |
| 177 | + |
| 178 | +These benchmarks are designed to be directly comparable with [go-casbin benchmarks](https://github.com/casbin/casbin/tree/master/benchmarks): |
| 179 | + |
| 180 | +- Identical data scales |
| 181 | +- Same policy generation logic |
| 182 | +- Same test cases (user, resource, action) |
| 183 | +- Consistent naming conventions |
| 184 | + |
| 185 | +This allows for fair performance comparisons between JCasbin and go-casbin implementations. |
| 186 | + |
| 187 | +## Contributing |
| 188 | + |
| 189 | +When adding new benchmarks: |
| 190 | + |
| 191 | +1. Follow the naming convention: `Benchmark<Scenario>.java` |
| 192 | +2. Add comprehensive Javadoc with: |
| 193 | + - Scenario description |
| 194 | + - Data scale details |
| 195 | + - Policy structure |
| 196 | + - Test case |
| 197 | + - Recommended JMH options |
| 198 | +3. Use deterministic data generation (no randomness) |
| 199 | +4. Use standard JMH parameters: `-f 2 -wi 3 -i 5 -t 1` |
| 200 | +5. Update this README with the new benchmark details |
| 201 | + |
| 202 | +## References |
| 203 | + |
| 204 | +- [Casbin Official Documentation](https://casbin.org/docs/en/overview) |
| 205 | +- [go-casbin Benchmarks](https://github.com/casbin/casbin/tree/master/benchmarks) |
| 206 | +- [JMH Documentation](https://github.com/openjdk/jmh) |
| 207 | +- [Casbin Performance Monitor](https://casbin.org/docs/en/benchmark) |
0 commit comments