Home
- nginx: 1.11.10
- ModSecurity: v3/dev/parser (8b8fd84)
- ModSecurity-nginx: v3/dev/parser (9f6d3a7)
- environment: 2-core VirtualBox VM on MBP A1502 (early 2015, 2-core i5 2.9GHz)
Summary for /modsec-off, RPS (count):
N Min Max Median Avg Stddev
x 10 39445.56 46228.25 44283.87 43476.616 2184.7312
latency (ms)
x 10 1.1 1.37 1.21 1.222 0.091627264
Summary for /modsec-light, RPS (count):
N Min Max Median Avg Stddev
x 10 8071.25 11953.68 10634.85 10374.126 1200.2281
latency (ms)
x 10 4.23 7.82 5 5.358 1.2109941
Summary for /modsec-full, RPS (count):
N Min Max Median Avg Stddev
x 10 238.8 259.02 246.88 247.418 5.9640549
latency (ms)
x 10 208.11 230.52 217.76 218.968 6.9131273
- nginx: 1.11.10
- ModSecurity: v3/master (3a41308)
- ModSecurity-nginx: master (134bd36)
- environment: 2-core VirtualBox VM on MBP A1502 (early 2015, 2-core i5 2.9GHz)
Summary for /modsec-off, RPS (count):
N Min Max Median Avg Stddev
x 10 38136.4 47561.79 44300.42 43351.954 2743.1755
latency (ms)
x 10 1.07 1.4 1.19 1.206 0.099911072
Summary for /modsec-light, RPS (count):
N Min Max Median Avg Stddev
x 10 10120.76 12979.9 12727.41 12290.594 891.67524
latency (ms)
x 10 3.88 5.02 3.98 4.128 0.34726871
Summary for /modsec-full, RPS (count):
N Min Max Median Avg Stddev
x 10 334.64 370.86 363.77 356.733 13.667706
latency (ms)
x 10 142.49 158.27 147.04 148.176 5.8598297
- nginx: 1.11.10
- ModSecurity: v3/dev/parser (8b8fd84)
- ModSecurity-nginx: v3/dev/parser (9f6d3a7)
- environment: 12-core KVM/libvirt VM on bare-metal server (Intel Xeon E5645 2.4GHz, 24 cores total)
- configuration details:
- nginx:
worker_processes 6; worker_cpu_affinity 111111000000; - wrk:
taskset -c 0-5 wrk -t6 -c600 -d30s(10 iterations)
Summary for /modsec-off, RPS (count):
N Min Max Median Avg Stddev
x 10 87538.41 97062.44 93506.49 92832.83 2717.0839
latency (ms)
x 10 6.22 6.88 6.49 6.508 0.21054427
Summary for /modsec-light, RPS (count):
N Min Max Median Avg Stddev
x 10 27984.07 31588.76 31013.95 30484.634 1151.9494
latency (ms)
x 10 18.98 21.57 20.06 20.01 0.81266366
Summary for /modsec-full, RPS (count):
N Min Max Median Avg Stddev
x 10 164.5 251.93 181.79 191.226 28.760167
latency (ms)
x 10 197.41 429.26 330.67 308.639 73.228166
The numbers for /modsec-off and /modsec-light with multi-worker nginx setup are significantly better than in single-worker mode, but /modsec-full does not show any difference. In the process of investigation it turned out that disabling audit log (by setting SecAuditEngine Off) greatly improves overall performance with OWASP CRS v3.0.0 loaded:
Summary for /modsec-full, RPS (count):
N Min Max Median Avg Stddev
x 10 577.99 864.63 854.69 823.902 87.17359
latency (ms)
x 10 633.42 730.81 646.95 654.834 28.487218
Further investigation showed that in case of SecAuditEngine RelevantOnly only one CPU core (among the set of cores nginx is using according to worker_cpu_affinity) is 100% busy:
With SecAuditEngine Off, all cores that are being used by nginx are constantly busy:
- nginx: 1.11.10
- ModSecurity: v3/master (53485c7)
- ModSecurity-nginx: master (5175214)
- environment: 2-core VirtualBox VM on MBP A1502 (early 2015, 2-core i5 2.9GHz)
Summary for /modsec-off, RPS (count):
N Min Max Median Avg Stddev
x 10 44040.08 56882.7 53699.65 51600.859 4057.3947
latency (ms)
x 10 0.87 1.14 0.99 0.977 0.084859361
Summary for /modsec-light, RPS (count):
N Min Max Median Avg Stddev
x 10 10568.21 13532.37 12921.28 12336.21 1207.9385
latency (ms)
x 10 3.71 4.74 3.9 4.104 0.42893149
Summary for /modsec-full, RPS (count):
N Min Max Median Avg Stddev
x 10 238.79 259.41 253.79 250.74 7.4913906
latency (ms)
x 10 207.47 221.61 213.96 213.635 5.7969556
- nginx: 1.11.10
- ModSecurity: v3/dev/speedup (d9fabea)
- ModSecurity-nginx: master (5175214)
- environment: 2-core VirtualBox VM on MBP A1502 (early 2015, 2-core i5 2.9GHz)
Summary for /modsec-off, RPS (count):
N Min Max Median Avg Stddev
x 10 48151.25 54508.49 52135.16 51337.515 2114.2494
latency (ms)
x 10 0.92 1.14 0.99 0.996 0.069633964
Summary for /modsec-light, RPS (count):
N Min Max Median Avg Stddev
x 10 10456.84 13941.7 13011.89 12441.49 1375.0929
latency (ms)
x 10 3.59 4.8 3.97 4.078 0.48276518
Summary for /modsec-full, RPS (count):
N Min Max Median Avg Stddev
x 10 270.44 284.01 278.54 278.824 4.1302763
latency (ms)
x 10 187.65 198.52 191.41 191.247 3.5242337
- nginx: 1.11.10
- ModSecurity: v3/master (b58f713)
- ModSecurity-nginx: master (3de175b)
- environment: 2-core VirtualBox VM on MBP A1502 (early 2015, 2-core i5 2.9GHz)
Summary for /modsec-off, RPS (count):
N Min Max Median Avg Stddev
x 10 46879.99 58925.23 53077.84 53435.713 3596.6361
latency (ms)
x 10 0.85 843.78 0.96 85.236 266.5252
Summary for /modsec-light, RPS (count):
N Min Max Median Avg Stddev
x 10 10060.67 13613.14 12955.81 12213.674 1312.7484
latency (ms)
x 10 3.67 5.16 4.02 4.164 0.5084661
Summary for /modsec-full, RPS (count):
N Min Max Median Avg Stddev
x 10 271.63 295.32 290.55 287.362 7.2825313
latency (ms)
x 10 179.23 192.48 183.71 185.278 4.2224369
- nginx: 1.11.10
- ModSecurity: v3/master (b58f713)
- ModSecurity-nginx: master (3de175b)
- environment: 12-core KVM/libvirt VM on bare-metal server (Intel Xeon E5645 2.4GHz, 24 cores total)
- configuration details:
- nginx:
worker_processes 6; worker_cpu_affinity 111111000000; - wrk:
taskset -c 0-5 wrk -t6 -c600 -d30s(10 iterations)
Summary for /modsec-off, RPS (count):
N Min Max Median Avg Stddev
x 10 97613.95 101573.29 99853.65 99354.804 1370.9853
latency (ms)
x 10 5.94 6.85 6.11 6.145 0.2596258
Summary for /modsec-light, RPS (count):
N Min Max Median Avg Stddev
x 10 23101.87 31886.19 31184.36 30033.129 2761.2698
latency (ms)
x 10 18.89 26.73 20 21.017 2.5738603
Summary for /modsec-full, RPS (count):
N Min Max Median Avg Stddev
x 10 931.68 946.17 939.08 939.032 4.0696869
latency (ms)
x 10 580.29 626.9 600.27 598.622 14.275303
With the recent sources all the nginx workers are busy while benchmarking /modsec-full location (previously only one worker process reached to 100% CPU usage).
Also, turning SecAuditEngine Off does not affect latency && rps anymore.
- OS updated to Ubuntu 17.04 "zesty" (kernel 4.10.0-30-generic #34-Ubuntu)
- nginx: 1.13.4
- ModSecurity: v3/master (8d6209f)
- ModSecurity-nginx: master (abbf2c4)
- environment: 12-core KVM/libvirt VM on bare-metal server (Intel Xeon E5-2660 2.20GHz, 32 cores total)
- configuration details:
- nginx:
worker_processes 6; worker_cpu_affinity 111111000000; - wrk:
taskset -c 0-5 wrk -t6 -c600 -d30s(10 iterations)
Summary for /modsec-off, RPS (count):
N Min Max Median Avg Stddev
x 10 53343.88 103336.29 100394.62 94998.443 14838.037
latency (ms)
x 10 5.81 11.35 6.12 6.58 1.6827689
Summary for /modsec-light, RPS (count):
N Min Max Median Avg Stddev
x 10 20488.98 29468.81 28974.4 28141.574 2706.7983
latency (ms)
x 10 20.44 29.72 21.36 22.29 2.7457927
Summary for /modsec-full, RPS (count):
N Min Max Median Avg Stddev
x 10 813.95 965.7 940.75 931.136 42.406455
latency (ms)
x 10 541.48 627.62 605.03 599.103 23.770327
- nginx: 1.13.4
- ModSecurity: v3/master (04f7009)
- ModSecurity-nginx: master (abbf2c4)
- environment: 12-core KVM/libvirt VM on bare-metal server (Intel Xeon E5-2660 2.20GHz, 32 cores total)
- configuration details:
- nginx:
worker_processes 6; worker_cpu_affinity 111111000000; - wrk:
taskset -c 0-5 wrk -t6 -c600 -d30s(10 iterations)
Summary for /modsec-off, RPS (count):
N Min Max Median Avg Stddev
x 10 59232.06 98088.52 94674.82 90202.254 11188.852
latency (ms)
x 10 6.18 10.15 6.44 6.799 1.1892336
Summary for /modsec-light, RPS (count):
N Min Max Median Avg Stddev
x 10 25687.36 26564.29 26299.24 26197.744 294.37041
latency (ms)
x 10 22.76 25.53 25.08 24.543 0.91399064
Summary for /modsec-full, RPS (count):
N Min Max Median Avg Stddev
x 10 988.93 1037.52 1008.77 1010.23 12.988515
latency (ms)
x 10 575.26 604.29 589.26 590.084 8.6321379
- nginx: 1.13.7
- ModSecurity: v3/master (81e1cdc)
- ModSecurity-nginx: master (a2a5858)
- environment: 12-core KVM/libvirt VM on bare-metal server (Intel Xeon E5-2660 2.20GHz, 32 cores total)
- configuration details:
- nginx:
worker_processes 6; worker_cpu_affinity 111111000000; - wrk:
taskset -c 0-5 wrk -t6 -c600 -d30s(10 iterations)
Summary for /modsec-off, RPS (count):
N Min Max Median Avg Stddev
x 10 98673.71 109369.41 107954.84 106506.79 3085.2164
latency (ms)
x 10 5.48 6.08 5.59 5.666 0.18530455
Summary for /modsec-light, RPS (count):
N Min Max Median Avg Stddev
x 10 27851.53 29340.27 28257.97 28321.345 475.51244
latency (ms)
x 10 20.8 22.41 22.26 21.944 0.5227959
Summary for /modsec-full, RPS (count):
N Min Max Median Avg Stddev
x 10 720.59 731.77 726.96 726.013 4.0179377
latency (ms)
x 10 425.48 743.31 604.91 606.949 117.04856
- nginx: 1.13.7
- ModSecurity: revisions from a2427df27f482c64ea8666dca9552c67d3a68904 to head of v3/master
- ModSecurity-nginx: https://github.com/SpiderLabs/ModSecurity-nginx/releases/tag/v1.0.0
- environment: 12-core KVM/libvirt VM on bare-metal server (Intel Xeon E5-2660 2.20GHz, 32 cores total)
- configuration details:
- nginx:
worker_processes 6; worker_cpu_affinity 111111000000; - wrk:
taskset -c 0-5 wrk -t6 -c600 -d30s(3 iterations, averages from ministat for/modsec-fulllocation)
;rps_avg,latency_avg,revision,date,commit_log
890.54667,601.61,a2427df27f482c64ea8666dca9552c67d3a68904,2017-08-27 23:39:43 -0300,fix: ignore .git directory while generating the release file
893.12,601.18,119a6fc07482096e8429399dc2d7c0d3f903a7ae,2017-09-07 22:23:07 -0300,test-only: Placing a mutex while evaluating the pm operator
896.33667,595.66667,7d786b335024f2c896eb30830427c54f28dcc44c,2017-09-07 22:23:34 -0300,Makes pm mutex optional via configuration flag
892.11333,605.76667,1c91e807778f826b20d45abcef9a204e8f313d01,2017-09-07 22:23:48 -0300,Extends acmp_prepare to pm_from_file
884.25333,602.43667,48be601ca74f2f6496c8fb47371a6f9d884f9cf1,2017-09-26 16:33:48 +0000,Very first version of our changes file
888.53333,601.2,082a0d3acabc6e5d87f7202b7fad9e9ba7d64953,2017-09-11 12:44:53 +0000,Adds ios::[open|app] to the parallel.cc to fix write over SELinux
889.55,599.48,4909713991765515b0b6120bd2cc4c3f8092aac6,2017-09-27 12:41:40 +0000,Adds CHANGES info for #1562
890.33667,604.25,495b47d8a21f209b4c297d207bccb4874f89c271,2017-09-21 17:48:42 +0200,Eliminate some reorder and sign warnings
889.54,601.62,ba4e2e3737837a888d1cf414f32658d00c1c5137,2017-09-29 17:18:06 +0000,Adds CHANGES info for #1572
902.22333,593.65667,a5266d6d1c144ccd5fbbba836e46eec502867abd,2017-09-21 17:51:06 +0200,Store the connection and url parameters in std::string
906.69,588.47333,658c9b5daecf80e5509b8cff45bb41f3f91982bf,2017-09-29 16:31:03 +0000,Adds CHANGES info for #1571
897.98667,604.07333,210e72aa213a1a2b8feec705257484f748395ed2,2017-10-06 18:42:32 +0000,Consideres under quote variable while loading the rules
898.01333,607.67667,a76030256ea914ca7bedf1a6636b4a3e3afa8713,2017-08-17 21:03:39 +0300,support macro expansion in @rx
905.4,593.38333,10c4f9b1b2476f71159fa5569d9238001760404c,2017-08-19 10:21:57 +0300,add a test for macro expansion in @rx
903.18,590.3,9e9db08b874fe7c1200aafd95fe6bccd41148ae5,2017-08-19 11:16:54 +0300,add @rx macro expansion test to list in Makefile
673.38,764.12667,fa7973a4ef99b0d91122d16ffee51744288d037f,2017-10-06 20:32:40 +0000,Removes a regex optimization added at #1536
670.78333,764.32667,2988c5bb07c4a5ad434855413f20fec11008c818,2017-10-06 20:35:09 +0000,CHANGES: add info about #1536
674.66333,751.00667,63bef3d142b2ae25ed42d344c40729fb5f3d552e,2017-10-03 20:50:02 +0000,Support to JSON stuff on serial logging
674.18,753.06,d285bc02b87a03e591c0b58f4abdf981c1085d52,2017-10-06 16:58:17 -0400,Add missing statements
675.27667,756.07,e09304a08ae2443acb21632b557e451abcab6c6d,2017-10-09 09:08:31 -0300,CHANGES: Adds info about #1583
671.56333,758.71,d3f979f1d237366fd1f494867ede326ec9a2305c,2017-10-10 09:30:21 -0300,Makes auditlog more verbose on debug logs
670.88667,762.67333,30364628a02b744651160adac8d2e40b00be7e3e,2017-10-10 10:25:22 -0300,Makes clear to the user when audit log is empty due to missing JSON sup.
670.15667,764.44,41bf7f716bb2e3bbb91bc4d7931a52c7e23f66b7,2017-10-10 15:03:50 -0300,Calls xml init and xml cleanup to avoid memory leak
668.83,762.43333,20edf9ab77e4c8016776eb10ffc11e8b6e683133,2017-10-10 18:14:41 -0300,Removes xml initialization from CURL if/def
674.1,759.65667,1ad95254cd8caec4a0af83d01844fc3cc65f3ce7,2017-10-11 12:37:13 -0300,Avoids unicode initialization on every rules block
675.15667,758.09,1518c43d6157e0762c138a49e840bacbbd387e66,2017-10-11 23:18:44 -0300,Adds test case for issue #1565
Reason: https://github.com/SpiderLabs/ModSecurity/commit/fa7973a4ef99b0d91122d16ffee51744288d037f