Merge branch 'v3.2/dev' into v3.3/dev

Author: lifeforms

CHANGES

@@ -5,7 +5,7 @@
   or the CRS mailinglist at
 * https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set
 
-== Version 3.2.0 - 9/20/2019 ==
+== Version 3.2.0 - 9/24/2019 ==
 
 New functionality:
  * Add AngularJS client side template injection 941380 PL2 (Franziska Bühler)
@@ -14,6 +14,7 @@ New functionality:
  * Add libinjection check on last path segment (Max Leske, Christian Folini)
  * Add PUBLIC identifier for XML entities (#1490) (Rufus125)
  * Add .rdb to default restricted_extensions (Walter Hop)
+ * Add .swp to default restricted_extensions (Andrea Menin)
  * Add rule 933200 PHP Wrappers (Andrea Menin)
  * Add send-payload-pls.sh script to test payload against multiple paranoia levels (Christian Folini)
  * Add support for shell evasions with $IFS (Walter Hop, Chaim Sanders)
@@ -50,19 +51,23 @@ Improved compatibility:
 Fixes and improvements:
  * 932140: fix ReDoS in FOR expression (Walter Hop)
  * 933200: Simplify pattern (Federico G. Schwindt, Andrea Menin)
+ * 941380: fix anomaly score variable (Franziska Bühler)
+ * 942510, 942511: fix anomaly score variable (Walter Hop)
  * Add content-type application/csp-report (Andrea Menin)
  * Add content-type application/xss-auditor-report (Andrea Menin)
  * Add CRS 3.2 Badge build support. (Chaim Sanders)
+ * Add CVE numbers for Apache Struts vulnerabilities to comments in rules (Franziska Bühler)
  * Add CVE-2018-11776 to comments of 933160 and 933161 (Franziska Bühler)
  * Add CVE-2018-2380 to comments of rules (Franziska Bühler)
- * Add CVE numbers for Apache Struts vulnerabilities to comments in rules (Franziska Bühler)
  * Add default env vars for anomaly scores in Docker (Franziska Bühler)
- * Added spaces in front of closing square brackets (Franziska Bühler)
- * Adding travis changes (#1316) (Chaim Sanders)
  * Add missing OWASP_CRS tags to 921xxx rules (Walter Hop)
  * Add REQUEST_FILENAME to rule id 944130 and add exploits to comment (Franziska Bühler)
+ * Add spaces in front of closing square brackets (Franziska Bühler)
+ * Add travis changes (#1316) (Chaim Sanders)
  * Allow dot characters in Content-Type multipart boundary (Walter Hop)
  * Also handle dot variant of X_Filename. PHP will transform dots to underscore in variable names since dot is invalid. (Federico G. Schwindt)
+ * As per the ref manual, it is compressWhitespace (Federico G. Schwindt)
+ * Avoid php leak false positive with WOFF files (Manuel Spartan)
  * Bring back CRS 2.x renumbering utility (Walter Hop)
  * Clean up travis and reorg (Federico G. Schwindt)
  * Code cosmetics: reorder the actions of rules (Ervin Hegedus)
@@ -96,6 +101,8 @@ Fixes and improvements:
  * Fix Travis Merge not being able to find HEAD (Chaim Sanders)
  * Fix vulnerable regexp in rule 942490 (CVE-2019-11387) (Christoph Hansen)
  * Fix wrong regex, assembly result, in 942370 (Franziska Bühler)
+ * INSTALL: advise to use release zips, remove upgrade.py, update Nginx (Walter Hop)
+ * Java: change tag from COMMAND_INJECTION to JAVA_INJECTION (Manuel Spartan)
  * Jwall auditconsole outbound anomaly scoring requirements (Christoph Hansen)
  * Mark patterns not supported by re2 (Federico G. Schwindt)
  * Move duplicated 900270 to 900280 Fixes #1236. (Federico G. Schwindt)
@@ -117,11 +124,11 @@ Fixes and improvements:
  * SQLI: removed unnecessary + (Christoph Hansen)
  * Switch Docker image to owasp/modsecurity:2.9-apache-ubuntu (Federico G. Schwindt)
  * unix-shell.data: fix typo in 'more' (Walter Hop)
+ * Update .travis.yml Update to support v3.1 (Chaim Sanders)
  * Update dockerfile to always use 3.2/dev (Federico G. Schwindt)
  * Update OWASP CRS Docker image to support the new upstream and 2.9.3 (Peter Bittner, Chaim Sanders)
  * Update RESPONSE-950-DATA-LEAKAGES.conf (Christoph Hansen)
  * Update RESPONSE-959-BLOCKING-EVALUATION.conf (Christoph Hansen)
- * Update .travis.yml Update to support v3.1 (Chaim Sanders)
  * Wordpress: add support for Gutenberg editor (siric_, Walter Hop)
  * Wordpress: allow searching for any term in admin posts/pages overview (Walter Hop)
  * WordPress: exclude Gutenberg via rest_route (Walter Hop)
@@ -133,7 +140,6 @@ Unit tests:
  * 932140: add regression tests (Walter Hop)
  * 933180: fix tests which were doing nothing (Walter Hop)
  * 941370: add some more tests, fix whitespace (Walter Hop)
- * Added regression tests for rules 942320, 942360, 942361, 942210, 942380, 942410, 942470, 942120, 942240, 942160, 942190, 942140, 942490, 942120 (Christoph Hansen)
  * Add more tests for 941130 (Christian Folini)
  * Add regression test for 941101 (Avery Wong)
  * Add regression tests for 942150, 942100, 942260 (Christian Folini)
@@ -142,10 +148,13 @@ Unit tests:
  * Add testing support for libmodsecurity running on Apache and Nginx (Chaim Sanders)
  * Add tests for 941360 that fights JSFuck and Hieroglyphy (Christian Folini)
  * Add tests for rule 921110 (Yu Yagihashi)
+ * Added regression tests for rules 942320, 942360, 942361, 942210, 942380, 942410, 942470, 942120, 942240, 942160, 942190, 942140, 942490, 942120 (Christoph Hansen)
  * Drop tests for removed rules (Federico G. Schwindt)
+ * Fix failing regression tests (Ervin Hegedus)
  * Fix failing tests (Manuel Spartan, Chaim Sanders)
  * Fix readme typos in example rule (Walter Hop)
  * Fix test 941110-2 (Federico G. Schwindt)
+ * Fix YAML 1.2 compliance with "true" (Federico G. Schwindt)
  * RCE: Add tests for the for command (Federico G. Schwindt)
  * Update regression tests for rules 931110, 931120, 931130 (Simon Studer)
 

util/docker/docker-compose.yaml

@@ -74,7 +74,7 @@ services:
       - ./RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf:/etc/modsecurity.d/owasp-crs/rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf
 
     #######################################################
-    # Add TLS server certificate and key 
+    # Add TLS server certificate and key
     # (only available if SETPROXY was enabled during the
     # parent ModSecurity image)
     #######################################################

util/regression-tests/tests/REQUEST-911-METHOD-ENFORCEMENT/911100.yaml

@@ -1,56 +1,56 @@
 ---
-  meta: 
+  meta:
     author: "csanders-git"
     enabled: true
     name: "911100.yaml"
     description: "Description"
-  tests: 
-    - 
+  tests:
+    -
       test_title: 911100-1
-      stages: 
-        - 
-          stage: 
+      stages:
+        -
+          stage:
             input:
               dest_addr: "127.0.0.1"
               port: 80
               headers:
                   User-Agent: "ModSecurity CRS 3 Tests"
                   Host: "localhost"
-            output: 
+            output:
               no_log_contains: "id \"911100\""
-    - 
+    -
       test_title: 911100-2
-      stages: 
-        - 
-          stage: 
+      stages:
+        -
+          stage:
             input:
               dest_addr: "127.0.0.1"
               port: 80
               method: "OPTIONS"
               headers:
                   User-Agent: "ModSecurity CRS 3 Tests"
                   Host: "localhost"
-            output: 
+            output:
               no_log_contains: "id \"911100\""
-    - 
+    -
       test_title: 911100-3
-      stages: 
-        - 
-          stage: 
+      stages:
+        -
+          stage:
             input:
               dest_addr: "127.0.0.1"
               method: "HEAD"
               port: 80
               headers:
                   User-Agent: "ModSecurity CRS 3 Tests"
                   Host: "localhost"
-            output: 
-              no_log_contains: "id \"911100\""                                          
-    - 
+            output:
+              no_log_contains: "id \"911100\""
+    -
       test_title: 911100-4
-      stages: 
-        - 
-          stage: 
+      stages:
+        -
+          stage:
             input:
               dest_addr: "127.0.0.1"
               method: "POST"
@@ -60,27 +60,27 @@
                   Host: "localhost"
                   Content-Type: "application/x-www-form-urlencoded"
               data: "test=value"
-            output: 
-              no_log_contains: "id \"911100\""  
-    - 
+            output:
+              no_log_contains: "id \"911100\""
+    -
       test_title: 911100-5
-      stages: 
-        - 
-          stage: 
+      stages:
+        -
+          stage:
             input:
               dest_addr: "127.0.0.1"
               method: "TEST"
               port: 80
               headers:
                   User-Agent: "ModSecurity CRS 3 Tests"
                   Host: "localhost"
-            output: 
-              log_contains: "id \"911100\""                
-    - 
+            output:
+              log_contains: "id \"911100\""
+    -
       test_title: 911100-6
       desc: Method is not allowed by policy (911100) from old modsec regressions
       stages:
-      - 
+      -
         stage:
           input:
             dest_addr: 127.0.0.1
@@ -100,11 +100,11 @@
           output:
             log_contains: id "911100"
 
-    - 
+    -
       test_title: 911100-7
       desc: Method is not allowed by policy (911100) from old modsec regressions
       stages:
-      - 
+      -
         stage:
           input:
             dest_addr: 127.0.0.1
@@ -124,11 +124,11 @@
           output:
             log_contains: id "911100"
 
-    - 
+    -
       test_title: 911100-8
       desc: Method is not allowed by policy (911100) from old modsec regressions
       stages:
-      - 
+      -
         stage:
           input:
             dest_addr: 127.0.0.1
@@ -146,4 +146,4 @@
             uri: /
             version: HTTP/1.0
           output:
-            log_contains: id "911100"              
+            log_contains: id "911100"

util/regression-tests/tests/REQUEST-913-SCANNER-DETECTION/913100.yaml

@@ -5,11 +5,11 @@
     enabled: true
     name: 913100.yaml
   tests:
-  - 
+  -
     test_title: 913100-1
     desc: Request Indicates a Security Scanner Scanned the Site (913100) from old modsec regressions
     stages:
-    - 
+    -
       stage:
         input:
           dest_addr: 127.0.0.1
@@ -29,11 +29,11 @@
           version: HTTP/1.0
         output:
           log_contains: id "913100"
-  - 
+  -
     test_title: 913100-2
     desc: Request Indicates a Security Scanner Scanned the Site (913100) from old modsec regressions
     stages:
-    - 
+    -
       stage:
         input:
           dest_addr: 127.0.0.1
@@ -53,11 +53,11 @@
         output:
           log_contains: id "913100"
 
-  - 
+  -
     test_title: 913100-3
     desc: Request Indicates a Security Scanner Scanned the Site (913100) from old modsec regressions
     stages:
-    - 
+    -
       stage:
         input:
           dest_addr: 127.0.0.1

util/regression-tests/tests/REQUEST-913-SCANNER-DETECTION/913110.yaml

@@ -5,12 +5,12 @@
     enabled: true
     name: 913110.yaml
   tests:
-  - 
+  -
     test_title: 913110-1
     desc: Request Indicates a Security Scanner Scanned the Site (913110) from old modsec
       regressions
     stages:
-    - 
+    -
       stage:
         input:
           dest_addr: 127.0.0.1

util/regression-tests/tests/REQUEST-913-SCANNER-DETECTION/913120.yaml

@@ -5,12 +5,12 @@
     enabled: true
     name: 913120.yaml
   tests:
-  - 
+  -
     test_title: 913120-1
     desc: Request Indicates a Security Scanner Scanned the Site (913120) from old modsec
       regressions
     stages:
-    - 
+    -
       stage:
         input:
           dest_addr: 127.0.0.1
@@ -44,7 +44,7 @@
           uri: /AppScan_fingerprint/MAC_ADDRESS_01234567890.html?9ABCDG1
           version: HTTP/1.0
         output:
-          log_contains: id "913120"          
+          log_contains: id "913120"
   -
     test_title: 913120-3
     desc: "Scanner identification based on uri"