Updated to v2.3.3

Author: Joshua Leaverton

CHANGELOG.md

@@ -14,3 +14,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Fixed README file to accurately reflect script params
 - Upgraded from Python 3.7 to 3.8
 - Changed RequestThreshold min limit from 2000 to 100
+## [2.3.3] - 2020-05-15
+### Added
+- Implemented Athena optimization: added partitioning for CloudFront, ALB and WAF logs and Athena queries
+### Changed
+- Fixed potential DoS vector within Bad Bots X-Forward-For header

NOTICE.txt

@@ -1,6 +1,6 @@
 AWS WAF Security Automations
 
-Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
 Licensed under the Apache License Version 2.0 (the "License"). You may not use this file except
 in compliance with the License. A copy of the License is located at http://www.apache.org/licenses/
 or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS,
@@ -17,3 +17,4 @@ sax under the Internet Systems Consortium (ISC) license
 xml2js under the Massachusetts Institute of Technology (MIT) license
 xmlbuilder under the Massachusetts Institute of Technology (MIT) license
 requests under the Apache Software License
+freezegun under the Apache License Version 2.0

README.md

@@ -3,8 +3,6 @@ A solution that contains all AWS WAF samples developed so far - waf-reactive-bla
 
 For the full solution overview visit [AWS WAF Security Automations](https://aws.amazon.com/answers/security/aws-waf-security-automations/).
 
->Be aware that using the ***Athena log parser*** option with the HTTP Flood Protection and/or Scanner and Probe Protection rules may result in significant Athena charges. The log parser runs a scheduled query against your S3 log bucket every 5 minutes. Charges are based on the amount of logs Athena must scan as part of this query."
-
 ## File Structure
 This project consists of microservices that facilitate the functional areas of the solution. These microservices are deployed to a serverless environment in AWS Lambda.
 
@@ -14,8 +12,9 @@ This project consists of microservices that facilitate the functional areas of t
   |-access-handler/ [microservice for processing bad bots honeypot endpoint access. This AWS Lambda function intercepts the suspicious request and adds the source IP address to the AWS WAF block list]
   |-custom-resource/ [custom helper for CloudFormation deployment template]
   |-helper/ [custom helper for CloudFormation deployment dependency check and auxiliary functions]
-  |-log-parser/ [microservice for processing access logs searching for suspicious behavior and add the corresponding source IP addresses to an AWS WAF block list]
+  |-log_parser/ [microservice for processing access logs searching for suspicious behavior and add the corresponding source IP addresses to an AWS WAF block list]
   |-reputation-lists-parser/ [microservice for processing third-party IP reputation lists and add malicious IP addresses to an AWS WAF block list]
+  |-tests/ [unit tests]
 ```
 
 ## Getting Started
@@ -30,40 +29,49 @@ The following procedures assumes that all of the OS-level configuration has been
 The AWS WAF Security Automations solution is developed with Node.js and Python for the microservices that run in AWS Lambda. The latest version has been tested with Node.js v10.x and Python v3.8.
 
 #### 02. Clone AWS WAF Security Automations repository
-Clone the aws-waf-security-automations GitHub repository:
+Clone the aws-waf-security-automations GitHub repository, then make the desired code changes
 
 ```
 git clone https://github.com/awslabs/aws-waf-security-automations.git
 ```
 
-#### 03. Declare enviroment variables:
+#### 03. Run unit tests
+Next, run unit tests to make sure added customization passes the tests
+
+``` 
+cd ./deployment 
+chmod +x ./run-unit-tests.sh
+./run-unit-tests.sh
+``` 
+
+#### 04. Declare enviroment variables:
 ```
-export TEMPLATE_OUTPUT_BUCKET=<YOUR_TEMPLATE_OUTPUT_BUCKET>
-export DIST_OUTPUT_BUCKET=<YOUR_DIST_OUTPUT_BUCKET>
-export SOLUTION_NAME="workspaces-cost-optimizer"
-export VERSION=<VERSION>
-export AWS_REGION=<AWS_REGION>
+export TEMPLATE_OUTPUT_BUCKET=<YOUR_TEMPLATE_OUTPUT_BUCKET> # Name for the S3 bucket where the template will be located
+export DIST_OUTPUT_BUCKET=<YOUR_DIST_OUTPUT_BUCKET> # Name for the S3 bucket where customized code will reside 
+export SOLUTION_NAME="aws-waf-security-automations" # name of the solution 
+export VERSION=<VERSION> # version number for the customized code
+export AWS_REGION=<AWS_REGION> # region where the distributable is deployed
 ```
+# _Note:_ You must manually create two buckets in S3 called $TEMPLATE_OUTPUT_BUCKET and $DIST_OUTPUT_BUCKET-$AWS_REGION to copy the distribution. The assets in bucket should be publicly accessible. The build-s3-dist.sh script DOES NOT do this and the CloudFormation template expects/references the REGION specific bucket.
 
-#### 04. Build the AWS WAF Security Automations solution for deployment:
+#### 05. Build the AWS WAF Security Automations solution for deployment:
 ```
 chmod +x ./build-s3-dist.sh && ./build-s3-dist.sh $TEMPLATE_OUTPUT_BUCKET $DIST_OUTPUT_BUCKET $SOLUTION_NAME $VERSION
 ```
-#### 05. Upload deployment assets to your Amazon S3 bucket:
-
-Note that you must manually create a bucket in S3 called `$DIST_OUTPUT_BUCKET-$AWS_REGION` to copy the distribution. The build-s3-dist.sh script DOES NOT do this and the CloudFormation template expects/references the REGION specific bucket.
-
+#### 06. Upload deployment assets to your Amazon S3 buckets:
 ```
-aws s3 cp ./dist s3://$DIST_OUTPUT_BUCKET-$AWS_REGION/aws-waf-security-automations/latest --recursive --acl bucket-owner-full-control
+aws s3 cp ./deployment/global-s3-assets s3://$TEMPLATE_OUTPUT_BUCKET/aws-waf-security-automations/$VERSION --recursive --acl bucket-owner-full-control
+aws s3 cp ./deployment/regional-s3-assets s3://$DIST_OUTPUT_BUCKET-$AWS_REGION/aws-waf-security-automations/$VERSION --recursive --acl bucket-owner-full-control
 ```
+# _Note:_ You must use proper acl and profile for the copy operation as applicable.
 
-#### 06. Deploy the AWS WAF Security Automations solution:
+#### 07. Deploy the AWS WAF Security Automations solution:
 * From your designated Amazon S3 bucket where you uploaded the deployment assets, copy the link location for the aws-waf-security-automations.template.
 * Using AWS CloudFormation, launch the AWS WAF Security Automations solution stack using the copied Amazon S3 link for the aws-waf-security-automations.template.
 
 ***
 
-Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.

deployment/aws-waf-security-automations-alb.template

@@ -1,4 +1,4 @@
-# Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License").
 # You may not use this file except in compliance with the License.
@@ -96,6 +96,13 @@ Conditions:
     - !Ref ActivateBadBotProtectionParam
     - 'yes'
 
+  AthenaLogParser: !Or
+    - Condition: HttpFloodAthenaLogParser
+    - Condition: ScannersProbesAthenaLogParser
+
+  CustomResourceLambdaAccess: !Or
+    - Condition: ReputationListsProtectionActivated
+    - Condition: AthenaLogParser
 
 Resources:
   WAFWhitelistSet:
@@ -359,7 +366,7 @@ Resources:
                     - 'athena:GetNamedQuery'
                     - 'athena:StartQueryExecution'
                   Resource:
-                    - !Sub 'arn:aws:athena:${AWS::Region}:${AWS::AccountId}:workgroup/*'
+                    - !Sub 'arn:aws:athena:${AWS::Region}:${AWS::AccountId}:workgroup/WAF*'
                 # S3 Resources
                 - Effect: Allow
                   Action:
@@ -378,6 +385,7 @@ Resources:
                 - Effect: Allow
                   Action:
                     - 'glue:GetTable'
+                    - 'glue:GetPartitions'
                   Resource:
                     - !Sub 'arn:aws:glue:${AWS::Region}:${AWS::AccountId}:catalog'
                     - !Sub 'arn:aws:glue:${AWS::Region}:${AWS::AccountId}:database/${GlueAccessLogsDatabase}'
@@ -418,7 +426,7 @@ Resources:
                     - 'athena:GetNamedQuery'
                     - 'athena:StartQueryExecution'
                   Resource:
-                    - !Sub 'arn:aws:athena:${AWS::Region}:${AWS::AccountId}:workgroup/*'
+                    - !Sub 'arn:aws:athena:${AWS::Region}:${AWS::AccountId}:workgroup/WAF*'
                 # S3 Resources
                 - Effect: Allow
                   Action:
@@ -437,6 +445,7 @@ Resources:
                 - Effect: Allow
                   Action:
                     - 'glue:GetTable'
+                    - 'glue:GetPartitions'
                   Resource:
                     - !Sub 'arn:aws:glue:${AWS::Region}:${AWS::AccountId}:catalog'
                     - !Sub 'arn:aws:glue:${AWS::Region}:${AWS::AccountId}:database/${GlueAccessLogsDatabase}'
@@ -476,6 +485,164 @@ Resources:
               LogsAccess - permission restricted to account, region and log group name substring (LogParser);
               CloudWatchAccess - this actions does not support resource-level permissions
 
+  LambdaRolePartitionS3Logs:
+    Type: 'AWS::IAM::Role'
+    Condition: ScannersProbesAthenaLogParser
+    Metadata:
+      cfn_nag:
+        rules_to_suppress:
+          -
+            id: W11
+            reason: >-
+              LogsAccess - permission restricted to account, region and log group name substring (MoveS3LogsForPartition)
+    Properties:
+      AssumeRolePolicyDocument:
+        Statement:
+          - Effect: Allow
+            Principal:
+              Service:
+                - lambda.amazonaws.com
+            Action:
+              - sts:AssumeRole
+      Path: '/'
+      Policies:
+        - !If
+          - ScannersProbesAthenaLogParser
+          - PolicyName: PartitionS3LogsAccess
+            PolicyDocument:
+              Statement:
+                # S3 Resources
+                - Effect: Allow
+                  Action:
+                    - 's3:GetObject'
+                    - 's3:DeleteObject'
+                    - 's3:PutObject'
+                  Resource:
+                    - !Sub 'arn:aws:s3:::${AppAccessLogBucket}/*'
+          - !Ref 'AWS::NoValue'
+        - PolicyName: LogsAccess
+          PolicyDocument:
+            Statement:
+              - Effect: Allow
+                Action:
+                  - 'logs:CreateLogGroup'
+                  - 'logs:CreateLogStream'
+                  - 'logs:PutLogEvents'
+                Resource:
+                  - !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*MoveS3LogsForPartition*'
+
+  LambdaRoleAddAthenaPartitions:
+    Type: 'AWS::IAM::Role'
+    Condition: AthenaLogParser
+    Metadata:
+      cfn_nag:
+        rules_to_suppress:
+          -
+            id: W11
+            reason: >-
+              LogsAccess - permission restricted to account, region and log group name substring (AddAthenaPartitions)
+    Properties:
+      AssumeRolePolicyDocument:
+        Statement:
+          - Effect: Allow
+            Principal:
+              Service:
+                - lambda.amazonaws.com
+            Action:
+              - sts:AssumeRole
+      Path: '/'
+      Policies:
+        - !If
+          - ScannersProbesAthenaLogParser
+          - PolicyName: AddAthenaPartitionsForAppAccessLog
+            PolicyDocument:
+              Statement:
+                # S3 Resources
+                - Effect: Allow
+                  Action:
+                    - 's3:GetObject'
+                    - 's3:PutObject'
+                    - 's3:GetBucketLocation'
+                    - 's3:ListBucket'
+                    - 's3:ListBucketMultipartUploads'
+                    - 's3:ListMultipartUploadParts'
+                    - 's3:AbortMultipartUpload'
+                    - 's3:CreateBucket'
+                  Resource:
+                    - !Sub 'arn:aws:s3:::${AppAccessLogBucket}/athena_results/*'
+                    - !Sub 'arn:aws:s3:::${AppAccessLogBucket}'
+                    - !Sub 'arn:aws:s3:::${AppAccessLogBucket}/*'
+                # Athena Resources
+                - Effect: Allow
+                  Action:
+                    - 'athena:StartQueryExecution'
+                  Resource:
+                    - !Sub 'arn:aws:athena:${AWS::Region}:${AWS::AccountId}:workgroup/WAF*'
+                # Glue Resources
+                - Effect: Allow
+                  Action:
+                    - 'glue:GetTable'
+                    - 'glue:GetDatabase'
+                    - 'glue:UpdateDatabase'
+                    - 'glue:CreateDatabase'
+                    - 'glue:BatchCreatePartition'
+                  Resource:
+                    - !Sub 'arn:aws:glue:${AWS::Region}:${AWS::AccountId}:catalog'
+                    - !Sub 'arn:aws:glue:${AWS::Region}:${AWS::AccountId}:database/default'
+                    - !Sub 'arn:aws:glue:${AWS::Region}:${AWS::AccountId}:database/${GlueAccessLogsDatabase}'
+                    - !Sub 'arn:aws:glue:${AWS::Region}:${AWS::AccountId}:table/${GlueAccessLogsDatabase}/${GlueAppAccessLogsTable}'
+          - !Ref 'AWS::NoValue'
+        - !If
+          - HttpFloodAthenaLogParser
+          - PolicyName: AddAthenaPartitionsForWAFLog
+            PolicyDocument:
+              Statement:
+                # S3 Resources
+                - Effect: Allow
+                  Action:
+                    - 's3:GetObject'
+                    - 's3:PutObject'
+                    - 's3:GetBucketLocation'
+                    - 's3:ListBucket'
+                    - 's3:ListBucketMultipartUploads'
+                    - 's3:ListMultipartUploadParts'
+                    - 's3:AbortMultipartUpload'
+                    - 's3:CreateBucket'
+                  Resource:
+                    - !Sub 'arn:aws:s3:::${WafLogBucket}/athena_results/*'
+                    - !Sub 'arn:aws:s3:::${WafLogBucket}'
+                    - !Sub 'arn:aws:s3:::${WafLogBucket}/*'
+                # Athena Resources
+                - Effect: Allow
+                  Action:
+                    - 'athena:StartQueryExecution'
+                  Resource:
+                    - !Sub 'arn:aws:athena:${AWS::Region}:${AWS::AccountId}:workgroup/WAF*'
+                # Glue Resources
+                - Effect: Allow
+                  Action:
+                    - 'glue:GetTable'
+                    - 'glue:GetDatabase'
+                    - 'glue:UpdateDatabase'
+                    - 'glue:CreateDatabase'
+                    - 'glue:BatchCreatePartition'
+                  Resource:
+                    - !Sub 'arn:aws:glue:${AWS::Region}:${AWS::AccountId}:catalog'
+                    - !Sub 'arn:aws:glue:${AWS::Region}:${AWS::AccountId}:database/default'
+                    - !Sub 'arn:aws:glue:${AWS::Region}:${AWS::AccountId}:database/${GlueAccessLogsDatabase}'
+                    - !Sub 'arn:aws:glue:${AWS::Region}:${AWS::AccountId}:table/${GlueAccessLogsDatabase}/${GlueWafAccessLogsTable}'
+          - !Ref 'AWS::NoValue'
+        - PolicyName: LogsAccess
+          PolicyDocument:
+            Statement:
+              - Effect: Allow
+                Action:
+                  - 'logs:CreateLogGroup'
+                  - 'logs:CreateLogStream'
+                  - 'logs:PutLogEvents'
+                Resource:
+                  - !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*AddAthenaPartitions*'
+
   LambdaRoleReputationListsParser:
     Type: 'AWS::IAM::Role'
     Condition: ReputationListsProtectionActivated
@@ -625,6 +792,7 @@ Resources:
                   - 's3:CreateBucket'
                   - 's3:GetBucketNotification'
                   - 's3:PutBucketNotification'
+                  - 's3:PutEncryptionConfiguration'
                 Resource:
                   - !Sub 'arn:aws:s3:::${AppAccessLogBucket}'
         - !If
@@ -671,7 +839,7 @@ Resources:
                     - !Sub 'arn:aws:s3:::${WafLogBucket}/${ParentStackName}-waf_log_conf.json'
           - !Ref 'AWS::NoValue'
         - !If
-          - ReputationListsProtectionActivated
+          - CustomResourceLambdaAccess
           - PolicyName: LambdaAccess
             PolicyDocument:
               Statement:
@@ -853,3 +1021,11 @@ Outputs:
 
   LambdaRoleCustomResourceArn:
     Value: !GetAtt LambdaRoleCustomResource.Arn
+
+  LambdaRolePartitionS3LogsArn:
+    Value: !GetAtt LambdaRolePartitionS3Logs.Arn
+    Condition: ScannersProbesAthenaLogParser
+
+  LambdaRoleAddAthenaPartitionsArn:
+    Value: !GetAtt LambdaRoleAddAthenaPartitions.Arn
+    Condition: AthenaLogParser