Update to version v3.2.0

Author: dscpinheiro

.gitignore

@@ -7,4 +7,11 @@
 **/.pyc
 /deployment/open-source/
 source/tests/__pycache__/
-source/log_parser/__pycache__/
+source/log_parser/__pycache__/
+deployment/global-s3-assets/
+deployment/regional-s3-assets/
+
+# coverage
+**/coverage
+**/package
+*coverage*

CHANGELOG.md

@@ -3,30 +3,35 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [2.3.1] - 2019-10-30
+## [3.2] - 2021-09-22
 ### Added
+- Added IP retention support on Allowed and Denied IP Sets
 ### Changed
-- Fixed error handling of intermittent issue: (WAFStaleDataException) when calling the UpdateWebACL
-- Upgrade from Node 8 to Node 10 for Lambda function
-## [2.3.2] - 2020-02-05
+- Bug fixes
+## [3.1] - 2020-10-22
+### Changed
+- Replaced s3 path-style with virtual-hosted style
+- Added partition variable to all ARNs
+- Updated bug report
+## [3.0] - 2020-07-08
 ### Added
+- Added an option to deploy AWS Managed Rules for WebACL on installation
 ### Changed
-- Fixed README file to accurately reflect script params
-- Upgraded from Python 3.7 to 3.8
-- Changed RequestThreshold min limit from 2000 to 100
+- Upgraded from WAF classic to WAFV2 API
+- Eliminated dependency on NodeJS and use Python as the standardized programming language
 ## [2.3.3] - 2020-06-15
 ### Added
 - Implemented Athena optimization: added partitioning for CloudFront, ALB and WAF logs and Athena queries
 ### Changed
 - Fixed potential DoS vector within Bad Bots X-Forward-For header
-## [3.0] - 2020-07-08
+## [2.3.2] - 2020-02-05
 ### Added
-- Added an option to deploy AWS Managed Rules for WebACL on installation
 ### Changed
-- Upgraded from WAF classic to WAFV2 API
-- Eliminated dependency on NodeJS and use Python as the standardized programming language
-## [3.1] - 2020-10-22
+- Fixed README file to accurately reflect script params
+- Upgraded from Python 3.7 to 3.8
+- Changed RequestThreshold min limit from 2000 to 100
+## [2.3.1] - 2019-10-30
+### Added
 ### Changed
-- Replaced s3 path-style with virtual-hosted style
-- Added partition variable to all ARNs 
-- Updated bug report
+- Fixed error handling of intermittent issue: (WAFStaleDataException) when calling the UpdateWebACL
+- Upgrade from Node 8 to Node 10 for Lambda function

NOTICE.txt

@@ -1,6 +1,6 @@
 AWS WAF Security Automations
 
-Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
 Licensed under the Apache License Version 2.0 (the "License"). You may not use this file except
 in compliance with the License. A copy of the License is located at http://www.apache.org/licenses/
 or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS,

README.md

@@ -1,87 +1,153 @@
-# AWS WAF Security Automations
-A solution that contains all AWS WAF samples developed so far - waf-reactive-blacklist, waf-bad-bot-blocking, waf-block-bad-behaving and waf-reputation-lists.
+**[🚀 Solution Landing Page](https://aws.amazon.com/solutions/implementations/aws-waf-security-automations/>)** | **[🚧 Feature request](https://github.com/awslabs/aws-waf-security-automations/issues/new?assignees=&labels=feature-request%2C+enhancement&template=feature_request.md&title=)** | **[🐛 Bug Report](https://github.com/awslabs/aws-waf-security-automations/issues/new?assignees=&labels=bug%2C+triage&template=bug_report.md&title=)**
 
-For the full solution overview visit [AWS WAF Security Automations](https://aws.amazon.com/answers/security/aws-waf-security-automations/).
+Note: If you want to use the solution without building from source, navigate to Solution Landing Page
 
-## File Structure
-This project consists of microservices that facilitate the functional areas of the solution. These microservices are deployed to a serverless environment in AWS Lambda.
+## Table of contents
 
-```
-|-deployment/ [folder containing templates and build scripts]
-|-source/
-  |-access_handler/ [microservice for processing bad bots honeypot endpoint access. This AWS Lambda function intercepts the suspicious request and adds the source IP address to the AWS WAF block list]
-  |-custom_resource/ [custom helper for CloudFormation deployment template]
-  |-helper/ [custom helper for CloudFormation deployment dependency check and auxiliary functions]
-  |-lib/ [library files including waf api calls and other common functions used in the solution]
-  |-log_parser/ [microservice for processing access logs searching for suspicious behavior and add the corresponding source IP addresses to an AWS WAF block list]
-  |-reputation_lists_parser/ [microservice for processing third-party IP reputation lists and add malicious IP addresses to an AWS WAF block list]
-  |-tests/ [unit tests]
-  |-timer/ [creates a sleep function for cloudformation to pace the creation of ip_sets]
-```
+- [Solution Overview](#solution-overview)
+- [Architecture Diagram](#architecture-diagram)
+- [Customizing the Solution](#customizing-the-solution)
+  - [Prerequisites for Customization](#prerequisites-for-customization)
+  - [Build](#build)
+  - [Upload Deployment Assets](#upload-deployment-assets)
+  - [Deploy](#deploy)
+- [File Structure](#file-structure)
+- [License](#license)
+
+<a name="solution-overview"></a>
+# Solution Overview
+
+The AWS DevOps Monitoring Dashboard solution is a reference implementation that automatically deploys a set of AWS WAF (web application firewall) rules that filter common web-based attacks. Users can select from preconfigured protective features that define the rules included in an AWS WAF web access control list (web ACL). Once deployed, AWS WAF protects your Amazon CloudFront distributions or Application Load Balancers by inspecting web requests.
+
+You can use AWS WAF to create custom, application-specific rules that block attack patterns to ensure application availability, secure resources, and prevent excessive resource consumption.
+
+This solution can be easily installed in your AWS accounts via launching the provided AWS CloudFormation template.
+
+For a detailed solution implementation guide, refer to Solution Landing Page [AWS WAF Security Automations](https://aws.amazon.com/solutions/implementations/aws-waf-security-automations)
+
+<a name="architecture-diagram"></a>
+# Architecture Diagram
+
+<p align="center">
+  <img src="source/image/architecture_diagram.png">
+  <br/>
+</p>
+
+AWS WAF Security Automations architecture
+
+AWS Managed Rules (A): This set of AWS managed core rules provides protection against exploitation of a wide range of common application vulnerabilities or other unwanted traffic.
+
+Manual IP lists (B and C): This component creates two specific AWS WAF rules that allow you to manually insert IP addresses that you want to block or allow.
+
+SQL Injection (D) and XSS (E): The solution configures two native AWS WAF rules that are designed to protect against common SQL injection or cross-site scripting (XSS) patterns in the URI, query string, or body of a request.
+
+HTTP flood (F): This component helps protect against attacks that consist of a large number of requests from a particular IP address, such as a web-layer DDoS attacks or a brute-force login attempt. This feature supports thresholds of less than 100 requests within a 5 minute period.
+
+Scanners and Probes (G): This component parses application access logs searching for suspicious behavior, such as an abnormal amount of errors generated by an origin. It then blocks those suspicious source IP addresses for a customer-defined period of time.
+
+IP Reputation Lists (H): This component is the IP Lists Parser AWS Lambda function which checks third-party IP reputation lists hourly for new ranges to block.
+
+Bad Bots (I): This component automatically sets up a honeypot, which is a security mechanism intended to lure and deflect an attempted attack.
 
-## Getting Started
+<a name="customizing-the-solution"></a>
+# Customizing the Solution
 
-#### 01. Prerequisites
-The following procedures assumes that all of the OS-level configuration has been completed. They are:
+<a name="prerequisites-for-customization"></a>
+## Prerequisites for Customization
 
 * [AWS Command Line Interface](https://aws.amazon.com/cli/)
 * Python 3.8
 
-The AWS WAF Security Automations solution is developed with Python for the microservices that run in AWS Lambda. The latest version has been tested with Python v3.8.
+<a name="build"></a>
+## Build
 
-#### 02. Clone AWS WAF Security Automations repository
-Clone the aws-waf-security-automations GitHub repository:
+Building from GitHub source will allow you to modify the solution, such as adding custom actions or upgrading to a new release. The process consists of downloading the source from GitHub, creating Amazon S3 buckets to store artifacts for deployment, building the solution, and uploading the artifacts to S3 in your account.
+
+#### 1. Clone the repository
+
+Clone or download the repository to a local directory on your linux client. Note: if you intend to modify the source code you may wish to create your own fork of the GitHub repo and work from that. This allows you to check in any changes you make to your private copy of the solution.
+
+**Git Clone example:**
 
 ```
 git clone https://github.com/awslabs/aws-waf-security-automations.git
 ```
 
-#### 03. Run unit tests
-Next, run unit tests to make sure added customization passes the tests
+**Download Zip example:**
+```
+wget https://github.com/awslabs/aws-waf-security-automations/archive/master.zip
+```
+
+#### 2. Unit test
+Next, run unit tests to make sure your customized code passes the tests
 
-``` 
-cd ./deployment 
+```
+cd <rootDir>/deployment
 chmod +x ./run-unit-tests.sh
 ./run-unit-tests.sh
-``` 
+```
+
+#### 3. Create S3 buckets for storing deployment assets
+
+AWS Solutions use two buckets:
 
-#### 04. Declare enviroment variables:
+* One global bucket that is access via the http end point. AWS CloudFormation templates are stored here. Ex. "mybucket"
+* One regional bucket for each region where you plan to deploy the solution. Use the name of the global bucket as the prefix of the bucket name, and suffixed with the region name. Regional assets such as Lambda code are stored here. Ex. "mybucket-us-east-1"
+* The assets in buckets must be accessible by your account
+
+#### 4. Declare enviroment variables
 ```
-export TEMPLATE_OUTPUT_BUCKET=<YOUR_TEMPLATE_OUTPUT_BUCKET> # Name for the S3 bucket where the template will be located
-export DIST_OUTPUT_BUCKET=<YOUR_DIST_OUTPUT_BUCKET> # Name for the S3 bucket where customized code will reside 
-export SOLUTION_NAME="aws-waf-security-automations" # name of the solution 
+export TEMPLATE_OUTPUT_BUCKET=<YOUR_TEMPLATE_OUTPUT_BUCKET> # Name of the global bucket where CloudFormation templates are stored
+export DIST_OUTPUT_BUCKET=<YOUR_DIST_OUTPUT_BUCKET> # Name for the regional bucket where regional assets are stored
+export SOLUTION_NAME=<SOLUTION_NAME> # name of the solution.
 export VERSION=<VERSION> # version number for the customized code
-export AWS_REGION=<AWS_REGION> # region where the distributable is deployed
+export AWS_REGION=<AWS_REGION> # region where the solution is deployed
 ```
-#### _Note:_ You must manually create two buckets in S3 called $TEMPLATE_OUTPUT_BUCKET and $DIST_OUTPUT_BUCKET-$AWS_REGION to copy the distribution. The assets in bucket should be publicly accessible. The build-s3-dist.sh script DOES NOT do this and the CloudFormation template expects/references the REGION specific bucket.
-
-#### 05. Build the AWS WAF Security Automations solution for deployment:
+#### 5. Build the solution
 ```
+cd <rootDir>/deployment
 chmod +x ./build-s3-dist.sh && ./build-s3-dist.sh $TEMPLATE_OUTPUT_BUCKET $DIST_OUTPUT_BUCKET $SOLUTION_NAME $VERSION
 ```
-#### 06. Upload deployment assets to your Amazon S3 buckets:
+
+<a name="upload-deployment-assets"></a>
+## Upload deployment assets
 ```
-aws s3 cp ./deployment/global-s3-assets s3://$TEMPLATE_OUTPUT_BUCKET/aws-waf-security-automations/$VERSION --recursive --acl bucket-owner-full-control
-aws s3 cp ./deployment/regional-s3-assets s3://$DIST_OUTPUT_BUCKET-$AWS_REGION/aws-waf-security-automations/$VERSION --recursive --acl bucket-owner-full-control
+aws s3 cp ./deployment/global-s3-assets s3://$TEMPLATE_OUTPUT_BUCKET/$SOLUTION_NAME/$VERSION --recursive --acl bucket-owner-full-control
+aws s3 cp ./deployment/regional-s3-assets s3://$DIST_OUTPUT_BUCKET-$AWS_REGION/$SOLUTION_NAME/$VERSION --recursive --acl bucket-owner-full-control
 ```
-#### _Note:_ You must use proper acl and profile for the copy operation as applicable.
+#### _Note:_ You must use proper acl and profile for the copy operation as applicable. Using randomized bucket names is recommended.
+
+<a name="deploy"></a>
+## Deploy
 
-#### 07. Deploy the AWS WAF Security Automations solution:
 * From your designated Amazon S3 bucket where you uploaded the deployment assets, copy the link location for the aws-waf-security-automations.template.
 * Using AWS CloudFormation, launch the AWS WAF Security Automations solution stack using the copied Amazon S3 link for the aws-waf-security-automations.template.
 
-***
+<a name="file-structure"></a>
+# File structure
+
+This project consists of microservices that facilitate the functional areas of the solution. These microservices are deployed to a serverless environment in AWS Lambda.
+
+<pre>
+|-deployment/ [folder containing templates and build scripts]
+|-source/
+  |-access_handler/         [microservice for processing bad bots honeypot endpoint access. This AWS Lambda function intercepts the suspicious request and adds the source IP address to the AWS WAF block list]
+  |-custom_resource/        [custom helper for CloudFormation deployment template]
+  |-helper/                 [custom helper for CloudFormation deployment dependency check and auxiliary functions]
+  |-image/                  [folder containing images of the solution such as architecture diagram]
+  |-lib/                    [library files including waf api calls and other common functions used in the solution]
+  |-ip_retention_handler/   [lambda code for setting ip retention and removing expired ips]
+  |-log_parser/             [microservice for processing access logs searching for suspicious behavior and add the corresponding source IP addresses to an AWS WAF block list]
+  |-reputation_lists_parser/ [microservice for processing third-party IP reputation lists and add malicious IP addresses to an AWS WAF block list]
+  |-timer/                   [creates a sleep function for cloudformation to pace the creation of ip_sets]
+</pre>
 
-Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+<a name="Collection of operational metrics"></a>
+# Collection of operational metrics
 
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
+This solution collects anonymous operational metrics to help AWS improve the quality and features of the solution. For more information, including how to disable this capability, please see the [implementation guide](https://docs.aws.amazon.com/solutions/latest/aws-waf-security-automations/appendix-g.html).
 
-    http://www.apache.org/licenses/LICENSE-2.0
+<a name="license"></a>
+# License
 
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
+See license [here](https://github.com/awslabs/aws-waf-security-automations/blob/master/LICENSE.txt)

deployment/aws-waf-security-automations-firehose-athena.template

@@ -46,6 +46,8 @@ Parameters:
     Type: String
   DeliveryStreamName:
     Type: String
+  UUID:
+    Type: String
 
 Conditions:
   AlbEndpoint: !Equals
@@ -144,6 +146,8 @@ Resources:
     Properties:
       DeliveryStreamName: !Ref DeliveryStreamName
       DeliveryStreamType: DirectPut
+      DeliveryStreamEncryptionConfigurationInput:
+        KeyType: AWS_OWNED_CMK
       ExtendedS3DestinationConfiguration:
         BucketARN: !Ref WafLogBucketArn
         BufferingHints:
@@ -404,7 +408,7 @@ Resources:
       Type: AWS::Athena::WorkGroup
       Condition: AthenaLogParser
       Properties:
-        Name: WAFAddPartitionAthenaQueryWorkGroup
+        Name: !Join ['-', ['WAFAddPartitionAthenaQueryWorkGroup', !Ref UUID]]
         Description: Athena WorkGroup for adding Athena partition queries used by AWS WAF Security Automations Solution
         State: ENABLED
         RecursiveDeleteOption: true

deployment/aws-waf-security-automations-webacl.template

@@ -191,7 +191,7 @@ Resources:
       Scope: !Sub '${RegionScope}'
       IPAddressVersion: 'IPV4'
       Name: !Sub '${ParentStackName}WhitelistSetIPV4'
-      Description: 'Allow whitelist for IPV4 addresses'
+      Description: 'Allow List for IPV4 addresses'
       Addresses: []
 
   WAFBlacklistSetV4:
@@ -201,7 +201,7 @@ Resources:
       Scope: !Sub '${RegionScope}'
       IPAddressVersion: 'IPV4'
       Name: !Sub '${ParentStackName}BlacklistSetIPV4'
-      Description: 'Block blacklist for IPV4 addresses'
+      Description: 'Block Denied List for IPV4 addresses'
       Addresses: []
 
   WAFHttpFloodSetV4:
@@ -259,7 +259,7 @@ Resources:
       Scope: !Sub '${RegionScope}'
       IPAddressVersion: IPV6
       Name: !Sub '${ParentStackName}WhitelistSetIPV6'
-      Description: 'Allow whitelist for IPV6 addresses'
+      Description: 'Allow list for IPV6 addresses'
       Addresses: []
 
   WAFBlacklistSetV6:
@@ -269,7 +269,7 @@ Resources:
       Scope: !Sub '${RegionScope}'
       IPAddressVersion: IPV6
       Name: !Sub '${ParentStackName}BlacklistSetIPV6'
-      Description: 'Block blacklist for IPV6 addresses'
+      Description: 'Block Denied List for IPV6 addresses'
       Addresses: []
 
   WAFHttpFloodSetV6:
@@ -356,6 +356,13 @@ Resources:
         Variables:
           SECONDS: '2'
           LOG_LEVEL: !Ref LogLevel
+    Metadata:
+      cfn_nag:
+        rules_to_suppress:
+        - id: W89
+          reason: There is no need to run this lambda in a VPC
+        - id: W92
+          reason: There is no need for Reserved Concurrency
 
 # Adding a (priority 0) rule for AWS Managed RuleSet, optionally triggered by params