Update to version v1.5.0

Author: aassadza

.gitignore

@@ -14,6 +14,7 @@ node_modules
 cdk.out
 
 !**/cdk-solution-helper/index.js
+!get-cdk-version.js
 
 deployment/global-s3-assets
 deployment/regional-s3-assets

CHANGELOG.md

@@ -5,6 +5,24 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.5.0] - 2021-07-22
+
+### Added
+
+-   Ingest RSS feeds from over ~3000+ news websites across the world
+
+### Updated
+
+-   AWS CDK version to 1.110.1
+-   AWS SDK version to 2.945.0
+-   Updated Nodejs Lambda runtimes to use Nodejs 14.x
+-   Updated Amazon QuickSight analysis and dashboard to reflect the new ingestion source
+-   Updated AWS StepFunction workflows to handle parallel ingestion (tweets from Twitter and RSS feeds from News websites)
+
+### Fixed
+
+-   Truncated tweets through merging [GitHub pull request #26](https://github.com/awslabs/discovering-hot-topics-using-machine-learning/pull/26)
+
 ## [1.4.0] - 2021-02-04
 
 ### Added

NOTICE.txt

@@ -28,6 +28,7 @@ crhelper - Apache-2.0
 jest - MIT license
 momentjs - MIT license
 moto - Apache-2.0
+newscatcher - MIT license
 nock - MIT license
 node - MIT license
 pytest-cov - MIT license

README.md

@@ -2,7 +2,7 @@
 
 The Discovering Hot Topics Using Machine Learning solution helps you identify the most dominant topics associated with your products, policies, events, and brands. Implementing this solution helps you react quickly to new growth opportunities, address negative brand associations, and deliver higher levels of customer satisfaction.
 
-The solution uses machine learning algorithms to automate digital asset (text and image) ingestion and perform near real-time topic modeling, sentiment analysis, and image detection. The solution then visualizes these large-scale customer analyses using an Amazon QuickSight dashboard. This guide provides step-by-step instructions to building a dashboard that provides you with the context and insights necessary to identify trends that help or harm you brand.
+The solution automates digital asset (text and image) ingestion from twitter and RSS news feeds to provide near-real-time inferences using machine learning algorithms through Amazon Comprehend, Amazon Translate, and Amazon Rekognition to perform topic modeling, sentiment analysis, entity and key phrase detection, and detect any unsafe images. The solution then visualizes these large-scale customer analyses using an Amazon QuickSight dashboard. This guide provides step-by-step instructions for deploying this solution including a pre-built dashboard that provides you with the context and insights necessary to identify trends that help or harm your brand.
 
 The solution performs the following key features:
 
@@ -11,6 +11,8 @@ The solution performs the following key features:
 -   **Determines if images associated with your brand contain unsafe content**: detects unsafe and negative imagery in content
 -   **Helps customers identify insights in near real-time**: you can use a visualization dashboard to better understand context, threats, and opportunities almost instantly
 
+This solution deploys an AWS CloudFormation template that supports both Twitter and RSS feeds as data source options for ingestion, but the solution can be customized to aggregate other social media platforms and internal enterprise systems.
+
 For a detailed solution deployment guide, refer to [Discovering Hot Topics using Machine Learning](https://aws.amazon.com/solutions/implementations/discovering-hot-topics-using-machine-learning)
 
 ## On this Page
@@ -52,6 +54,7 @@ After you deploy the solution, use the included Amazon QuickSight dashboard to v
 -   aws-lambda-dynamodb
 -   aws-lambda-s3
 -   aws-lambda-step-function
+-   aws-sqs-lambda
 
 ## Deployment
 
@@ -64,26 +67,37 @@ The solution is deployed using a CloudFormation template with a lambda backed cu
 ```
 ├── deployment                          [folder containing build scripts]
 │   ├── cdk-solution-helper             [A helper function to help deploy lambda function code through S3 buckets]
+│   ├── build-s3-dist.sh                [Build script to build the solution]
 └── source                              [source code containing CDK App and lambda functions]
     ├── bin                             [entrypoint of the CDK application]
     ├── lambda                          [folder containing source code the lambda functions]
-    │   ├── firehose-text-proxy         [lambda function to write text analysis output to Amazon Kinesis Firehose]
+    │   ├── capture_news_feed           [lambda function to ingest news feeds]
     │   ├── firehose_topic_proxy        [lambda function to write topic analysis output to Amazon Kinesis Firehose]
+    │   ├── firehose-text-proxy         [lambda function to write text analysis output to Amazon Kinesis Firehose]
     │   ├── ingestion-consumer          [lambda function that consumes messages from Amazon Kinesis Data Stream]
     │   ├── ingestion-producer          [lambda function that makes Twitter API call and pushes data to Amazon Kinesis Data Stream]
     │   ├── integration                 [lambda function that publishes inference outputs to Amazon Events Bridge]
+    │   ├── layers                      [lambda layer function library]
+    │   │   ├── aws-nodesk-custom-config
+    │   ├── quicksight-custom-resources [lambda function to create Amazon QuickSight resources, example: data source, data sets, analysis and dashboards]
+    │   ├── shared                      [lambda layer function library (specific to python lambda runtimes)]
+    │   ├── solution_helper             [lambda function that allows capturing metrics for this solution]
     │   ├── storage-firehose-processor  [lambda function that writes data to S3 buckets to build a relational model]
     │   ├── wf-analyze-text             [lambda function to detect sentiments, key phrases and entities using Amazon Comprehend]
     │   ├── wf-check-topic-model        [lambda function to check status of topic modeling jobs on Amazon Comprehend]
+    │   ├── wf-detect-language          [lambda function to detect language of ingested text content using Amazon Comprehend]
     │   ├── wf-detect-moderation-labels [lambda function to detect content moderation using Amazon Rekognition]
     │   ├── wf-extract-text-in-image    [lambda function to extract text content from images using Amazon Rekognition]
     │   ├── wf-publish-text-inference   [lambda function to publish Amazon Comprehend inferences]
     │   ├── wf-submit-topic-model       [lambda function to submit topic modeling job]
     │   ├── wf-translate-text           [lambda function to translate non-english text using Amazon Translate]
     │   └── wf_publish_topic_model      [lambda function to publish topic modeling inferences from Amazon Comprehend]
     ├── lib
+    │   ├── aspects                     [CDK Aspects definitions to inject attributes during the prepare phase]
+    │   ├── awsnodejs-lambda-layer      [Lambda layer construct for lambda functions that run on Nodejs runtime]
     │   ├── ingestion                   [CDK constructs for data ingestion]
     │   ├── integration                 [CDK constructs for Amazon Events Bridge]
+    │   ├── quicksight-custom-resources [CDK construct that invokes custom resources to create Amazon QuickSight resources]
     │   ├── storage                     [CDK constructs that define storage of the inference events]
     │   ├── text-analysis-workflow      [CDK constructs for text analysis of ingested data]
     │   ├── topic-analysis-workflow     [CDK constructs for topic visualization of ingested data]
@@ -136,13 +150,23 @@ $CF_TEMPLATE_BUCKET_NAME - The name of the S3 bucket where the CloudFormation te
 $QS_TEMPLATE_ACCOUNT - The account from which the Amazon QuickSight templates should be sourced for Amazon QuickSight Analysis and Dashboard creation
 ```
 
+-   When creating and using buckets it is recommeded to:
+
+    -   Use randomized names or uuid as part of your bucket naming strategy.
+    -   Ensure buckets are not public.
+    -   Verify bucket ownership prior to uploading templates or code artifacts.
+
 -   Deploy the distributable to an Amazon S3 bucket in your account. _Note:_ you must have the AWS Command Line Interface installed.
 
 ```
 aws s3 cp ./global-s3-assets/ s3://my-bucket-name-<aws_region>/discovering-hot-topics-using-machine-learning/<my-version>/ --recursive --acl bucket-owner-full-control --profile aws-cred-profile-name
 aws s3 cp ./regional-s3-assets/ s3://my-bucket-name-<aws_region>/discovering-hot-topics-using-machine-learning/<my-version>/ --recursive --acl bucket-owner-full-control --profile aws-cred-profile-name
 ```
 
+## Collection of operational metrics
+
+This solution collects anonymous operational metrics to help AWS improve the quality and features of the solution. For more information, including how to disable this capability, please see the [implementation guide](https://docs.aws.amazon.com/solutions/latest/discovering-hot-topics-using-machine-learning/operational-metrics.html).
+
 ---
 
 Copyright 2020-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.

deployment/build-s3-dist.sh

@@ -25,9 +25,6 @@
 [ "$DEBUG" == 'true' ] && set -x
 set -e
 
-# Important: CDK global version number
-cdk_version=1.78.0
-
 # Check to see if input has been provided:
 if [ -z "$1" ] || [ -z "$2" ] || [ -z "$3" ] || [ -z "$4" ] || [ -z "$5" ] || [ -z "$6" ]; then
     echo "Please provide all required parameters for the build script"
@@ -100,10 +97,18 @@ echo "[Synth] CDK Project"
 echo "------------------------------------------------------------------------------"
 cd $source_dir
 
+# Important: CDK global version number
+cdk_version=$(node ../deployment/get-cdk-version.js) # Note: grabs from node_modules/aws-cdk/package.json
+
+echo "------------------------------------------------------------------------------"
+echo "[Install] Installing CDK $cdk_version"
+echo "------------------------------------------------------------------------------"
+
 npm install aws-cdk@$cdk_version
 
 ## Option to suppress the Override Warning messages while synthesizing using CDK
-# export overrideWarningsEnabled=false
+export overrideWarningsEnabled=false
+echo "setting override warning to $overrideWarningsEnabled"
 
 node_modules/aws-cdk/bin/cdk synth --output=$staging_dist_dir
 

deployment/cdk-solution-helper/index.js

@@ -17,6 +17,9 @@ const fs = require('fs');
 // Paths
 const global_s3_assets = '../global-s3-assets';
 
+//this regular express also takes into account lambda functions defined in nested stacks
+const _regex = /[\w]*AssetParameters/g;
+
 // For each template in global_s3_assets ...
 fs.readdirSync(global_s3_assets).forEach(file => {
     // Import and parse template file
@@ -26,27 +29,34 @@ fs.readdirSync(global_s3_assets).forEach(file => {
     // Clean-up Lambda function code dependencies
     const resources = (template.Resources) ? template.Resources : {};
     const lambdaFunctions = Object.keys(resources).filter(function (key) {
-        return resources[key].Type === 'AWS::Lambda::Function';
+        return (resources[key].Type === 'AWS::Lambda::Function' || resources[key].Type === 'AWS::Lambda::LayerVersion');
     });
 
     lambdaFunctions.forEach(function (f) {
         const fn = template.Resources[f];
-        if (fn.Properties.Code.hasOwnProperty('S3Bucket')) {
+        let prop;
+        if (fn.Properties.hasOwnProperty('Code')) {
+            prop = fn.Properties.Code;
+        } else if (fn.Properties.hasOwnProperty('Content')) {
+            prop = fn.Properties.Content;
+        }
+
+        if (prop.hasOwnProperty('S3Bucket')){
             // Set the S3 key reference
-            let artifactHash = Object.assign(fn.Properties.Code.S3Bucket.Ref);
+            let artifactHash = Object.assign(prop.S3Bucket.Ref);
             // console.debug(`Old artificatHash is ${artifactHash}`);
-            artifactHash = artifactHash.replace(/[\w]*AssetParameters/g, '');
+            artifactHash = artifactHash.replace(_regex, '');
             artifactHash = artifactHash.substring(0, artifactHash.indexOf('S3Bucket'));
             // console.debug(`New artificatHash is ${artifactHash}`);
             const assetPath = `asset${artifactHash}`;
-            fn.Properties.Code.S3Key = `%%SOLUTION_NAME%%/%%VERSION%%/${assetPath}.zip`;
+            prop.S3Key = `%%SOLUTION_NAME%%/%%VERSION%%/${assetPath}.zip`;
 
             // Set the S3 bucket reference
-            fn.Properties.Code.S3Bucket = {
+            prop.S3Bucket = {
                 'Fn::Sub': '%%BUCKET_NAME%%-${AWS::Region}'
             };
         } else {
-            // console.debug(`Here is the fn dump ${JSON.stringify(fn)}`);
+            console.warn(`No S3Bucket Property found for ${JSON.stringify(prop)}`);
         }
     });
 
@@ -61,19 +71,19 @@ fs.readdirSync(global_s3_assets).forEach(file => {
             'Fn::Join': [
                 '',
                 [
-                    'https://s3.',
+                    'https://%%TEMPLATE_BUCKET_NAME%%.s3.',
                     {
                         'Ref' : 'AWS::URLSuffix'
                     },
                     '/',
-                    `%%TEMPLATE_BUCKET_NAME%%/%%SOLUTION_NAME%%/%%VERSION%%/${fn.Metadata.nestedStackFileName}`
+                    `%%SOLUTION_NAME%%/%%VERSION%%/${fn.Metadata.nestedStackFileName}`
                 ]
             ]
         };
 
         const params = fn.Properties.Parameters ? fn.Properties.Parameters : {};
         const nestedStackParameters = Object.keys(params).filter(function(key) {
-            if (key.search(/[\w]*AssetParameters/g) > -1) {
+            if (key.search(_regex) > -1) {
                 return true;
             }
             return false;
@@ -87,12 +97,9 @@ fs.readdirSync(global_s3_assets).forEach(file => {
     // Clean-up parameters section
     const parameters = (template.Parameters) ? template.Parameters : {};
     const assetParameters = Object.keys(parameters).filter(function (key) {
-        console.debug(`key to analyze ${key}`);
-        if (key.search(/[\w]*AssetParameters/g) > -1) {
-            // console.debug('Pattern match');
+        if (key.search(_regex) > -1) {
             return true;
         }
-        // console.debug('Pattern did not match');
         return false;
     });
     assetParameters.forEach(function (a) {

source/bin/discovering-hot-topics-app.ts

@@ -14,11 +14,12 @@
 
 
 import * as cdk from '@aws-cdk/core';
+import { ApplytoLambda } from '../lib/aspects/apply-to-lambda';
 import { DiscoveringHotTopicsStack } from '../lib/discovering-hot-topics-stack';
 
 const app = new cdk.App();
-new DiscoveringHotTopicsStack(app, 'discovering-hot-topics-using-machine-learning', {
-    description: '(SO0122) - Discovering Hot Topics using Machine Learning. Version %%VERSION%%',
-    solutionID: 'SO0122',
-    solutionName: 'discovering-hot-topics-using-machine-learning'
+const dht = new DiscoveringHotTopicsStack(app, 'discovering-hot-topics-using-machine-learning', {
+    description: `(${app.node.tryGetContext('solution_id')}) - Discovering Hot Topics using Machine Learning. Version %%VERSION%%`
 });
+
+app.node.applyAspect(new ApplytoLambda(dht, 'CustomConfig'));

source/cdk.json

@@ -1,6 +1,9 @@
 {
   "app": "npx ts-node bin/discovering-hot-topics-app.ts",
   "context": {
-    "quicksight_source_template_arn": "arn:aws:quicksight:us-east-1:%%TEMPLATE_ACCOUNT_ID%%:template/%%DIST_QUICKSIGHT_NAMESPACE%%_%%SOLUTION_NAME%%_%%DASHED_VERSION%%"
+    "quicksight_source_template_arn": "arn:aws:quicksight:us-east-1:%%TEMPLATE_ACCOUNT_ID%%:template/%%DIST_QUICKSIGHT_NAMESPACE%%_%%SOLUTION_NAME%%_%%DASHED_VERSION%%",
+    "solution_id": "SO0122",
+    "solution_name": "%%SOLUTION_NAME%%",
+    "solution_version": "%%VERSION%%"
   }
 }