Merge pull request #43 from aws-solutions/feature/v1.6.0

Release v1.6.0 changes

Author: aassadza

.gitignore

@@ -272,3 +272,5 @@ dmypy.json
 .pytype/
 source/lambda/wf_publish_topic_model/test/fixtures/output/doc-topics.csv
 source/lambda/wf_publish_topic_model/test/fixtures/output/topic-terms.csv
+
+.idea/

CHANGELOG.md

@@ -5,6 +5,18 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.6.0] - 2021-09-27
+
+### Added
+
+-   Capability to ingest YouTube comments
+
+### Updated
+
+-   AWS CDK version to 1.121.0
+-   AWS SDK version to 2.991.0
+-   Updated Amazon QuickSight analysis and dashboard to reflect the new ingestion source
+
 ## [1.5.0] - 2021-07-22
 
 ### Added

README.md

@@ -2,7 +2,7 @@
 
 The Discovering Hot Topics Using Machine Learning solution helps you identify the most dominant topics associated with your products, policies, events, and brands. Implementing this solution helps you react quickly to new growth opportunities, address negative brand associations, and deliver higher levels of customer satisfaction.
 
-The solution automates digital asset (text and image) ingestion from twitter and RSS news feeds to provide near-real-time inferences using machine learning algorithms through Amazon Comprehend, Amazon Translate, and Amazon Rekognition to perform topic modeling, sentiment analysis, entity and key phrase detection, and detect any unsafe images. The solution then visualizes these large-scale customer analyses using an Amazon QuickSight dashboard. This guide provides step-by-step instructions for deploying this solution including a pre-built dashboard that provides you with the context and insights necessary to identify trends that help or harm your brand.
+The solution automates digital asset (text and image) ingestion from twitter, RSS news feeds, and YouTube comments to provide near-real-time inferences using machine learning algorithms through Amazon Comprehend, Amazon Translate, and Amazon Rekognition to perform topic modeling, sentiment analysis, entity and key phrase detection, and detect any unsafe images. The solution then visualizes these large-scale customer analyses using an Amazon QuickSight dashboard. This guide provides step-by-step instructions for deploying this solution including a pre-built dashboard that provides you with the context and insights necessary to identify trends that help or harm your brand.
 
 The solution performs the following key features:
 
@@ -11,7 +11,7 @@ The solution performs the following key features:
 -   **Determines if images associated with your brand contain unsafe content**: detects unsafe and negative imagery in content
 -   **Helps customers identify insights in near real-time**: you can use a visualization dashboard to better understand context, threats, and opportunities almost instantly
 
-This solution deploys an AWS CloudFormation template that supports both Twitter and RSS feeds as data source options for ingestion, but the solution can be customized to aggregate other social media platforms and internal enterprise systems.
+This solution deploys an AWS CloudFormation template that supports Twitter, RSS feeds, and YouTube comments as data source options for ingestion, but the solution can be customized to aggregate other social media platforms and internal enterprise systems.
 
 For a detailed solution deployment guide, refer to [Discovering Hot Topics using Machine Learning](https://aws.amazon.com/solutions/implementations/discovering-hot-topics-using-machine-learning)
 
@@ -33,7 +33,7 @@ Deploying this solution with the default parameters builds the following environ
 
 The architecture of the solution includes the following key components and workflows:
 
-1. Ingestion – Social media and RSS feed ingestion and management using Lambda functions, Amazon DynamoDB, and Amazon CloudWatch Event Scheduler.
+1. Ingestion – Twitter, RSS feeds, and YouTube comments ingestion and management using Lambda functions, Amazon DynamoDB, and Amazon CloudWatch Event Scheduler. For detailed reference architecture diagrams for Twitter, YouTube comment, and RSS news feed ingestion, refer to the [implementation guide](https://docs.aws.amazon.com/solutions/latest/discovering-hot-topics-using-machine-learning/overview.html).
 
 2. Data Stream — The data is buffered through Amazon Kinesis Data Streams to provide resiliency and throttle incoming requests. The Data Streams have a configured DLQ to catch any errors in processing feeds.
 
@@ -78,15 +78,17 @@ The solution is deployed using a CloudFormation template with a lambda backed cu
     ├── bin                             [entrypoint of the CDK application]
     ├── lambda                          [folder containing source code the lambda functions]
     │   ├── capture_news_feed           [lambda function to ingest news feeds]
+    │   ├── create-partition            [lambda function to create glue partitions]
     │   ├── firehose_topic_proxy        [lambda function to write topic analysis output to Amazon Kinesis Firehose]
     │   ├── firehose-text-proxy         [lambda function to write text analysis output to Amazon Kinesis Firehose]
     │   ├── ingestion-consumer          [lambda function that consumes messages from Amazon Kinesis Data Stream]
     │   ├── ingestion-producer          [lambda function that makes Twitter API call and pushes data to Amazon Kinesis Data Stream]
+    │   ├── ingestion-youtube           [lambda function that ingests comments from YouTube videos and pushes data to Amazon Kinesis Data Stream]
     │   ├── integration                 [lambda function that publishes inference outputs to Amazon Events Bridge]
-    │   ├── layers                      [lambda layer function library]
-    │   │   ├── aws-nodesk-custom-config
+    │   ├── layers                      [lambda layer function library for Node and Python layers]
+    │   │   ├── aws-nodesdk-custom-config
+    │   │   ├── python_lambda_layer
     │   ├── quicksight-custom-resources [lambda function to create Amazon QuickSight resources, example: data source, data sets, analysis and dashboards]
-    │   ├── shared                      [lambda layer function library (specific to python lambda runtimes)]
     │   ├── solution_helper             [lambda function that allows capturing metrics for this solution]
     │   ├── storage-firehose-processor  [lambda function that writes data to S3 buckets to build a relational model]
     │   ├── wf-analyze-text             [lambda function to detect sentiments, key phrases and entities using Amazon Comprehend]

source/bin/discovering-hot-topics-app.ts

@@ -22,4 +22,4 @@ const dht = new DiscoveringHotTopicsStack(app, 'discovering-hot-topics-using-mac
     description: `(${app.node.tryGetContext('solution_id')}) - Discovering Hot Topics using Machine Learning. Version %%VERSION%%`
 });
 
-app.node.applyAspect(new ApplytoLambda(dht, 'CustomConfig'));
+cdk.Aspects.of(app).add(new ApplytoLambda(dht, 'CustomConfig'));

source/images/architecture.png

@@ -1,5 +1,6 @@
 [run]
 omit =
+    charset_normalizer/*
     *certifi/*
     *chardet/*
     feedparser.py

source/lambda/capture_news_feed/.coveragerc

@@ -15,7 +15,7 @@
 import json
 import os
 
-from dht_config import custom_boto_config, custom_logging
+from shared_util import custom_boto_config, custom_logging
 
 from util import config_helper, ddb_helper, event_bus_helper
 from util.ddb_helper import get_query_tracker

source/lambda/capture_news_feed/lambda_function.py

@@ -1,4 +1,5 @@
-moto==2.0.0
-pytest==6.2.2
-pytest-cov==2.11.1
-botocore
+moto==2.2.2
+pytest==6.2.4
+pytest-cov==2.12.1
+botocore
+mock==4.0.3

source/lambda/capture_news_feed/requirements-dev.txt

@@ -17,8 +17,8 @@
 import boto3
 import pytest
 from botocore.stub import Stubber
-from dht_config import custom_boto_config
 from moto.dynamodb2 import mock_dynamodb2
+from shared_util import custom_boto_config
 
 
 @pytest.fixture(autouse=True)

source/lambda/capture_news_feed/test/conftest.py

@@ -21,7 +21,7 @@
 @pytest.fixture()
 def get_event_bus_stubber():
     global event_bus_stubber
-    from util.helper import get_service_client
+    from shared_util.service_helper import get_service_client
 
     if not event_bus_stubber:
         event_bus_client = get_service_client("events")