Update to version v2.0.1

Fix for issue #69 and library updates - v2.0.1

Author: tabdunabi

CHANGELOG.md

@@ -5,6 +5,17 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.0.1] - 2022-11-16
+
+### Fixed
+
+- Github [issue #69](https://github.com/aws-solutions/discovering-hot-topics-using-machine-learning/issues/69). The fix required the ingestion  infrastructure code in CDK to be pass the AWS CloudFormation parameter to be passed to the Reddit nested stack.
+
+### Updated
+
+- CDK version 1.177.0
+- AWS SDK verson 2.1223.0
+
 ## [2.0.0] - 2022-05-31
 
 ### Added
@@ -79,7 +90,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
-- Truncated tweets through merging [GitHub pull request #26](https://github.com/awslabs/discovering-hot-topics-using-machine-learning/pull/26)
+- Truncated tweets through merging [GitHub pull request #26](https://github.com/aws-solutions/discovering-hot-topics-using-machine-learning/pull/26)
 
 ## [1.4.0] - 2021-02-04
 

CONTRIBUTING.md

@@ -6,7 +6,6 @@ documentation, we greatly value feedback and contributions from our community.
 Please read through this document before submitting any issues or pull requests to ensure we have all the necessary
 information to effectively respond to your bug report or contribution.
 
-
 ## Reporting Bugs/Feature Requests
 
 We welcome you to use the GitHub issue tracker to report bugs or suggest features.
@@ -19,8 +18,8 @@ reported the issue. Please try to include as much information as you can. Detail
 * Any modifications you've made relevant to the bug
 * Anything unusual about your environment or deployment
 
-
 ## Contributing via Pull Requests
+
 Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that:
 
 1. You are working against the latest source on the *master* branch.
@@ -39,20 +38,19 @@ To send us a pull request, please:
 GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and
 [creating a pull request](https://help.github.com/articles/creating-a-pull-request/).
 
-
 ## Finding contributions to work on
-Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels ((enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/aws-solutions/discovering-hot-topics-using-machine-learning/labels/help%20wanted) issues is a great place to start.
 
+Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels ((enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/aws-solutions/discovering-hot-topics-using-machine-learning/labels/help%20wanted) issues is a great place to start.
 
 ## Code of Conduct
+
 This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
 For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
 [email protected] with any additional questions or comments.
 
-
 ## Security issue notifications
-If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue.
 
+If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue.
 
 ## Licensing
 

README.md

@@ -80,46 +80,46 @@ The solution is deployed using a CloudFormation template with a lambda backed cu
 
 ```
 ├── deployment                          [folder containing build scripts]
-│   ├── cdk-solution-helper             [A helper function to help deploy lambda function code through S3 buckets]
+│   ├── cdk-solution-helper             [A helper function to help deploy lambda function code through S3 buckets]
 │   ├── build-s3-dist.sh                [Build script to build the solution]
 └── source                              [source code containing CDK App and lambda functions]
     ├── bin                             [entrypoint of the CDK application]
     ├── lambda                          [folder containing source code the lambda functions]
     │   ├── capture_news_feed           [lambda function to ingest news feeds]
-    │   ├── firehose_topic_proxy        [lambda function to write topic analysis output to Amazon Kinesis Firehose]
-    │   ├── firehose-text-proxy         [lambda function to write text analysis output to Amazon Kinesis Firehose]
-    │   ├── ingestion-consumer          [lambda function that consumes messages from Amazon Kinesis Data Streams]
-    │   ├── ingestion-custom            [lambda function that reads files from Amazon S3 bucket and pushes data to Amazon Kinesis Data Streams]
-    │   ├── ingestion-producer          [lambda function that makes Twitter API call and pushes data to Amazon Kinesis Data Stream]
-    │   ├── ingestion-reddit            [lambda function that makes Reddit API call to retrieve comments from subreddits of interest and pushes data to Amazon Kinesis Data Stream]
-    │   ├── ingestion-youtube           [lambda function that ingests comments from YouTube videos and pushes data to Amazon Kinesis Data Streams]
-    │   ├── integration                 [lambda function that publishes inference outputs to Amazon Events Bridge]
+    │   ├── firehose_topic_proxy        [lambda function to write topic analysis output to Amazon Kinesis Firehose]
+    │   ├── firehose-text-proxy         [lambda function to write text analysis output to Amazon Kinesis Firehose]
+    │   ├── ingestion-consumer          [lambda function that consumes messages from Amazon Kinesis Data Streams]
+    │   ├── ingestion-custom            [lambda function that reads files from Amazon S3 bucket and pushes data to Amazon Kinesis Data Streams]
+    │   ├── ingestion-producer          [lambda function that makes Twitter API call and pushes data to Amazon Kinesis Data Stream]
+    │   ├── ingestion-reddit            [lambda function that makes Reddit API call to retrieve comments from subreddits of interest and pushes data to Amazon Kinesis Data Stream]
+    │   ├── ingestion-youtube           [lambda function that ingests comments from YouTube videos and pushes data to Amazon Kinesis Data Streams]
+    │   ├── integration                 [lambda function that publishes inference outputs to Amazon Events Bridge]
     │   ├── layers                      [lambda layer function library for Node and Python layers]
     │   │   ├── aws-nodesdk-custom-config
     │   │   ├── python_lambda_layer
     │   ├── quicksight-custom-resources [lambda function to create Amazon QuickSight resources, example: data source, data sets, analysis and dashboards]
     │   ├── solution_helper             [lambda function that allows capturing metrics for this solution]
-    │   ├── storage-firehose-processor  [lambda function that writes data to S3 buckets to build a relational model]
-    │   ├── wf-analyze-text             [lambda function to detect sentiments, key phrases and entities using Amazon Comprehend]
-    │   ├── wf-check-topic-model        [lambda function to check status of topic modeling jobs on Amazon Comprehend]
+    │   ├── storage-firehose-processor  [lambda function that writes data to S3 buckets to build a relational model]
+    │   ├── wf-analyze-text             [lambda function to detect sentiments, key phrases and entities using Amazon Comprehend]
+    │   ├── wf-check-topic-model        [lambda function to check status of topic modeling jobs on Amazon Comprehend]
     │   ├── wf-detect-language          [lambda function to detect language of ingested text content using Amazon Comprehend]
-    │   ├── wf-detect-moderation-labels [lambda function to detect content moderation using Amazon Rekognition]
-    │   ├── wf-extract-text-in-image    [lambda function to extract text content from images using Amazon Rekognition]
-    │   ├── wf-publish-text-inference   [lambda function to publish Amazon Comprehend inferences]
-    │   ├── wf-submit-topic-model       [lambda function to submit topic modeling job]
-    │   ├── wf-translate-text           [lambda function to translate non-english text using Amazon Translate]
-    │   └── wf_publish_topic_model      [lambda function to publish topic modeling inferences from Amazon Comprehend]
+    │   ├── wf-detect-moderation-labels [lambda function to detect content moderation using Amazon Rekognition]
+    │   ├── wf-extract-text-in-image    [lambda function to extract text content from images using Amazon Rekognition]
+    │   ├── wf-publish-text-inference   [lambda function to publish Amazon Comprehend inferences]
+    │   ├── wf-submit-topic-model       [lambda function to submit topic modeling job]
+    │   ├── wf-translate-text           [lambda function to translate non-english text using Amazon Translate]
+    │   └── wf_publish_topic_model      [lambda function to publish topic modeling inferences from Amazon Comprehend]
     ├── lib
     │   ├── aspects                     [CDK Aspects definitions to inject attributes during the prepare phase]
     │   ├── awsnodejs-lambda-layer      [Lambda layer construct for lambda functions that run on Nodejs runtime]
-    │   ├── ingestion                   [CDK constructs for data ingestion]
-    │   ├── integration                 [CDK constructs for Amazon Events Bridge]
+    │   ├── ingestion                   [CDK constructs for data ingestion]
+    │   ├── integration                 [CDK constructs for Amazon Events Bridge]
     │   ├── quicksight-custom-resources [CDK construct that invokes custom resources to create Amazon QuickSight resources]
     │   ├── s3-event-notification       [CDK construct that configures S3 events to be pushed to Amazon EventBridge]
-    │   ├── storage                     [CDK constructs that define storage of the inference events]
-    │   ├── text-analysis-workflow      [CDK constructs for text analysis of ingested data]
-    │   ├── topic-analysis-workflow     [CDK constructs for topic visualization of ingested data]
-    │   └── visualization               [CDK constructs to build a relational database model for visualization]
+    │   ├── storage                     [CDK constructs that define storage of the inference events]
+    │   ├── text-analysis-workflow      [CDK constructs for text analysis of ingested data]
+    │   ├── topic-analysis-workflow     [CDK constructs for topic visualization of ingested data]
+    │   └── visualization               [CDK constructs to build a relational database model for visualization]
     ├── discovering-hot-topics.ts
 ```
 

deployment/build-s3-dist.sh

@@ -144,6 +144,11 @@ done
 
 node $template_dir/cdk-solution-helper/index
 
+app_registry_name=$(echo "$solution_name" | sed 's/ /-/g' | awk '{print tolower($0)}' | cut -c 1-64)
+echo "-----------------------------------------------------------------------------"
+echo "Setting App Registry Application Name as $app_registry_name"
+echo "-----------------------------------------------------------------------------"
+
 echo "------------------------------------------------------------------------------"
 echo "Updating placeholders"
 echo "------------------------------------------------------------------------------"
@@ -155,6 +160,9 @@ do
     replace="s/%%SOLUTION_NAME%%/$solution_name/g"
     sed -i -e $replace $file
 
+    replace="s/%%APP_REG_NAME%%/$app_registry_name/g"
+    sed -i -e $replace $file
+
     replace="s/%%VERSION%%/$solution_version/g"
     sed -i -e $replace $file
 

source/bin/discovering-hot-topics-app.ts

@@ -1,8 +1,8 @@
 #!/usr/bin/env node
 /**********************************************************************************************************************
- *  Copyright 2020-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.                                      *
+ *  Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.                                                *
  *                                                                                                                    *
- *  Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance    *
+ *  Licensed under the Apache License, Version 2.0 (the 'License'). You may not use this file except in compliance    *
  *  with the License. A copy of the License is located at                                                             *
  *                                                                                                                    *
  *      http://www.apache.org/licenses/LICENSE-2.0                                                                    *
@@ -12,14 +12,15 @@
  *  and limitations under the License.                                                                                *
  *********************************************************************************************************************/
 
-
 import * as cdk from '@aws-cdk/core';
 import { ApplytoLambda } from '../lib/aspects/apply-to-lambda';
 import { DiscoveringHotTopicsStack } from '../lib/discovering-hot-topics-stack';
 
 const app = new cdk.App();
 const dht = new DiscoveringHotTopicsStack(app, 'discovering-hot-topics-using-machine-learning', {
-    description: `(${app.node.tryGetContext('solution_id')}) - Discovering Hot Topics using Machine Learning. Version %%VERSION%%`
+    description: `(${app.node.tryGetContext(
+        'solution_id'
+    )}) - Discovering Hot Topics using Machine Learning. Version %%VERSION%%`
 });
 
 cdk.Aspects.of(app).add(new ApplytoLambda(dht, 'CustomConfig'));

source/cdk.json

@@ -4,6 +4,8 @@
     "quicksight_source_template_arn": "arn:aws:quicksight:us-east-1:%%TEMPLATE_ACCOUNT_ID%%:template/%%DIST_QUICKSIGHT_NAMESPACE%%_%%SOLUTION_NAME%%_%%DASHED_VERSION%%",
     "solution_id": "SO0122",
     "solution_name": "%%SOLUTION_NAME%%",
-    "solution_version": "%%VERSION%%"
+    "solution_version": "%%VERSION%%",
+    "app_registry_name": "%%APP_REG_NAME%%",
+    "application_type": "AWS-Solutions"
   }
-}
+}

source/lambda/capture_news_feed/lambda_function.py

@@ -29,7 +29,7 @@ class IncorrectEventNameSpaceError(Exception):
     pass
 
 
-def publish_config_handler(event, context):
+def publish_config_handler(event, _):  # NOSONAR - lambda signature
     """ If lambda environment variable is set to read from os.environ, if not read from DDB """
     if os.environ.get("CONFIG_PARAM", None):
         """
@@ -63,7 +63,7 @@ def publish_config_handler(event, context):
             logger.debug(f"Event published is: {config_event}")
 
 
-def process_config_handler(event, context):
+def process_config_handler(event, _):  # NOSONAR - lambda signature
     if event["source"] == os.environ["INGESTION_NAMESPACE"]:
         data = event["detail"]
 

source/lambda/capture_news_feed/requirements-dev.txt

@@ -1,6 +1,6 @@
-moto==3.1.7
-pytest==7.1.2
-pytest-cov==3.0.0
+moto==4.0.5
+pytest==7.1.3
+pytest-cov==4.0.0
 botocore
 mock==4.0.3
-responses==0.20.0
+responses==0.21.0

source/lambda/capture_news_feed/util/ddb_helper.py

@@ -80,7 +80,7 @@ def update_query_tracker(account, url, search_query, topic=None):
     )
 
 
-def get_query_tracker(account, url, search_query, topic=None, **item_kwargs):
+def get_query_tracker(account, url, search_query, topic=None, **item_kwargs):  # NOSONAR allow passing keyword args
     dynamodb = service_helper.get_service_resource("dynamodb")
 
     table = dynamodb.Table(os.environ["TARGET_DDB_TABLE"])