Merge pull request #65 from aws-solutions/feature/v2.0.0

Feature/v2.0.0

Author: aijunpeng

CHANGELOG.md

@@ -5,6 +5,18 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.0.0] - 2022-05-31
+
+### Added
+
+- The capability to ingest Subreddit comments using the Reddit API
+
+### Updated
+
+- CDK version 1.152.0
+- AWS SDK version 2.1127.0
+- Amazon QuickSight dashboard to support NLP analysis on Subreddit comments
+
 ## [1.7.1] - 2022-03-29
 
 ### Updated

CONTRIBUTING.md

@@ -11,7 +11,7 @@ information to effectively respond to your bug report or contribution.
 
 We welcome you to use the GitHub issue tracker to report bugs or suggest features.
 
-When filing an issue, please check [existing open](https://github.com/awslabs/discovering-hot-topics-using-machine-learning/issues), or [recently closed](https://github.com/awslabs/discovering-hot-topics-using-machine-learning/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed20), issues to make sure somebody else hasn't already
+When filing an issue, please check [existing open](https://github.com/aws-solutions/discovering-hot-topics-using-machine-learning/issues), or [recently closed](https://github.com/aws-solutions/discovering-hot-topics-using-machine-learning/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed20), issues to make sure somebody else hasn't already
 reported the issue. Please try to include as much information as you can. Details like these are incredibly useful:
 
 * A reproducible test case or series of steps
@@ -41,7 +41,7 @@ GitHub provides additional document on [forking a repository](https://help.githu
 
 
 ## Finding contributions to work on
-Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels ((enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/awslabs/discovering-hot-topics-using-machine-learning/labels/help%20wanted) issues is a great place to start.
+Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels ((enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/aws-solutions/discovering-hot-topics-using-machine-learning/labels/help%20wanted) issues is a great place to start.
 
 
 ## Code of Conduct
@@ -56,6 +56,6 @@ If you discover a potential security issue in this project we ask that you notif
 
 ## Licensing
 
-See the [LICENSE](https://github.com/awslabs/discovering-hot-topics-using-machine-learning/blob/master/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
+See the [LICENSE](https://github.com/aws-solutions/discovering-hot-topics-using-machine-learning/blob/master/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
 
 We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes.

NOTICE.txt

@@ -38,7 +38,8 @@ pytest-cov - MIT license
 pytest - MIT license
 requests - Apache-2.0
 sinon - BSD license
-tenacity under the Apache License Version 2.0
+snoowrap - MIT license
+tenacity - Apache-2.0
 ts-jest - MIT license
 ts-node - MIT license
 twitter-lite - MIT License

README.md

@@ -83,6 +83,7 @@ The solution is deployed using a CloudFormation template with a lambda backed cu
     │   ├── ingestion-consumer          [lambda function that consumes messages from Amazon Kinesis Data Streams]
     │   ├── ingestion-custom            [lambda function that reads files from Amazon S3 bucket and pushes data to Amazon Kinesis Data Streams]
     │   ├── ingestion-producer          [lambda function that makes Twitter API call and pushes data to Amazon Kinesis Data Stream]
+    │   ├── ingestion-reddit            [lambda function that makes Reddit API call to retrieve comments from subreddits of interest and pushes data to Amazon Kinesis Data Stream]
     │   ├── ingestion-youtube           [lambda function that ingests comments from YouTube videos and pushes data to Amazon Kinesis Data Streams]
     │   ├── integration                 [lambda function that publishes inference outputs to Amazon Events Bridge]
     │   ├── layers                      [lambda layer function library for Node and Python layers]
@@ -122,7 +123,7 @@ The solution can be deployed through the CloudFormation template available on th
 
 Clone this git repository
 
-`git clone https://github.com/awslabs/<repository_name>`
+`git clone https://github.com/aws-solutions/<repository_name>`
 
 ### 2. Build the solution for deployment
 

source/images/architecture.png

@@ -1,6 +1,6 @@
-moto==3.0.7
-pytest==6.2.5
+moto==3.1.7
+pytest==7.1.2
 pytest-cov==3.0.0
 botocore
 mock==4.0.3
-responses==0.16.0
+responses==0.20.0

source/lambda/capture_news_feed/requirements-dev.txt

@@ -17,7 +17,7 @@
 import boto3
 import pytest
 from botocore.stub import Stubber
-from moto.dynamodb2 import mock_dynamodb2
+from moto.dynamodb import mock_dynamodb
 from shared_util import custom_boto_config
 
 

source/lambda/capture_news_feed/test/conftest.py

@@ -18,12 +18,12 @@
 import boto3
 from boto3.dynamodb.conditions import Attr, Key
 from botocore.exceptions import EndpointConnectionError
-from moto import mock_dynamodb2
+from moto import mock_dynamodb
 from shared_util import custom_boto_config, service_helper
 from util import ddb_helper
 
 
-@mock_dynamodb2
+@mock_dynamodb
 def ddb_setup(table_name):
     # ddb = boto3.resource("dynamodb", config=custom_boto_config.init())
     ddb = service_helper.get_service_resource("dynamodb")
@@ -39,7 +39,7 @@ def ddb_setup(table_name):
     return ddb
 
 
-@mock_dynamodb2
+@mock_dynamodb
 class TestConfigDDBHelper(unittest.TestCase):
     def setUp(self):
         self.table_name = os.environ["DDB_CONFIG_TABLE_NAME"]

source/lambda/capture_news_feed/test/test_config_ddb_helper.py

@@ -23,7 +23,7 @@
 import mock
 import pytest
 from botocore import stub
-from moto import mock_dynamodb2, mock_kinesis, mock_sts
+from moto import mock_dynamodb, mock_kinesis, mock_sts
 from shared_util import custom_boto_config
 
 
@@ -186,7 +186,7 @@ def lambda_event_bus_event():
 
 
 @mock_sts
-@mock_dynamodb2
+@mock_dynamodb
 def test_invoke_lambda_for_ddb_config(get_event_bus_stubber):
     lambda_event = create_cw_schedule_event()
     create_ddb_table_for_US_en()
@@ -215,7 +215,7 @@ def test_invoke_lambda_for_ddb_config(get_event_bus_stubber):
 
 
 @mock_sts
-@mock_dynamodb2
+@mock_dynamodb
 @mock.patch.dict(os.environ, {"CONFIG_PARAM": '{"country":"US", "language":"en"}'})
 @mock.patch.dict(os.environ, {"SEARCH_QUERY": "fakequery"})
 def test_invoke_lambda_for_json_str(get_event_bus_stubber):
@@ -245,7 +245,7 @@ def test_invoke_lambda_for_json_str(get_event_bus_stubber):
 
 
 @mock_sts
-@mock_dynamodb2
+@mock_dynamodb
 def test_invoke_lambda_for_ddb_config_with_failed_count(get_event_bus_stubber):
     lambda_event = create_cw_schedule_event()
     create_ddb_table_for_US_en()
@@ -274,7 +274,7 @@ def test_invoke_lambda_for_ddb_config_with_failed_count(get_event_bus_stubber):
 
 
 @mock_sts
-@mock_dynamodb2
+@mock_dynamodb
 @mock_kinesis
 def test_invoke_process_config_handler():
     lambda_event = create_event_bus_consumer_event()

source/lambda/capture_news_feed/test/test_lambda.py

@@ -17,7 +17,7 @@
 from datetime import datetime, timedelta, timezone
 from unittest import mock
 
-from moto import mock_dynamodb2, mock_kinesis
+from moto import mock_dynamodb, mock_kinesis
 from util import newscatcher_helper
 
 
@@ -57,7 +57,7 @@ def test_retrieve_feed_from_all_topics(self, mocked_feed_call):
         self.assertEqual(mocked_feed_call.call_count, len(newscatcher_helper.get_topic_list()))
 
     @mock_kinesis
-    @mock_dynamodb2
+    @mock_dynamodb
     def test_create_and_publish_record(self):
         setup_test_case()
         news_feed = newscatcher_helper.retrieve_feed("latimes.com")
@@ -73,7 +73,7 @@ def test_create_and_publish_record(self):
         )
 
     @mock_kinesis
-    @mock_dynamodb2
+    @mock_dynamodb
     def test_published_date(self):
         setup_test_case()
         news_feed = newscatcher_helper.retrieve_feed("cnn.com")
@@ -93,7 +93,7 @@ def test_published_date(self):
         )
 
     @mock_kinesis
-    @mock_dynamodb2
+    @mock_dynamodb
     def test_with_query_string(self):
         setup_test_case()
         news_feed = newscatcher_helper.retrieve_feed("cnn.com")