updates for release 1.7.0

Author: knihit

CHANGELOG.md

@@ -5,96 +5,116 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.7.0] - 2022-02-14
+
+### Added
+
+- The capability to ingest custom data by uploading files as JSON, XLSX, or CSV files
+
+### Updated
+
+- Use Amazon Kinesis Data Firehose partition projection to store and partition the data by source date (instead of system processing date)
+- Use Amazon Athena dynamic partitioning features to run SQL queries on data stored in S3 bucket
+- AWS CDK version 1.137.0
+- AWS SDK version 2.1067.0
+
+### Removed
+
+- Creating of AWS Glue partitions (replaced with Amazon Athena dynamic partitions)
+
 ## [1.6.1] - 2021-10-26
+
 ### Fixed
--   GitHub [issue #42](https://github.com/aws-solutions/discovering-hot-topics-using-machine-learning/issues/42). To fix the issue, RSS feed ingestion lambda function and SQLs related to the Amazon QuickSight dashboard were updated.
+
+- GitHub [issue #42](https://github.com/aws-solutions/discovering-hot-topics-using-machine-learning/issues/42). To fix the issue, RSS feed ingestion lambda function and SQLs related to the Amazon QuickSight dashboard were updated.
 
 ### Updated
--   AWS CDK version to 1.125.0
--   AWS SDK version to 2.1008.0
+
+- AWS CDK version to 1.125.0
+- AWS SDK version to 2.1008.0
 
 ## [1.6.0] - 2021-09-27
 
 ### Added
 
--   Capability to ingest YouTube comments
+- Capability to ingest YouTube comments
 
 ### Updated
 
--   AWS CDK version to 1.121.0
--   AWS SDK version to 2.991.0
--   Updated Amazon QuickSight analysis and dashboard to reflect the new ingestion source
+- AWS CDK version to 1.121.0
+- AWS SDK version to 2.991.0
+- Updated Amazon QuickSight analysis and dashboard to reflect the new ingestion source
 
 ## [1.5.0] - 2021-07-22
 
 ### Added
 
--   Ingest RSS feeds from over ~3000+ news websites across the world
+- Ingest RSS feeds from over ~3000+ news websites across the world
 
 ### Updated
 
--   AWS CDK version to 1.110.1
--   AWS SDK version to 2.945.0
--   Updated Nodejs Lambda runtimes to use Nodejs 14.x
--   Updated Amazon QuickSight analysis and dashboard to reflect the new ingestion source
--   Updated AWS StepFunction workflows to handle parallel ingestion (tweets from Twitter and RSS feeds from News websites)
+- AWS CDK version to 1.110.1
+- AWS SDK version to 2.945.0
+- Updated Nodejs Lambda runtimes to use Nodejs 14.x
+- Updated Amazon QuickSight analysis and dashboard to reflect the new ingestion source
+- Updated AWS StepFunction workflows to handle parallel ingestion (tweets from Twitter and RSS feeds from News websites)
 
 ### Fixed
 
--   Truncated tweets through merging [GitHub pull request #26](https://github.com/awslabs/discovering-hot-topics-using-machine-learning/pull/26)
+- Truncated tweets through merging [GitHub pull request #26](https://github.com/awslabs/discovering-hot-topics-using-machine-learning/pull/26)
 
 ## [1.4.0] - 2021-02-04
 
 ### Added
 
--   Capability to use geo coordinates when invoking the Twitter API to filter tweets returned by its Search API
--   New visuals and sheets (tabs) on Amazon QuickSight to perform analysis using geo coordinates (when available with tweets)
--   Additional remediation to handle throttling conditions from Twitter v1.1 API calls and push additional information to Amazon CloudWatch Logs that can be used to create alarms or notifications using CloudWatch Metric Filters
+- Capability to use geo coordinates when invoking the Twitter API to filter tweets returned by its Search API
+- New visuals and sheets (tabs) on Amazon QuickSight to perform analysis using geo coordinates (when available with tweets)
+- Additional remediation to handle throttling conditions from Twitter v1.1 API calls and push additional information to Amazon CloudWatch Logs that can be used to create alarms or notifications using CloudWatch Metric Filters
 
 ### Updated
 
--   Switched to AWS Managed KMS keys for AWS Glue Security Configuration
--   AWS CDK version to 1.83.0
--   AWS SDK version to 2.828.0
+- Switched to AWS Managed KMS keys for AWS Glue Security Configuration
+- AWS CDK version to 1.83.0
+- AWS SDK version to 2.828.0
 
 ## [1.3.0] - 2020-11-24
 
 ### Changed
 
--   Implementation to refactor and to reuse the following architecture patterns from [AWS Solutions Constructs](https://aws.amazon.com/solutions/constructs/)
-    -   aws-kinesisfirehose-s3
-    -   aws-kinesisstreams-lambda
-    -   aws-lambda-step-function
+- Implementation to refactor and to reuse the following architecture patterns from [AWS Solutions Constructs](https://aws.amazon.com/solutions/constructs/)
+  - aws-kinesisfirehose-s3
+  - aws-kinesisstreams-lambda
+  - aws-lambda-step-function
 
 ### Updated
 
--   The join condition for Topic Modeling in Amazon QuickSight dataset to provide accurate topic identification for a specific run
--   ID and name generation for Amazon QuickSigh resource to use dynamic value based on the stack name
--   AWS CDK version to 1.73.0
--   AWS SDK version to 2.790.0
+- The join condition for Topic Modeling in Amazon QuickSight dataset to provide accurate topic identification for a specific run
+- ID and name generation for Amazon QuickSigh resource to use dynamic value based on the stack name
+- AWS CDK version to 1.73.0
+- AWS SDK version to 2.790.0
 
 ## [1.2.0] - 2020-10-29
 
 ### Added
 
--   New and simplified interactive Amazon QuickSight dashboard that is now automatically generated through an AWS CloudFormation deployment and that customers can extend to suit their business case
+- New and simplified interactive Amazon QuickSight dashboard that is now automatically generated through an AWS CloudFormation deployment and that customers can extend to suit their business case
 
 ### Updated
 
--   Updated to AWS CDK v1.69.0
--   Consolidate Amazon S3 access Log bucket across the solution. All access log files have a prefix that corresponds to the bucket for which they are generated
+- Updated to AWS CDK v1.69.0
+- Consolidate Amazon S3 access Log bucket across the solution. All access log files have a prefix that corresponds to the bucket for which they are generated
 
 ## [1.1.0] - 2020-09-29
 
 ### Updated
 
--   S3 storage for inference outputs to use Apache Parquet
--   Add partitioning to AWS Glue tables
--   Update to AWS CDK v1.63.0
--   Update to AWS SDK v2.755.0
+- S3 storage for inference outputs to use Apache Parquet
+- Add partitioning to AWS Glue tables
+- Update to AWS CDK v1.63.0
+- Update to AWS SDK v2.755.0
 
 ## [1.0.0] - 2020-08-28
 
 ### Added
 
--   Initial release
+- Initial release

NOTICE.txt

@@ -25,12 +25,15 @@ boto3 - Apache-2.0
 botocore - Apache-2.0
 chai - MIT license
 crhelper - Apache-2.0
+googleapis - Apache-2.0
 jest - MIT license
+jmespath - MIT License
 momentjs - MIT license
 moto - Apache-2.0
 newscatcher - MIT license
 nock - MIT license
 node - MIT license
+openpyx - MIT license
 pytest-cov - MIT license
 pytest - MIT license
 requests - Apache-2.0

source/.eslintignore

@@ -0,0 +1,12 @@
+# don't ever lint node_modules
+node_modules
+cdk.out
+# don't lint build output (make sure it's set to your correct build folder name)
+dist
+# don't lint test folders
+test
+# don't lint coverage output
+coverage
+*.config.js
+# dont lint eslint config
+*.eslint*

source/.gitignore

@@ -0,0 +1,18 @@
+*.js
+!jest.config.js
+*.d.ts
+node_modules
+
+# CDK asset staging directory
+.cdk.staging
+cdk.out
+
+# Parcel build directories
+.cache
+.build
+
+# JS files in the lambda folder should not be ignored
+!lambda/**/*.js
+
+# Coverage reports
+test/coverage-reports

source/.prettierignore

@@ -0,0 +1 @@
+node_modules

source/.prettierrc.yml

@@ -0,0 +1,8 @@
+# .prettierrc or .prettierrc.yaml
+proseWrap: 'preserve'
+trailingComma: 'none'
+tabWidth: 4
+semi: true
+singleQuote: true
+quoteProps: 'preserve'
+printWidth: 120

source/images/architecture.png

@@ -0,0 +1,14 @@
+module.exports = {
+    root: true,
+    parserOptions: {
+        ecmaVersion: 2021
+    },
+    env: {
+        node: true
+    },
+    extends: ['eslint:recommended'],
+    rules: {
+        indent: ['error', 4],
+        quotes: ['warn', 'single']
+    }
+};

source/lambda/.eslintrc.js

@@ -1,5 +1,6 @@
-moto==2.2.2
-pytest==6.2.4
-pytest-cov==2.12.1
+moto==2.3.1
+pytest==6.2.5
+pytest-cov==3.0.0
 botocore
-mock==4.0.3
+mock==4.0.3
+responses==0.16.0

source/lambda/capture_news_feed/requirements-dev.txt

@@ -15,11 +15,11 @@
 import json
 import os
 import unittest
-from datetime import datetime
+from datetime import datetime, timezone
 
 from moto import mock_kinesis
 from shared_util.service_helper import get_service_client
-from util.stream_helper import buffer_data_into_stream
+from shared_util.stream_helper import buffer_data_into_stream
 
 
 @mock_kinesis
@@ -48,7 +48,7 @@ def test_buffer_data_into_stream(self):
             "platform": "fakeplatform",
             "search_query": "query_str",
             "feed": {
-                "created_at": datetime.now().timestamp(),
+                "created_at": datetime.now(timezone.utc).timestamp(),
                 "id": "fakeid",
                 "id_str": "fakeid",
                 "text": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.",