Transition to pyiceberg

.gitignore

@@ -4,4 +4,8 @@ assets/cloudwatch-dashboard.rendered.json
 samconfig.toml
 .aws-sam
 .env.local.json
-events/my.event.json
+events/my.event.json
+lambda/tests/.pytest_cache
+lambda/tests/test_db
+lambda/tests/__pycache__
+lambda/__pycache__

README.md

@@ -18,7 +18,6 @@ This repository provides you with a sample solution that collects metrics of exi
 ### Solution Tenets
 * Solution is designed to provide time-series metrics for Apache Iceberg to monitor Apache Iceberg tables over-time to recognize trends and anomalies. 
 * Solution is designed to be lightweight and collect metrics exclusively from Apache Iceberg metadata layer without scanning the data layer hense without the need for heavy compute capacity.
-* In the future we strive to reduce the dependency on AWS Glue in favor of using AWS Lambda compute when required features are available in [PyIceberg](https://py.iceberg.apache.org) library.
 
 ### Technical implementation
 
@@ -90,7 +89,7 @@ https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/i
 
 ### Build and Deploy
 
-> ! Important - The guidance below uses AWS Serverless Application Model (SAM) for easier packaging and deployment of AWS Lambda. However if you use your own packaging tool or if you want to deploy AWS Lambda manually you can explore following files:
+> ! Important - The guidance below uses AWS Serverless Application Model (SAM) and Amazon ECR for easier packaging and deployment of AWS Lambda. However if you use your own packaging tool or if you want to deploy AWS Lambda manually you can explore following files:
 > - template.yaml
 > - lambda/requirements.txt
 > - lambda/app.py
@@ -100,22 +99,30 @@ https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/i
 Once you've installed [Docker](#install-docker) and [SAM CLI](#install-sam-cli) you are ready to build the AWS Lambda. Open your terminal and run command below.
 
 ```bash
+cd  lambda
+docker build -f Dockerfile --platform linux/amd64 -t iceberg-monitoring:main --build-arg CLOUDWATCH_NAMESPACE={{ cw_namespace }} .
 sam build --use-container
 ```
 
-#### 2. Deploy AWS Lambda using AWS SAM CLI
+#### 2. Deploy AWS Lambda using AWS SAM CLI and Amazon ECR
 
-Once build is finished you can deploy your AWS Lambda. SAM will upload packaged code and deploy AWS Lambda resource using AWS CloudFormation. Run below command using your terminal.
+Once build is finished you can deploy your AWS Lambda. ECR will upload packaged code and SAM will deploy AWS Lambda resource using AWS CloudFormation. Run below command using your terminal.
 
 ```bash
-sam deploy --guided
+aws ecr get-login-password --region {{ aws_region }} | docker login --username AWS --password-stdin {{ aws_account_id }}.dkr.ecr.us-east-1.amazonaws.com
+aws ecr create-repository --repository-name iceberg-monitoring --region {{ aws_region }} --image-scanning-configuration scanOnPush=true --image-tag-mutability MUTABLE
+docker tag iceberg-monitoring:main {{ ecr_repository_uri }}:latest
+docker push {{ aws_account_id }}.dkr.ecr.{{ aws_region }}.amazonaws.com/iceberg-monitoring:latest
+sam deploy --debug --region {{ aws_region }} \
+        --parameter-overrides ImageURL={{ aws_account_id }}.dkr.ecr.{{ aws_region }}.amazonaws.com/iceberg-monitoring:latest \
+        --image-repository {{ aws_account_id }}.dkr.ecr.{{ aws_region }}.amazonaws.com/iceberg-monitoring \
+        --stack-name iceberg-monitoring --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND \
+        --s3-bucket {{ s3_bucket }} --s3-prefix iceberg-monitoring
 ```
 
 ##### Parameters
 
-- `CWNamespace` - A namespace is a container for CloudWatch metrics.
-- `GlueServiceRole` - AWS Glue Role arn you created [earlier](#configuring-iam-permissions-for-aws-glue).
-- `Warehouse` - Required catalog property to determine the root path of the data warehouse on S3. This can be any path on your S3 bucket. Not critical for the solution.
+- `CLOUDWATCH_NAMESPACE` - A namespace is a container for CloudWatch metrics.
 
 
 #### 3. Configure EventBridge Trigger
@@ -235,6 +242,16 @@ sam local invoke IcebergMetricsLambda --env-vars .env.local.json
 `.env.local.json` - The JSON file that contains values for the Lambda function's environment variables. Lambda code is dependent on env vars that you are passing in the deploy section. You need to create the file it and include relevant [parameters](#parameters) before you calling `sam local invoke`.
 
 
+### Unit Tests
+
+You can test the metrics generation locally through unit-tests. From lambda folder - 
+
+```bash
+cd lambda
+docker build -f tests/Dockerfile -t iceberg-metrics-tests .
+docker run --rm iceberg-metrics-tests
+```
+
 ## Dependencies
 
 PyIceberg is a Python implementation for accessing Iceberg tables, without the need of a JVM. \

lambda/Dockerfile

@@ -0,0 +1,12 @@
+FROM public.ecr.aws/lambda/python:3.10
+
+COPY . ${LAMBDA_TASK_ROOT}
+
+# Install the function's dependencies
+RUN pip install --upgrade pip && \
+    pip install -r requirements.txt
+
+ARG CLOUDWATCH_NAMESPACE
+ENV CW_NAMESPACE=$CLOUDWATCH_NAMESPACE
+
+CMD [ "app.lambda_handler" ]