[sdlf-datalakelibrary] simplified datalake-library (#540)

* [sdlf-datalakelibrary] simplified datalake-library

* [sdlf-stage-lambda/glue] add DEPRECATED title in README

* [sdlf-stage-A/B] simplify and update with datalakeLibrary

Author: dlpzx

sdlf-datalakeLibrary/README.md

@@ -1,36 +1,151 @@
 # Datalake Library
-The data lake library repository is where a team pushes the transformation code (i.e. business logic) that they wish to apply to their datasets. After each new commit, the repository is automatically packaged into a lambda layer and mounted to the individual Lambda functions of the pipelines belonging to the team. The repository also holds helper functions that automate boiler plate code such as SQS, S3, and DynamoDB operations.
-
-## IMPORTANT
-Please ensure that you follow this file structure, with a folder named `python` at the root containing all the Lambda code that should be part of the Layer. The automated build process depends on the file structure being as follows: 
- 
-    ./
-        ├── README.md
-        └── python
-            └── datalake_library
-                ├── configuration
-                ├── interfaces
-                ├── octagon
-                ├── tests                
-                └── transforms
-                    ├── stage_a_transforms
-                    │   ├── light_transform_blueprint.py
-                    │   ├── ...
-                    ├── stage_b_transforms
-                    │   ├── heavy_transform_blueprint.py
-                    │   └── ...
-                    └── transform_handler.py
- 
-## Adding Transformations 
-When adding custom transformations to the Lambda Layer, simply add your code to this repository (see example of `light_transform_blueprint.py` in file structure above) in the relevant location (e.g. stage_a_transforms for light transformations in StageA). Any changes to this repository should stay in branches while in development, and once tested/stable, these changes can then be merged into the relevant environment branch (`dev, test or main`). The pipeline will trigger upon commits made to this branch, and release these changes automatically.
+
+A modern, unified Python library for AWS data lake operations. The datalake_library provides clean interfaces for S3, DynamoDB, Step Functions, SQS, and KMS operations with shared boto3 sessions for optimal performance.
+
+## Architecture
+
+The library is built around a unified `DataLakeClient` that provides access to all AWS services through dedicated interfaces:
+
+```
+datalake_library/
+├── interfaces/                    # Service-specific interfaces
+│   ├── base_interface.py         # BaseInterface
+│   ├── s3_interface.py           # S3 operations
+│   ├── dynamo_interface.py       # DynamoDB operations  
+│   ├── states_interface.py       # Step Functions operations
+│   ├── sqs_interface.py          # SQS operations
+│   └── kms_interface.py          # KMS operations
+├── client.py                     # Unified DataLakeClient
+├── __init__.py                   # Public API exports
+└── commons.py                    # Shared utilities
+```
+
+## Installation
+
+The library is packaged as a Lambda Layer and automatically mounted to Lambda functions. No manual installation required.
+
+## Usage
+
+### Basic Usage
+
+```python
+from datalake_library import DataLakeClient
+
+# Initialize client with team/pipeline/stage context
+client = DataLakeClient(team="engineering", pipeline="stage", stage="a")
+
+# Access all services through the client
+client.s3.download_object(bucket, key)
+client.dynamo.update_object_metadata_catalog(metadata)
+client.states.run_state_machine(arn, payload)
+client.sqs.send_message_to_fifo_queue(message, group_id, queue_url)
+client.kms.data_kms_key
+```
+
+### Individual Interfaces
+
+```python
+from datalake_library import S3Interface, DynamoInterface
+
+# Use individual interfaces with shared session
+import boto3
+session = boto3.Session()
+
+s3 = S3Interface(team="engineering", session=session)
+dynamo = DynamoInterface(team="engineering", session=session)
+```
+
+## API Reference
+
+### DataLakeClient
+
+```python
+client = DataLakeClient(
+    team="engineering",      # Team name for configuration lookup
+    dataset="legislators",   # Dataset name (optional)
+    pipeline="stage",        # Pipeline name
+    stage="a",              # Stage name
+    log_level="INFO",       # Logging level (optional)
+    session=None            # Boto3 session (optional)
+)
+```
+
+### S3Interface
+
+```python
+# Download object to /tmp
+local_path = client.s3.download_object(bucket, key)
+
+# Upload object with KMS encryption
+client.s3.upload_object(local_path, bucket, key, kms_key=kms_key)
+
+# Copy object between buckets
+client.s3.copy_object(src_bucket, src_key, dest_bucket, dest_key)
+
+# Get object metadata
+size, last_modified = client.s3.get_size_and_last_modified(bucket, key)
+
+# Access bucket names
+client.s3.raw_bucket
+client.s3.stage_bucket
+client.s3.analytics_bucket
+```
+
+### DynamoInterface
+
+```python
+# Update object metadata catalog
+metadata = {
+    "bucket": bucket,
+    "key": key,
+    "team": team,
+    "dataset": dataset
+}
+client.dynamo.update_object_metadata_catalog(metadata)
+
+# Put item to any table
+client.dynamo.put_item(table_name, item)
+
+# Access table names
+client.dynamo.object_metadata_table
+client.dynamo.manifests_table
+```
+
+### StatesInterface
+
+```python
+# Run state machine
+response = client.states.run_state_machine(arn, payload)
+
+# Access state machine ARN
+client.states.state_machine_arn
+```
+
+### SQSInterface
+
+```python
+# Send message to FIFO queue
+client.sqs.send_message_to_fifo_queue(message, group_id, queue_url)
+
+# Receive messages
+messages = client.sqs.receive_messages(max_messages, queue_url)
+
+# Access queue URLs
+client.sqs.stage_queue_url
+client.sqs.stage_dlq_url
+```
+
+### KMSInterface
+
+```python
+# Access KMS key ARN
+kms_key = client.kms.data_kms_key
+```
 
 ## Pipeline
-The CICD pipeline for this repository is defined in the `sdlf-team` repository for each team (`nested-stacks/template-cicd.yaml`). A CodeBuild job is used to package the code in this repository into a `.zip` file, while leaving out any `__pycache__` files, which is then published as a Lambda Layer. Due to limitations on the size of packages, the code in this repository must not exceed 50MB when zipped, and no more than 250Mb unzipped.
-
-Configuration details, e.g. the name of the Lambda Layer built from this repository, will be defined in the template containing the **sdlf-pipeline** infrastructure. Some of the configuration details available for customization:
-1. Through the pipeline:
-   1. Main Git branch to use — currently set to `dev`
-2. Through the CodeBuild job:
-   1. Name of the resulting Layer
-   2. Compatible runtimes
-   3. SSM parameter used to store the ARN of the latest version
+
+The library is automatically packaged into a Lambda Layer via CodeBuild when changes are committed to the environment branch (`dev`, `test`, or `main`). The layer is then made available to all Lambda functions in the data lake pipeline.
+
+**Size Limits:**
+- Zipped: 50MB maximum
+- Unzipped: 250MB maximum

sdlf-datalakeLibrary/python/datalake_library/__init__.py

@@ -0,0 +1,4 @@
+from .client import DataLakeClient
+from .interfaces import DynamoInterface, KMSInterface, S3Interface, SQSInterface, StatesInterface
+
+__all__ = ["DataLakeClient", "S3Interface", "DynamoInterface", "StatesInterface", "SQSInterface", "KMSInterface"]

sdlf-datalakeLibrary/python/datalake_library/client.py

@@ -0,0 +1,19 @@
+import boto3
+
+from .interfaces import DynamoInterface, KMSInterface, S3Interface, SQSInterface, StatesInterface
+
+
+class DataLakeClient:
+    def __init__(self, team=None, dataset=None, pipeline=None, stage=None, log_level=None, session=None):
+        """
+        Unified client for all data lake operations with shared boto3 session
+        """
+        # Shared session across all interfaces
+        self.session = session or boto3.Session()
+
+        # Initialize all interfaces with shared session
+        self.s3 = S3Interface(team, dataset, pipeline, stage, log_level, self.session)
+        self.dynamo = DynamoInterface(team, dataset, pipeline, stage, log_level, self.session)
+        self.states = StatesInterface(team, dataset, pipeline, stage, log_level, self.session)
+        self.sqs = SQSInterface(team, dataset, pipeline, stage, log_level, self.session)
+        self.kms = KMSInterface(team, dataset, pipeline, stage, log_level, self.session)

sdlf-datalakeLibrary/python/datalake_library/commons.py

@@ -1,7 +1,7 @@
 import logging
 from typing import TYPE_CHECKING, Any, Dict, Mapping, Optional
 
-from boto3.dynamodb.types import TypeDeserializer, TypeSerializer
+from boto3.dynamodb.types import TypeSerializer
 
 if TYPE_CHECKING:
     from mypy_boto3_dynamodb.type_defs import (
@@ -23,10 +23,3 @@ def serialize_dynamodb_item(
 ) -> Dict[str, "AttributeValueTypeDef"]:
     serializer = serializer if serializer else TypeSerializer()
     return {k: serializer.serialize(v) for k, v in item.items()}
-
-
-def deserialize_dynamodb_item(
-    item: Mapping[str, "AttributeValueTypeDef"], deserializer: Optional[TypeDeserializer] = None
-) -> Dict[str, Any]:
-    deserializer = deserializer if deserializer else TypeDeserializer()
-    return {k: deserializer.deserialize(v) for k, v in item.items()}