|
1 | | -# Redis Sync Lambda |
| 1 | +# Shared Lambda Components |
2 | 2 |
|
3 | 3 | ## Overview |
4 | 4 |
|
5 | | -**Redis Sync** is an AWS Lambda function designed to monitor a configuration S3 bucket and synchronize its contents with an ElastiCache Redis instance. Whenever a new configuration file is uploaded or updated in the S3 bucket, the Lambda function processes the file, applies any required transformations, and uploads the result to the Redis cache. |
| 5 | +**Shared Lambda Components** is a reusable library containing common utilities, clients, and helper functions designed to be shared across multiple AWS Lambda functions in the immunisation FHIR API project. This package provides standardized implementations for database connections, AWS service clients, logging, validation, and other common operations. |
| 6 | + |
| 7 | +## Purpose |
| 8 | + |
| 9 | +This shared library promotes: |
| 10 | +- **Code Reusability:** Common functionality used across multiple Lambda functions |
| 11 | +- **Consistency:** Standardized patterns for AWS service interactions |
| 12 | +- **Maintainability:** Centralized location for shared utilities and configurations |
| 13 | +- **Efficiency:** Reduced code duplication and faster development |
6 | 14 |
|
7 | 15 | ## Features |
8 | 16 |
|
9 | | -- **S3 Event Driven:** Automatically triggered by S3 events (e.g., file uploads or updates) in the config bucket. |
10 | | -- **Transformation Support:** Applies custom transformation logic to configuration files before caching. |
11 | | -- **Redis Integration:** Uploads processed configuration data to ElastiCache Redis for fast, centralized access. |
12 | | -- **Logging:** Provides detailed logging for monitoring and troubleshooting. |
| 17 | +- **AWS Service Clients:** Pre-configured clients for DynamoDB, S3, SQS, SNS, and other AWS services |
| 18 | +- **Database Utilities:** Common database operations and connection management |
| 19 | +- **Logging Framework:** Standardized logging configuration and utilities |
| 20 | +- **Validation Helpers:** Input validation and data transformation functions |
| 21 | +- **Error Handling:** Common exception classes and error handling patterns |
| 22 | +- **Configuration Management:** Environment variable handling and configuration utilities |
| 23 | + |
| 24 | +## Components |
13 | 25 |
|
14 | | -## How It Works |
| 26 | +### Core Modules |
15 | 27 |
|
16 | | -1. **S3 Event Trigger:** The Lambda is triggered by S3 events on the config bucket. |
17 | | -2. **File Processing:** The Lambda reads the new or updated file from S3. |
18 | | -3. **Transformation:** If required, the file content is transformed to the appropriate format. |
19 | | -4. **Redis Upload:** The transformed data is uploaded to the Redis cache under a key corresponding to the file. |
20 | | -5. **Monitoring:** Logs are generated for each step, aiding in monitoring and debugging. |
| 28 | +- **`clients.py`** - AWS service clients (DynamoDB, S3, SQS, etc.) |
| 29 | +- **`logging_utils.py`** - Centralized logging configuration |
| 30 | +- **`validators.py`** - Input validation and data transformation |
| 31 | +- **`exceptions.py`** - Custom exception classes |
| 32 | +- **`config.py`** - Environment configuration management |
| 33 | +- **`utils.py`** - General utility functions |
21 | 34 |
|
22 | | -## Configuration |
| 35 | +### Database Operations |
23 | 36 |
|
24 | | -- **Environment Variables:** |
25 | | - - `CONFIG_BUCKET_NAME`: Name of the S3 bucket to monitor. |
26 | | - - `AWS_REGION`: AWS region for S3 and Redis. |
27 | | - - `REDIS_HOST`: Redis endpoint. |
28 | | - - `REDIS_PORT`: Redis port (default: 6379). |
| 37 | +- **`db_operations.py`** - Common database query patterns |
| 38 | +- **`table_utils.py`** - DynamoDB table management utilities |
| 39 | + |
| 40 | +### AWS Integrations |
| 41 | + |
| 42 | +- **`s3_utils.py`** - S3 file operations |
| 43 | +- **`sqs_utils.py`** - SQS message handling |
| 44 | +- **`sns_utils.py`** - SNS notification utilities |
29 | 45 |
|
30 | 46 | ## Usage |
31 | 47 |
|
32 | | -1. **Deploy the Lambda** using your preferred IaC tool (e.g., Terraform, AWS SAM). |
33 | | -2. **Configure S3 event notifications** to trigger the Lambda on object creation or update. |
34 | | -3. **Ensure Redis and S3 permissions** are set for the Lambda execution role. |
| 48 | +### In Lambda Functions |
| 49 | + |
| 50 | +Import shared components in your Lambda functions: |
| 51 | + |
| 52 | +```python |
| 53 | +# Import AWS clients |
| 54 | +from shared.clients import dynamodb_resource, s3_client |
| 55 | + |
| 56 | +# Import utilities |
| 57 | +from shared.logging_utils import setup_logger |
| 58 | +from shared.validators import validate_nhs_number |
| 59 | +from shared.exceptions import ValidationException |
| 60 | + |
| 61 | +# Use in your Lambda function |
| 62 | +logger = setup_logger(__name__) |
| 63 | + |
| 64 | +def lambda_handler(event, context): |
| 65 | + try: |
| 66 | + # Use shared DynamoDB client |
| 67 | + table = dynamodb_resource.Table('my-table') |
| 68 | + |
| 69 | + # Use shared validation |
| 70 | + nhs_number = validate_nhs_number(event.get('nhs_number')) |
| 71 | + |
| 72 | + # Your Lambda logic here... |
| 73 | + |
| 74 | + except ValidationException as e: |
| 75 | + logger.error(f"Validation error: {e}") |
| 76 | + return {"statusCode": 400, "body": str(e)} |
| 77 | +``` |
| 78 | + |
| 79 | +### Installation as Lambda Layer |
| 80 | + |
| 81 | +This package is designed to be deployed as an AWS Lambda Layer for optimal reuse: |
| 82 | + |
| 83 | +```bash |
| 84 | +# Build the layer package |
| 85 | +make build-layer |
| 86 | + |
| 87 | +# Deploy as Lambda Layer (future implementation) |
| 88 | +make deploy-layer |
| 89 | +``` |
35 | 90 |
|
36 | 91 | ## Development |
37 | 92 |
|
38 | | -- Code is located in the `src/` directory. |
39 | | -- Unit tests are in the `tests/` directory. |
40 | | -- Use the provided Makefile and Dockerfile for building, testing, and packaging. |
| 93 | +### Project Structure |
| 94 | + |
| 95 | +``` |
| 96 | +shared/ |
| 97 | +├── src/ |
| 98 | +│ ├── common/ |
| 99 | +│ │ ├── clients.py |
| 100 | +│ │ ├── logging_utils.py |
| 101 | +│ │ ├── validators.py |
| 102 | +│ │ ├── exceptions.py |
| 103 | +│ │ └── config.py |
| 104 | +│ ├── database/ |
| 105 | +│ │ ├── db_operations.py |
| 106 | +│ │ └── table_utils.py |
| 107 | +│ └── aws/ |
| 108 | +│ ├── s3_utils.py |
| 109 | +│ ├── sqs_utils.py |
| 110 | +│ └── sns_utils.py |
| 111 | +├── tests/ |
| 112 | +│ ├── test_clients.py |
| 113 | +│ ├── test_validators.py |
| 114 | +│ └── test_utils.py |
| 115 | +├── requirements.txt |
| 116 | +├── Makefile |
| 117 | +└── README.md |
| 118 | +``` |
| 119 | + |
| 120 | +### Testing |
| 121 | + |
| 122 | +```bash |
| 123 | +# Run all tests |
| 124 | +make test |
| 125 | + |
| 126 | +# Run specific test |
| 127 | +python -m pytest tests/test_clients.py |
| 128 | + |
| 129 | +# Run with coverage |
| 130 | +make test-coverage |
| 131 | +``` |
| 132 | + |
| 133 | +### Building |
| 134 | + |
| 135 | +```bash |
| 136 | +# Install dependencies |
| 137 | +make install |
| 138 | + |
| 139 | +# Run linting |
| 140 | +make lint |
| 141 | + |
| 142 | +# Format code |
| 143 | +make format |
| 144 | + |
| 145 | +# Build package |
| 146 | +make build |
| 147 | +``` |
| 148 | + |
| 149 | +## Future Implementation: Lambda Layer Deployment |
| 150 | + |
| 151 | +This shared library will be packaged and deployed as an AWS Lambda Layer to: |
| 152 | + |
| 153 | +- **Reduce deployment package sizes** for individual Lambda functions |
| 154 | +- **Enable version management** of shared components |
| 155 | +- **Improve cold start performance** through layer caching |
| 156 | +- **Simplify dependency management** across Lambda functions |
| 157 | + |
| 158 | +### Planned Layer Structure |
| 159 | + |
| 160 | +``` |
| 161 | +lambda-layer/ |
| 162 | +├── python/ |
| 163 | +│ └── shared/ |
| 164 | +│ ├── common/ |
| 165 | +│ ├── database/ |
| 166 | +│ └── aws/ |
| 167 | +└── layer.zip |
| 168 | +``` |
| 169 | + |
| 170 | +### Usage with Layer |
| 171 | + |
| 172 | +Once deployed as a layer, Lambda functions will reference it: |
| 173 | + |
| 174 | +```python |
| 175 | +# After layer deployment, import from layer |
| 176 | +from shared.common.clients import dynamodb_resource |
| 177 | +from shared.common.logging_utils import setup_logger |
| 178 | +``` |
| 179 | + |
| 180 | +## Environment Variables |
| 181 | + |
| 182 | +Common environment variables used by shared components: |
| 183 | + |
| 184 | +- `AWS_REGION` - AWS region for service clients |
| 185 | +- `LOG_LEVEL` - Logging level (DEBUG, INFO, WARN, ERROR) |
| 186 | +- `ENVIRONMENT` - Environment name (dev, test, prod) |
| 187 | + |
| 188 | +## Dependencies |
| 189 | + |
| 190 | +See `requirements.txt` for Python package dependencies: |
| 191 | +- `boto3` - AWS SDK |
| 192 | +- `botocore` - AWS core library |
| 193 | +- Additional utilities as needed |
| 194 | + |
| 195 | +## Contributing |
| 196 | + |
| 197 | +1. Follow existing code patterns and naming conventions |
| 198 | +2. Add unit tests for new functionality |
| 199 | +3. Update documentation for new components |
| 200 | +4. Ensure backward compatibility when making changes |
| 201 | + |
| 202 | +## Version Management |
| 203 | + |
| 204 | +- **Semantic versioning** for layer releases |
| 205 | +- **Changelog** documentation for version history |
| 206 | +- **Compatibility matrix** for Lambda runtime versions |
41 | 207 |
|
42 | 208 | ## License |
43 | 209 |
|
|
0 commit comments