Skip to content

Commit 29bffef

Browse files
mfjarvisnhsdevws
mfjarvis
and
nhsdevws
authored
VED-303: Add Lambda to populate Redis. (#592)
* WIP baseline * lint * unit test record processor DISEASE_MAPPING_FILE_KEY PERMISSIONS_CONFIG_FILE_KEY error handling * makefile tab * rollback * rollbak * rb * rb * lint * rb * rb * rb * lint * lint * new map * tests * lint * test data * key correction * key diseases_to_vacc * json serialise * Read functionality * VED-303: Use the existing build mechanism for the Redis sync lambda. * Read functionality key diseases_to_vacc * VED-303: Add Dockerfile (oops) * VED-303: A little tidying. * VED-303: Fix tests. Tidy up some more. --------- Co-authored-by: nhsdevws <[email protected]>
1 parent b5d4471 commit 29bffef

40 files changed

+2165
-121
lines changed

‎.github/dependabot.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,7 @@ updates:
5757
- "/filenameprocessor"
5858
- "/mesh_processor"
5959
- "/recordprocessor"
60+
- "/redis_sync"
6061
schedule:
6162
interval: "daily"
6263
open-pull-requests-limit: 1

‎.github/workflows/sonarcloud.yml‎

Lines changed: 19 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ jobs:
4343
poetry env use 3.10
4444
poetry install
4545
poetry run coverage run -m unittest discover || echo "filenameprocessor tests failed" >> ../failed_tests.txt
46-
poetry run coverage xml -o ../sonarcloud-coverage-filenameprocessor-coverage.xml
46+
poetry run coverage xml -o ../filenameprocessor-coverage.xml
4747
4848
- name: Run unittest with recordprocessor-coverage
4949
working-directory: recordprocessor
@@ -53,7 +53,7 @@ jobs:
5353
poetry env use 3.10
5454
poetry install
5555
poetry run coverage run -m unittest discover || echo "recordprocessor tests failed" >> ../failed_tests.txt
56-
poetry run coverage xml -o ../sonarcloud-coverage-recordprocessor-coverage.xml
56+
poetry run coverage xml -o ../recordprocessor-coverage.xml
5757
5858
# This step is redundant - all of these tests will be run in the backend step below
5959
- name: Run unittest with recordforwarder-coverage
@@ -64,7 +64,7 @@ jobs:
6464
poetry env use 3.11
6565
poetry install
6666
poetry run coverage run -m unittest discover -s "./tests" -p "*batch*.py" || echo "recordforwarder tests failed" >> ../failed_tests.txt
67-
poetry run coverage xml -o ../sonarcloud-coverage-recordforwarder-coverage.xml
67+
poetry run coverage xml -o ../recordforwarder-coverage.xml
6868
6969
- name: Run unittest with coverage-ack-lambda
7070
working-directory: ack_backend
@@ -74,7 +74,7 @@ jobs:
7474
poetry env use 3.10
7575
poetry install
7676
poetry run coverage run -m unittest discover || echo "ack-lambda tests failed" >> ../failed_tests.txt
77-
poetry run coverage xml -o ../sonarcloud-coverage-ack-lambda.xml
77+
poetry run coverage xml -o ../ack-lambda-coverage.xml
7878
7979
- name: Run unittest with coverage-delta
8080
working-directory: delta_backend
@@ -86,7 +86,7 @@ jobs:
8686
poetry env use 3.11
8787
poetry install
8888
poetry run coverage run -m unittest discover || echo "delta tests failed" >> ../failed_tests.txt
89-
poetry run coverage xml -o ../sonarcloud-coverage-delta.xml
89+
poetry run coverage xml -o ../delta-coverage.xml
9090
9191
- name: Run unittest with coverage-fhir-api
9292
working-directory: backend
@@ -96,7 +96,7 @@ jobs:
9696
poetry env use 3.11
9797
poetry install
9898
poetry run coverage run -m unittest discover || echo "fhir-api tests failed" >> ../failed_tests.txt
99-
poetry run coverage xml -o ../sonarcloud-coverage.xml
99+
poetry run coverage xml -o ../backend-coverage.xml
100100
101101
- name: Run unittest with coverage-mesh-processor
102102
working-directory: mesh_processor
@@ -106,7 +106,19 @@ jobs:
106106
poetry env use 3.10
107107
poetry install
108108
poetry run coverage run -m unittest discover || echo "mesh_processor tests failed" >> ../failed_tests.txt
109-
poetry run coverage xml -o ../sonarcloud-mesh_processor-coverage.xml
109+
poetry run coverage xml -o ../mesh_processor-coverage.xml
110+
111+
- name: Run unittest with redis_sync
112+
working-directory: redis_sync
113+
id: redis_sync
114+
env:
115+
PYTHONPATH: ${{ github.workspace }}/redis_sync/src:${{ github.workspace }}/redis_sync/tests
116+
continue-on-error: true
117+
run: |
118+
poetry env use 3.11
119+
poetry install
120+
poetry run coverage run -m unittest discover || echo "redis_sync tests failed" >> ../failed_tests.txt
121+
poetry run coverage xml -o ../redis_sync-coverage.xml
110122
111123
- name: Run Test Failure Summary
112124
id: check_failure

‎Makefile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ build-proxy:
3838
scripts/build_proxy.sh
3939

4040
#Files to loop over in release
41-
_dist_include="pytest.ini poetry.lock poetry.toml pyproject.toml Makefile build/. e2e e2e_batch specification sandbox terraform scripts backend delta_backend ack_backend filenameprocessor recordprocessor mesh_processor"
41+
_dist_include="pytest.ini poetry.lock poetry.toml pyproject.toml Makefile build/. e2e e2e_batch specification sandbox terraform scripts backend delta_backend ack_backend filenameprocessor recordprocessor mesh_processor redis_sync"
4242

4343

4444
#Create /dist/ sub-directory and copy files into directory

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ See https://nhsd-confluence.digital.nhs.uk/display/APM/Glossary.
2323
| `filenameprocessor` | **Imms Batch** – Processes batch file names. |
2424
| `mesh_processor` | **Imms Batch** – MESH-specific batch processing functionality. |
2525
| `recordprocessor` | **Imms Batch** – Handles batch record processing. |
26+
| `redis_sync` | **Imms Redis** – Handles sync s3 to REDIS. |
2627
---
2728

2829
### Pipelines

‎immunisation-fhir-api.code-workspace‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,8 +26,10 @@
2626
},
2727
{
2828
"path": "e2e_batch"
29-
}
30-
29+
},
30+
{
31+
"path": "redis_sync"
32+
}
3133
],
3234
"settings": {},
33-
}
35+
}

‎recordprocessor/Makefile‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,12 +8,12 @@ test:
88
python -m unittest
99

1010
coverage run:
11-
coverage run -m unittest discover
11+
coverage run -m unittest discover
1212

1313
coverage report:
1414
coverage report -m
1515

1616
coverage html:
17-
coverage html
17+
coverage html
1818

19-
.PHONY: build package
19+
.PHONY: build package

‎redis_sync/Dockerfile‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
FROM public.ecr.aws/lambda/python:3.11 AS base
2+
3+
# Create non-root user
4+
RUN mkdir -p /home/appuser && \
5+
echo 'appuser:x:1001:1001::/home/appuser:/sbin/nologin' >> /etc/passwd && \
6+
echo 'appuser:x:1001:' >> /etc/group && \
7+
chown -R 1001:1001 /home/appuser && pip install "poetry~=1.5.0"
8+
9+
# Install Poetry as root
10+
COPY poetry.lock pyproject.toml README.md ./
11+
RUN poetry config virtualenvs.create false && poetry install --no-interaction --no-ansi --no-root --only main
12+
# -----------------------------
13+
FROM base AS test
14+
COPY src src
15+
COPY tests tests
16+
RUN poetry install --no-interaction --no-ansi --no-root && \
17+
pytest --disable-warnings tests
18+
19+
# -----------------------------
20+
FROM base AS build
21+
COPY src .
22+
RUN chmod 644 $(find . -type f) && chmod 755 $(find . -type d)
23+
# Build as non-root user
24+
USER 1001:1001
25+
CMD ["redis_sync.handler"]

‎redis_sync/Makefile‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
2+
test:
3+
@PYTHONPATH=src:tests python -m unittest
4+
5+
coverage-run:
6+
coverage run -m unittest discover -v
7+
8+
coverage-report:
9+
coverage report -m
10+
11+
coverage-html:
12+
coverage html
13+
14+
.PHONY: build package

‎redis_sync/README.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# Redis Sync Lambda
2+
3+
## Overview
4+
5+
**Redis Sync** is an AWS Lambda function designed to monitor a configuration S3 bucket and synchronize its contents with an ElastiCache Redis instance. Whenever a new configuration file is uploaded or updated in the S3 bucket, the Lambda function processes the file, applies any required transformations, and uploads the result to the Redis cache.
6+
7+
## Features
8+
9+
- **S3 Event Driven:** Automatically triggered by S3 events (e.g., file uploads or updates) in the config bucket.
10+
- **Transformation Support:** Applies custom transformation logic to configuration files before caching.
11+
- **Redis Integration:** Uploads processed configuration data to ElastiCache Redis for fast, centralized access.
12+
- **Logging:** Provides detailed logging for monitoring and troubleshooting.
13+
14+
## How It Works
15+
16+
1. **S3 Event Trigger:** The Lambda is triggered by S3 events on the config bucket.
17+
2. **File Processing:** The Lambda reads the new or updated file from S3.
18+
3. **Transformation:** If required, the file content is transformed to the appropriate format.
19+
4. **Redis Upload:** The transformed data is uploaded to the Redis cache under a key corresponding to the file.
20+
5. **Monitoring:** Logs are generated for each step, aiding in monitoring and debugging.
21+
22+
## Configuration
23+
24+
- **Environment Variables:**
25+
- `CONFIG_BUCKET_NAME`: Name of the S3 bucket to monitor.
26+
- `AWS_REGION`: AWS region for S3 and Redis.
27+
- `REDIS_HOST`: Redis endpoint.
28+
- `REDIS_PORT`: Redis port (default: 6379).
29+
30+
## Usage
31+
32+
1. **Deploy the Lambda** using your preferred IaC tool (e.g., Terraform, AWS SAM).
33+
2. **Configure S3 event notifications** to trigger the Lambda on object creation or update.
34+
3. **Ensure Redis and S3 permissions** are set for the Lambda execution role.
35+
36+
## Development
37+
38+
- Code is located in the `src/` directory.
39+
- Unit tests are in the `tests/` directory.
40+
- Use the provided Makefile and Dockerfile for building, testing, and packaging.
41+
42+
## License
43+
44+
This project is maintained by NHS. See [LICENSE](../LICENSE) for details.

0 commit comments

Comments
 (0)