You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
These tools automate the Python version and environment creation and use, and selects them automatically when entering a
35
-
directory.
36
-
`pyenv` separates Python versions and `direnv` handles the environment.
37
-
`direnv` uses `pyenv` to select the version and creates an environment under `.direnv/`.
38
-
39
-
At this point you'll get a warning when you enter this directory, telling you to run `direnv allow`. This is fine and
40
-
the next step will resolve it.
41
-
42
-
### Install Python versions and environments
43
-
44
-
This will set up for both the root and `backend`.
13
+
## Directories
14
+
**Note:** Each Lambda has its own `README.md` file for detailed documentation. For non-Lambda-specific folders, refer to `README.specification.md`.
15
+
16
+
### Lambdas
17
+
18
+
| Folder | Description |
19
+
|---------------------|-------------|
20
+
|`backend`|**Imms API** – Handles CRUD operations for the Immunisation API. |
21
+
|`delta_backend`|**Imms Sync** – Lambda function that reacts to events in the Immunisation database. |
22
+
|`ack_backend`|**Imms Batch** – Generates the final Business Acknowledgment (BUSACK) file from processed messages and writes it to the designated S3 location. |
|`recordprocessor`|**Imms Batch** – Handles batch record processing. |
26
+
---
27
+
28
+
### Pipelines
29
+
30
+
| Folder | Description |
31
+
|---------|-------------|
32
+
|`azure`| Pipeline definition and orchestration code. |
33
+
34
+
---
35
+
36
+
### Infrastructure
37
+
38
+
| Folder | Description |
39
+
|------------------------|-------------|
40
+
|`infra`| Base infrastructure components. |
41
+
|`grafana`| Terraform configuration for Grafana, built on top of core infra. |
42
+
|`terraform`| Core Terraform infrastructure code. This is run in each PR and sets up lambdas associated with the PR.|
43
+
|`terraform_sandbox`| Sandbox environment for testing infrastructure changes. |
44
+
|`terraform_aws_backup`| Streamlined backup processing with AWS. |
45
+
|`mesh-infra`| Infrastructure setup for Imms batch MESH integration. |
46
+
|`proxies`| Apigee API proxy definitions. |
47
+
---
48
+
49
+
### Tests
50
+
51
+
| Folder | Description |
52
+
|---------------|-------------|
53
+
|`e2e`| End-to-end tests executed during PR pipelines. |
54
+
|`e2e_batch`| E2E tests specifically for batch-related functionality, also run in the PR pipeline. |
55
+
|`tests`| Sample e2e test. |
56
+
---
57
+
58
+
### Utilities
59
+
60
+
| Folder | Description |
61
+
|----------------|-------------|
62
+
|`devtools`| Helper tools and utilities for local development |
63
+
|`scripts`| Standalone or reusable scripts for development and automation |
64
+
|`specification`| Specification files to document API and related definitions |
65
+
|`sandbox`| Simple sandbox API |
66
+
67
+
---
68
+
69
+
## Background: Python Package Management and Virtual Environments
70
+
-`pyenv` manages multiple Python versions at the system level, allowing you to install and switch between different Python versions for different projects.
71
+
-`direnv` automates the loading of environment variables and can auto-activate virtual environments (.venv) when entering a project directory, making workflows smoother.
72
+
-`.venv` (created via python -m venv or poetry) is Python’s built-in tool for isolating dependencies per project, ensuring that packages don’t interfere with global Python packages.
73
+
-`Poetry` is an all-in-one dependency and virtual environment manager that automatically creates a virtual environment (.venv), manages package installations, and locks dependencies (poetry.lock) for reproducibility, making it superior to using pip manually and it is used in all the lambda projects.
74
+
75
+
## Project structure
76
+
To support a modular and maintainable architecture, each Lambda function in this project is structured as a self-contained folder with its own dependencies, configuration, and environment.
77
+
78
+
We use Poetry to manage dependencies and virtual environments, with the virtualenvs.in-project setting enabled to ensure each Lambda has an isolated `.venv` created within its folder.
79
+
80
+
Additionally, direnv is used alongside `.envrc` and `.env` files to automatically activate the appropriate virtual environment and load environment-specific variables when entering a folder.
81
+
82
+
Each Lambda folder includes its own `.env` file for Lambda-specific settings, while the project root contains a separate `.env` and `.venv` for managing shared tooling, scripts, or infrastructure-related configurations. This setup promotes clear separation of concerns, reproducibility across environments, and simplifies local development, testing, and packaging for deployment.
83
+
84
+
## Environment setup
85
+
These dependencies are required for running and debugging the Lambda functions and end-to-end (E2E) tests.
86
+
87
+
### Install dependencies
88
+
Steps:
89
+
1. Install [WSL](https://learn.microsoft.com/en-us/windows/wsl/install) if running on Windows and install [Docker](https://docs.docker.com/engine/install/).
90
+
2. Install the following tools inside WSL. These will be used by the lambda and infrastructure code:
The steps below must be performed in each Lambda function folder and e2e folder to ensure the environment is correctly configured.
119
+
120
+
For detailed instructions on running individual Lambdas, refer to the README.md files located inside each respective Lambda folder.
121
+
122
+
Steps:
123
+
1. Set the python version in the folder with the code used by lambda for example `./backend` (see [lambdas](#lambdas)) folder.
124
+
```
125
+
pyenv local 3.10.16 # Set version in backend (this creates a .python-version file)
126
+
```
127
+
128
+
2. Configure poetry
129
+
```
130
+
### Point poetry virtual environment to .venv
131
+
poetry config virtualenvs.in-project true
132
+
poetry env use $(pyenv which python)
133
+
poetry env info
134
+
```
135
+
136
+
3. Create an .env file and add environment variables.
137
+
```
138
+
AWS_PROFILE={your_profile}
139
+
IMMUNIZATION_ENV=local
140
+
```
141
+
142
+
4. Configure `direnv` by creating a `.envrc` file in the backend folder. This points direnv to the `.venv` created by poetry and loads env variables specified in the `.env` file
143
+
```
144
+
export VIRTUAL_ENV=".venv"
145
+
PATH_add "$VIRTUAL_ENV/bin"
146
+
147
+
dotenv
148
+
```
149
+
150
+
5. Restart bash and run `direnv allow`. You should see something similar like:
Test if environment variables have been loaded into shell: `echo $IMMUNIZATION_ENV`.
156
+
157
+
### Setting up the root level environment
158
+
The root-level virtual environment is primarily used for linting, as we create separate virtual environments for each folder that contains Lambda functions.
159
+
Steps:
160
+
1. Follow instructions above to [install dependencies](#install-dependencies) & [set up a virtual environment](#setting-up-a-virtual-environment-with-poetry).
161
+
**Note: While this project uses Python 3.10 (e.g. for Lambdas), the NHSDigital/api-management-utils repository — which orchestrates setup and linting — defaults to Python 3.8.
162
+
The linting command is executed from within that repo but calls the Makefile in this project, so be aware of potential Python version mismatches when running or debugging locally or in the pipeline.**
163
+
2. Run `make lint`. This will:
164
+
- Check the linting of the API specification yaml.
165
+
- Run Flake8 on all Python files in the repository, excluding files inside .venv and .terraform directories.
166
+
167
+
## IDE setup
168
+
The current team uses VS Code mainly. So this setup is targeted towards VS code. If you use another IDE please add the documentation to set up workspaces here.
169
+
170
+
### VS Code
45
171
46
-
Copy `.env.default` to `.env` or merge it with your existing file.
47
-
Copy `.envrc.default` to `.envrc` or merge it with your existing file.
172
+
The project must be opened as a multi-root workspace for VS Code to know that `backend` has its own environment.
48
173
49
-
These are kept separate so other tools can use `.env` if wanted.
174
+
- Open the workspace `immunisation-fhir-api.code-workspace`.
175
+
- Copy `backend/.vscode/settings.json.default` to `backend/.vscode/settings.json`, or merge the contents with
176
+
your existing file.
50
177
51
-
Edit `.env` with your details.
178
+
VS Code will automatically use the `backend` environment when you're editing a file under `backend`.
52
179
53
-
```shell
54
-
make setup-python-envs
55
-
poetry install --no-root
56
-
```
180
+
Depending on your existing setup VS Code might automatically choose the wrong virtualenvs. Change it
181
+
with `Python: Select Interpreter`.
57
182
58
-
### IDEs
183
+
The root (`immunisation-fhir-api`) should point to `/mnt/d/Source/immunisation-fhir-api/.venv/bin/python`.
59
184
60
-
The `backend` tests are in a separate module so in order for them to see each other we need to let the IDE know
61
-
about the relationship.
185
+
`backend` should be pointing at `/mnt/d/Source/immunisation-fhir-api/backend/.venv/bin/python`
62
186
63
-
####IntelliJ
187
+
### IntelliJ
64
188
65
189
- Open the root repo directory in IntelliJ.
66
190
- In Project Structure add an existing virtualenv SDK for `.direnv/python-x.x.x/bin/python`.
@@ -72,20 +196,8 @@ about the relationship.
72
196
- Add the `src` and `tests` directories as sources.
73
197
- Add `.direnv` as an exclusion if it's not already.
74
198
75
-
#### VS Code
76
-
77
-
The project must be opened as a multi-root workspace for VS Code to know that `backend` has its own environment.
78
-
79
-
- Open the workspace `immunisation-fhir-api.code-workspace`.
80
-
- Copy `backend/.vscode/settings.json.default` to `backend/.vscode/settings.json`, or merge the contents with
81
-
your existing file.
82
-
83
-
VS Code will automatically use the `backend` environment when you're editing a file under `backend`.
84
-
85
-
Depending on your existing setup VS Code might automatically choose the wrong virtualenvs. Change it
86
-
with `Python: Select Interpreter`.
87
-
88
-
The root (immunisation-fhir-api) should be pointing at `.direnv/python-x.x.x/bin/python.`
89
-
90
-
`backend` should be pointing at `backend/.direnv/python-x.x.x/bin/python.`
91
199
200
+
## Verified commits
201
+
Please note that this project requires that all commits are verified using a GPG key.
202
+
To set up a GPG key please follow the instructions specified here:
This document describes the environment setup for the backend API Lambda.
4
+
This Lambda handles incoming CRUD operation requests from APIGEE and interacts with the immunisation events database to store immunisation records. All commands listed below are run in the `./backend` folder.
4
5
5
-
## Install dependencies
6
+
## Setting up the backend lambda
7
+
Note: Paths are relative to this directory, `backend`.
6
8
7
-
```shell
8
-
pip install poetry
9
-
poetry install
10
-
pip install terraform-local
11
-
```
9
+
1. Follow the instructions in the root level README.md to setup the [dependencies](../README.md#environment-setup) and create a [virtual environment](../README.md#) for this folder.
12
10
11
+
2. Replace the `.env` file in the backend folder. Note the variables might change in the future. These environment variables will be loaded automatically when using `direnv`.
0 commit comments