Skip to content

Commit ae2bad2

Browse files
authored
Merge pull request #393 from NHSDigital/release/2024-11-06
Release/2024-11-06
2 parents 4dbdf7b + d658f03 commit ae2bad2

File tree

46 files changed

+535
-269
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

46 files changed

+535
-269
lines changed

.github/workflows/on-pr-close.yml

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
name: "Workflow: PR Close"
2+
3+
permissions:
4+
id-token: write
5+
contents: read
6+
actions: write
7+
8+
env:
9+
TF_CLI_ARGS: -no-color
10+
CI_ROLE_NAME: ${{ secrets.CI_ROLE_NAME }}
11+
BRANCH_NAME: ${{ github.event.pull_request.head.ref }}
12+
13+
on:
14+
pull_request:
15+
types: [closed]
16+
17+
jobs:
18+
parse-secrets:
19+
runs-on: [self-hosted, ci]
20+
steps:
21+
- id: parse-secrets
22+
run: |
23+
echo "::add-mask::${{ secrets.CI_ROLE_NAME }}"
24+
25+
build-base:
26+
runs-on: [self-hosted, ci]
27+
needs: [parse-secrets]
28+
steps:
29+
- uses: actions/checkout@v4
30+
with:
31+
ref: ${{ env.BRANCH_NAME }}
32+
- uses: ./.github/actions/make/
33+
with:
34+
command: build
35+
save-to-cache: "true"
36+
restore-from-cache: "false"
37+
38+
destroy-pr-workspaces:
39+
runs-on: [self-hosted, ci]
40+
needs: [parse-secrets, build-base]
41+
steps:
42+
- uses: actions/checkout@v4
43+
with:
44+
ref: ${{ env.BRANCH_NAME }}
45+
fetch-depth: 0
46+
- name: Remove PR workspaces
47+
uses: ./.github/actions/make/
48+
with:
49+
command: destroy--redundant-workspaces BRANCH_NAME=origin/${{ env.BRANCH_NAME }} DESTROY_ALL_COMMITS_ON_BRANCH=true KILL_ALL=true TF_CLI_ARGS=${{ env.TF_CLI_ARGS }}
50+
requires-aws: true

.github/workflows/pull-requests.yml

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -218,6 +218,7 @@ jobs:
218218
test--unit,
219219
test--feature--local,
220220
terraform-head-build,
221+
apigee--attach-product,
221222
]
222223
runs-on: [self-hosted, ci]
223224
strategy:
@@ -276,7 +277,13 @@ jobs:
276277
requires-aws: true
277278

278279
apigee--detach-product:
279-
needs: [build-head, test--smoke, apigee--attach-product]
280+
needs:
281+
[
282+
build-head,
283+
test--smoke,
284+
apigee--attach-product,
285+
test--feature--integration,
286+
]
280287
runs-on: [self-hosted, ci]
281288
if: ${{ needs.apigee--attach-product.result == 'success' }}
282289
steps:

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,7 @@ dist
7878
openapi
7979

8080
postman-collection.json
81+
postman-environment.json
8182
token_cache.json
8283
cpm.cdx.json
8384

CHANGELOG.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,14 @@
11
# Changelog
22

3+
## 2024-11-06
4+
- [PI-593] Readme updates
5+
- [PI-594] More smoke tests
6+
- [PI-575] Remove implicit OK from responses
7+
- [PI-558] Add search wrapper
8+
- [PI-591] Better postman
9+
- [PI-293] E2E tests with apigee
10+
- [PI-601] Destroy workspaces on PR close
11+
312
## 2024-11-05
413
- [PI-585] PR/release CI builds in dev
514

README.md

Lines changed: 61 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -10,9 +10,12 @@
1010
3. [AWS SSO Setup](#aws-sso-setup)
1111
4. [Other helpful commands](#other-helpful-commands)
1212
2. [Tests](#tests)
13-
3. [Workflow](#workflow)
14-
4. [Swagger](#swagger)
15-
5. [ETL](#etl)
13+
3. [pytest tests](#pytest-tests)
14+
4. [End-to-End feature tests](#end-to-end-feature-tests)
15+
5. [Generate the Feature Test Postman collection](#generate-the-feature-test-postman-collection)
16+
6. [Workflow](#workflow)
17+
7. [Swagger](#swagger)
18+
8. [ETL](#etl)
1619

1720
---
1821

@@ -22,16 +25,20 @@
2225

2326
We use `asdf` to fetch the required versions of prerequisite libraries instead of your system's default version. To get it up and running go to https://asdf-vm.com/guide/getting-started.html. You can check it installed properly by using the command `asdf --version`.
2427

25-
If you are using `pyenv` (you can check by typing `pyenv` and seeing whether it returns a nice list of commands) then you should run:
26-
27-
```
28-
pyenv install $(cat .python-version)
29-
```
28+
However, you will also need to install the `docker engine` separately
3029

3130
Additionally you will need `wget` (doing `which wget` will return blank if not installed). Please Google "how to install wget on my operating system", if you don't already have this installed.
3231

32+
Update any dependencies on your system as required.
33+
3334
Otherwise `asdf` should do the work for you.
3435

36+
### Useful tools
37+
38+
`VScode` is useful and we have a workspace file setup to allow easy integration
39+
40+
`Postman` &/or `Newman` Feature tests create a postman.collection which can be used for manual testing.
41+
3542
### Project build
3643

3744
Do `make build` every time you would like to pick up and install new local/project dependencies and artifacts. This will always detect changes to:
@@ -220,6 +227,52 @@ The VSCode settings for "Run and Debug" are also set up to run these tests if yo
220227
`make test--sds--matrix` is used for testing responses match in SDS FHIR between CPM and LDAP. You must provide `SDS_PROD_APIKEY` and `SDS_DEV_APIKEY`. There are 3 optional variables `USE_CPM_PROD`, defaults to `FALSE`, `COMPARISON_ENV`, defaults to `local` and `TEST_COUNT`, defaults to `10` and is the number of requests to make.
221228
Add `PYTEST_FLAGS='-sv'`.
222229

230+
### End-to-End feature tests
231+
232+
The Feature tests use `behave` (rather than `pytest`) to execute cucumber/gherkin-style end-to-end tests of specific features, in principle
233+
giving full end-to-end test coverage for API operations.
234+
235+
Executing feature tests locally will give you a good idea whether you have implemented a well-behaved feature whilst in development (i.e. no need to redeploy whilst developing).
236+
237+
Executing feature tests in integration mode will then give you confidence that the feature is ready to deploy in production, but has a much slower development cycle as it will need a full redeploy after codebase or infrastructure changes are implemented.
238+
239+
#### Local
240+
241+
To execute the feature tests entirely locally (executing lambdas directly, and otherwise mocking databases and responses to a high standard) you can do:
242+
243+
```shell
244+
make test--feature-local
245+
```
246+
247+
If you would like to pass `behave` flags, e.g. to \[stop after the first failure\]:
248+
249+
```shell
250+
make test--feature-local BEHAVE_FLAGS="--stop"
251+
```
252+
253+
#### Integration
254+
255+
To execute the feature tests across the entire stack (including Apigee and AWS) you can do
256+
257+
```shell
258+
make test--feature-integration
259+
```
260+
261+
### Generate the Feature Test Postman collection
262+
263+
Our [end-to-end feature tests](#end-to-end-feature-tests) also generate working Postman collections. To generate the Postman collection quickly, without Apigee credentials, you can run the local feature tests. If you would like the Apigee
264+
credentials generating then you should run the integration feature tests. The generated files are:
265+
266+
- `src/api/tests/feature_tests/postman-collection.json`
267+
- `src/api/tests/feature_tests/postman-environment.json`
268+
269+
You can drag and drop `postman-collection.json` into the `Collections` tab on Postman,
270+
and `postman-environment.json` on to the `Environments` tab (remember to activate it). If you generated these
271+
with the local feature tests, then you will need to manually update the Apigee `baseUrl` and `apiKey` fields
272+
in the environment (but these are filled out already if generated with the integration feature tests).
273+
274+
💡 **The feature tests are only guaranteed to work out-of-the-box with an empty database**
275+
223276
## Workflow
224277

225278
In order to create new branches, use the commands listed below. Note that the commands will throw an error if

VERSION

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
2024.11.05
1+
2024.11.06

changelog/2024-11-06.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
- [PI-593] Readme updates
2+
- [PI-594] More smoke tests
3+
- [PI-575] Remove implicit OK from responses
4+
- [PI-558] Add search wrapper
5+
- [PI-591] Better postman
6+
- [PI-293] E2E tests with apigee
7+
- [PI-601] Destroy workspaces on PR close

pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[tool.poetry]
22
name = "connecting-party-manager"
3-
version = "2024.11.05"
3+
version = "2024.11.06"
44
description = "Repository for the Connecting Party Manager API and related services"
55
authors = ["NHS England"]
66
license = "LICENSE.md"

scripts/builder/build.mk

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ POSTMAN_COLLECTION = $(CURDIR)/src/api/tests/feature_tests/postman-collection.js
66
TOOL_VERSIONS_COPY = $(TIMESTAMP_DIR)/tool-versions.copy
77
POETRY_LOCK = $(CURDIR)/poetry.lock
88
INIT_TIMESTAMP = $(CURDIR)/.timestamp/init.timestamp
9-
SRC_FILES = $(shell find src -type f -name "*.py" -not -path "*/test_*" -not -path "*/fhir/r4/strict_models.py" -not -path "*/fhir/r4/models.py")
9+
SRC_FILES = $(shell find src -type f -name "*.py" -not -path "*/feature_tests/*" -not -path "*/test_*" -not -path "*/fhir/r4/strict_models.py" -not -path "*/fhir/r4/models.py")
1010
THIRD_PARTY_DIST = $(CURDIR)/src/layers/third_party/dist
1111
SWAGGER_DIST = $(CURDIR)/infrastructure/swagger/dist
1212
SWAGGER_PUBLIC = $(SWAGGER_DIST)/public/swagger.yaml

scripts/infrastructure/apigee.mk

Lines changed: 20 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,14 @@
33
APIGEE_CONFIG_PATH = $(CURDIR)/infrastructure/apigee
44
APIGEE_TIMESTAMP = $(TIMESTAMP_DIR)/.apigee.stamp
55
PROXYGEN_TIMESTAMP = $(TIMESTAMP_DIR)/.proxygen.stamp
6+
PROXYGEN_PRODUCT_TIMESTAMP = $(TIMESTAMP_DIR)/.proxygen-product.stamp
67

78
SWAGGER_APIGEE = $(SWAGGER_DIST)/apigee/swagger.yaml
89
WORKSPACE_OUTPUT_JSON = $(CURDIR)/infrastructure/terraform/per_workspace/output.json
910
ENVIRONMENT_MAPPING_YAML = $(CURDIR)/infrastructure/apigee/environment_mapping.yaml
1011
STAGE_MAPPING_YAML = $(CURDIR)/infrastructure/apigee/stage_mapping.yaml
1112

12-
apigee--deploy: $(PROXYGEN_TIMESTAMP)
13+
apigee--deploy: aws--login $(PROXYGEN_TIMESTAMP)
1314

1415

1516
apigee--delete: aws--login
@@ -26,15 +27,7 @@ apigee--delete: aws--login
2627
apigee--clean:
2728
[[ -f $(PROXYGEN_TIMESTAMP) ]] && rm $(PROXYGEN_TIMESTAMP) || :
2829

29-
apigee--attach-product: aws--login
30-
WORKSPACE_OUTPUT_JSON=$(WORKSPACE_OUTPUT_JSON) \
31-
ENVIRONMENT_MAPPING_YAML=$(ENVIRONMENT_MAPPING_YAML) \
32-
STAGE_MAPPING_YAML=$(STAGE_MAPPING_YAML) \
33-
APIGEE_CONFIG_PATH=$(APIGEE_CONFIG_PATH) \
34-
AWS_ACCESS_KEY_ID=$(AWS_ACCESS_KEY_ID) \
35-
AWS_SECRET_ACCESS_KEY=$(AWS_SECRET_ACCESS_KEY) \
36-
AWS_SESSION_TOKEN=$(AWS_SESSION_TOKEN) \
37-
bash $(PATH_TO_INFRASTRUCTURE)/apigee/apigee.sh attach_product $(PERSISTENT_ENVIRONMENT_BUILD)
30+
apigee--attach-product: aws--login $(PROXYGEN_PRODUCT_TIMESTAMP)
3831

3932
apigee--detach-product: aws--login
4033
WORKSPACE_OUTPUT_JSON=$(WORKSPACE_OUTPUT_JSON) \
@@ -45,8 +38,10 @@ apigee--detach-product: aws--login
4538
AWS_SECRET_ACCESS_KEY=$(AWS_SECRET_ACCESS_KEY) \
4639
AWS_SESSION_TOKEN=$(AWS_SESSION_TOKEN) \
4740
bash $(PATH_TO_INFRASTRUCTURE)/apigee/apigee.sh detach_product $(PERSISTENT_ENVIRONMENT_BUILD)
41+
[[ -f $(PROXYGEN_PRODUCT_TIMESTAMP) ]] && rm $(PROXYGEN_PRODUCT_TIMESTAMP) || :
4842

49-
$(PROXYGEN_TIMESTAMP): aws--login $(SWAGGER_APIGEE) $(WORKSPACE_OUTPUT_JSON)
43+
44+
$(PROXYGEN_TIMESTAMP): $(SWAGGER_APIGEE) $(WORKSPACE_OUTPUT_JSON)
5045
[[ -f $(PROXYGEN_TIMESTAMP) ]] && rm $(PROXYGEN_TIMESTAMP) || :
5146

5247
WORKSPACE_OUTPUT_JSON=$(WORKSPACE_OUTPUT_JSON) \
@@ -60,3 +55,17 @@ $(PROXYGEN_TIMESTAMP): aws--login $(SWAGGER_APIGEE) $(WORKSPACE_OUTPUT_JSON)
6055
bash $(PATH_TO_INFRASTRUCTURE)/apigee/proxygen.sh generate_proxy $(PERSISTENT_ENVIRONMENT_BUILD)
6156

6257
touch $(PROXYGEN_TIMESTAMP)
58+
59+
60+
$(PROXYGEN_PRODUCT_TIMESTAMP): $(PROXYGEN_TIMESTAMP)
61+
[[ -f $(PROXYGEN_PRODUCT_TIMESTAMP) ]] && rm $(PROXYGEN_PRODUCT_TIMESTAMP) || :
62+
63+
WORKSPACE_OUTPUT_JSON=$(WORKSPACE_OUTPUT_JSON) \
64+
ENVIRONMENT_MAPPING_YAML=$(ENVIRONMENT_MAPPING_YAML) \
65+
STAGE_MAPPING_YAML=$(STAGE_MAPPING_YAML) \
66+
APIGEE_CONFIG_PATH=$(APIGEE_CONFIG_PATH) \
67+
AWS_ACCESS_KEY_ID=$(AWS_ACCESS_KEY_ID) \
68+
AWS_SECRET_ACCESS_KEY=$(AWS_SECRET_ACCESS_KEY) \
69+
AWS_SESSION_TOKEN=$(AWS_SESSION_TOKEN) \
70+
bash $(PATH_TO_INFRASTRUCTURE)/apigee/apigee.sh attach_product $(PERSISTENT_ENVIRONMENT_BUILD)
71+
touch $(PROXYGEN_PRODUCT_TIMESTAMP)

0 commit comments

Comments
 (0)