Skip to content

Commit 6ecccee

Browse files
harrim91harrim91
committed
CCM-10483: add readme
1 parent 9d68d82 commit 6ecccee

File tree

2 files changed

+216
-0
lines changed

2 files changed

+216
-0
lines changed

‎scripts/config/vale/styles/config/vocabularies/words/accept.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ bot
44
Cognito
55
Cyber
66
Dependabot
7+
dev
78
draw.io
89
drawio
910
endcapture
@@ -21,8 +22,10 @@ onboarding
2122
Podman
2223
Python
2324
rawContent
25+
repo
2426
sed
2527
Syft
2628
Terraform
2729
toolchain
2830
Trufflehog
31+
Zod

‎tests/contracts/README.md‎

Lines changed: 213 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,213 @@
1+
# Contract Testing Spike
2+
3+
## Summary
4+
5+
This spike demonstrates an event-driven contract testing framework with the following features:
6+
7+
- Consumers defining their expectations with Pact
8+
- Providers defining canonical schema contracts with JSON schema (golden contracts)
9+
- Contract sharing with S3
10+
- CI enforcing both consumer-driven and provider-driven correctness
11+
12+
---
13+
14+
## Terminology
15+
16+
**Consumer**
17+
The system (usually a service or Lambda) that **receives** or **reacts to** an event. In Pact contract testing, the consumer defines what kind of event payloads it can handle.
18+
19+
**Provider**
20+
The system that **produces** and emits an event. In this setup, providers define canonical JSON Schemas (golden contracts) for the events they emit.
21+
22+
> N.B. A service can be both a provider and a consumer of events
23+
24+
**Golden Contract**
25+
A JSON Schema file generated and owned by the provider, representing the authoritative shape of an event. Consumers use this to validate they are handling messages correctly.
26+
27+
**Pact**
28+
A contract testing tool used for consumer-driven contracts. Consumers define what they expect from the provider, and providers verify that they meet those expectations.
29+
30+
---
31+
32+
## Project Structure
33+
34+
```txt
35+
├── scripts # bash heaven
36+
│   ├── ci-verify-provider.sh - mushes together steps required for provider-side validation in CI
37+
│   ├── clean.sh # deletes all local generated/downloaded contract files
38+
│   ├── download-consumer-pacts.sh # downloads pact files generated by consumers for use in provider-side pact tests
39+
│   ├── download-golden-contracts.sh # downloads golden contracts generated by providers, for use in consumer-side validation
40+
│   ├── generate-golden-contracts.ts # generates golden contracts for event providers
41+
│   ├── upload-consumer-pacts.sh # uploads pact files generated by consumer-side pact tests
42+
│   ├── upload-golden-contracts.sh # uploads generated golden contracts to s3
43+
│   └── verify-golden-contracts.sh # verifies client-provided example events against provider-generated golden contracts
44+
|
45+
├── src # pseudo source-code
46+
│   ├── <service>
47+
│   │   ├── events # code related to publishing events
48+
│   │   │   ├── .schemas # generated golden contracts - gitignored
49+
│   │   │   │   └── <EventName>.schema.json
50+
│   │   │   |
51+
│   │   │   └── <event-name>.event.ts # code that generates an event payload
52+
│   │   |
53+
│   │   └── handlers # code related to consuming events
54+
│   │   └── template-deleted.handler.ts # event parsing and handling code
55+
|
56+
├── tests # contract testing code
57+
│   ├── <service>
58+
│   │   ├── consumer # tests for event consumption
59+
│   │   │   ├── .pacts # generated pact files outlining consumers expectations - gitignored
60+
│   │   │   │   └── <consumer>-<provider>.json
61+
│   │   │   |
62+
│   │   │   ├── .schemas # downloaded golden contracts from providers - gitignored
63+
│   │   │   │   └── <provider>
64+
│   │   │   │   └── <EventName>.schema.json
65+
│   │   │   |
66+
│   │   │   ├── config.json # Lists the provider(s) and event(s) that the consumer depends on. Used to fetch only the relevant golden contracts
67+
│   │   │   |
68+
│   │   │   ├── examples # sample json files representing what the consumer thinks it might receive. These are validated against golden contracts to catch schema mismatches
69+
│   │   │   │   └── <provider>
70+
│   │   │   │   └── <EventName>
71+
│   │   │   │   └── <scenario>.json
72+
│   │   │   │
73+
│   │   │   └── <event-name>.consumer.pact.test.ts # pact test file, outlines consumers expectations and generates pact file
74+
│   │   |
75+
│   │   └── provider # tests for event production
76+
│   │   ├── .pacts # downloaded pact files from consumers - gitignored
77+
│   │   │   └── <consumer>-<provider>.json
78+
│   │   └── <event-name>.provider.pact.test.ts # validates an emitted event against consumer expectations
79+
```
80+
81+
---
82+
83+
## Scenario
84+
85+
The POC defines 3 services - `auth`, `core` and `templates`, which act as event providers and consumers:
86+
87+
### Service Event Responsibilities
88+
89+
#### auth
90+
91+
**Emits:**
92+
93+
| Event | Consumed By |
94+
|-------------|-------------|
95+
| UserCreated | templates |
96+
97+
**Consumes:**
98+
99+
| Event | Emitted By |
100+
|------------------|-------------|
101+
| TemplateDeleted | templates |
102+
103+
---
104+
105+
#### templates
106+
107+
**Emits:**
108+
109+
| Event | Consumed By |
110+
|------------------|------------------|
111+
| TemplateDeleted | auth, core |
112+
113+
**Consumes:**
114+
115+
| Event | Emitted By |
116+
|-------------|-------------|
117+
| UserCreated | auth |
118+
119+
---
120+
121+
#### core
122+
123+
**Emits:** _(none)_
124+
125+
**Consumes:**
126+
127+
| Event | Emitted By |
128+
|------------------|-------------|
129+
| TemplateDeleted | templates |
130+
131+
---
132+
133+
## Contract Types
134+
135+
### Consumer-Driven (Pact)
136+
137+
- Pact consumer tests generate `.json` contracts for expected messages.
138+
- Stored under: `tests/<service>/consumer/.pacts/`
139+
- Uploaded to: `s3://<bucket>/pacts/<provider>/` - indexed by provider for easy download by provider
140+
- Downloaded and validated on provider-side, to ensure that the provider is meeting consumer expectations
141+
142+
### Provider-Driven (Golden Contracts)
143+
144+
- JSON Schemas are generated by event providers using Zod + `zod-to-json-schema`.
145+
- Flattened at generation time to avoid `$ref` - this is a bit of a quirk
146+
- Saved to: `src/<service>/events/.schemas/`
147+
- Uploaded to: `s3://<bucket>/golden/<provider>/<Event>.schema.json`
148+
- Downloaded and validated against example events on client-side
149+
150+
## Running locally
151+
152+
Ensure you are signed into AWS, and have `PACT_BUCKET` set in your environment variables. I've been using the artifact bucket in the templates dev account.
153+
154+
Then from the repository root, let's test out some golden contracts:
155+
156+
- Start by generating some golden contracts for the provider services - `npm run test:contracts:generate:provider`
157+
- Upload the golden contracts to S3 - `npm run test:contracts:upload:provider`
158+
- Download them from S3 into the consumer tests - `npm run test:contracts:download:provider`
159+
- Validate that the consumers expectations are valid against the golden contracts - `npm run test:contracts:consumer:golden`
160+
161+
Next, lets run some consumer-driven pact tests:
162+
163+
- Run the consumer tests, and generate some Pact contract files - `npm run test:contracts:consumer`
164+
- Upload those to S3 - `npm run test:contracts:upload:consumer`
165+
- Now download those Pact contracts into the provider tests - `npm run test:contracts:download:consumer`
166+
- And validate that the events emitted by providers match the expectations of the consumers - `npm run test:contracts:provider`
167+
168+
At any point, clean up all of the locally stored contract files using `npm run test:contracts:clean`
169+
170+
I apologize for the very confusing command names.
171+
172+
## CI Flow
173+
174+
The CI flow contains both consumer and provider test jobs - a service can be both a provider and consumer of events.
175+
176+
Currently the CI bits have been plumbed into the existing CI job. Because this repo defines multiple services which emit and consume events, it loops over each service and runs the tests all at once. In reality a repo would only define a single service, and only one set of provider tests and one set of consumer tests would be executed.
177+
178+
### Consumer job
179+
180+
The consumer test job in CI performs the following steps:
181+
182+
1. **Download golden contracts**
183+
Based on the consumer's `config.json`, only the required event schemas are pulled from S3.
184+
185+
2. **Validate consumer example payloads**
186+
Local examples are validated against the downloaded golden contracts using `@sourcemeta/jsonschema`. If any validation fails, the job stops here - no Pact contracts are generated or uploaded.
187+
188+
3. **Run Pact consumer tests**
189+
If schema validation passes, Pact tests are run to verify that the consumer's expectations are correctly captured in the contract.
190+
191+
4. **Upload Pact contracts to S3**
192+
Validated contracts are published to S3 under a provider-specific path for use in provider-side verification.
193+
194+
### Provider Job
195+
196+
The provider test job in CI performs the following steps:
197+
198+
1. **Download Pact contracts from S3**
199+
All Pact contracts for the provider are downloaded from `s3://<bucket>/pacts/<provider>/`. These contracts are written by consumers and define their expected message shapes. (N.B. Different consumers can have different expectations!)
200+
201+
2. **Run Pact provider tests**
202+
The provider uses real message-producing code to generate event payloads, which are then validated against each downloaded Pact contract.
203+
204+
3. **Skip verification gracefully if no contracts found**
205+
If no contracts are present in S3 (i.e. if nobody is consuming any events) the job logs a warning and exits successfully without running tests.
206+
207+
This job ensures that the provider is compatible with all currently published consumer expectations.
208+
209+
## Next Steps
210+
211+
- Try open-source, self-hosted Pact broker for sharing contracts (instead of S3)
212+
- Think about versioning contracts (using git branch/tags/commit-sha?)
213+
- Simplify scripts and CI - remove iteration, run steps for one service at a time

0 commit comments

Comments
 (0)