Skip to content

Commit 3f0d583

Browse files
ekremneyclaude
ekremney
and
claude
authored
docs(data-access): add v3 architecture docs and CLAUDE.md (#1389)
## Summary - Added **Architecture** section to README.md showing the v3 data flow: Lambda -> this package -> postgrest-js -> mysticat-data-service (PostgREST + Aurora PostgreSQL) - Added **Changing Entities** section documenting the two-repo workflow after the DynamoDB-to-PostgreSQL migration: (1) create DB migration in mysticat-data-service, (2) update model/collection/schema in this package, (3) integration test - Created **CLAUDE.md** with developer reference covering key files, entity structure, SchemaBuilder patterns, field mapping, testing, and environment variables - Updated **Migrating from V2** section with note that schema changes now go through mysticat-data-service migrations - Removed **CODEOWNERS** migration lock that required gatekeeper approval for data-access changes during the DynamoDB-to-PostgreSQL migration ## Test plan - [x] Verify README.md renders correctly on GitHub - [x] Verify CLAUDE.md renders correctly on GitHub - [ ] Confirm links to mysticat-data-service resolve 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 298c6e7 commit 3f0d583

File tree

3 files changed

+281
-2
lines changed

3 files changed

+281
-2
lines changed

‎.github/CODEOWNERS‎

Lines changed: 0 additions & 2 deletions
This file was deleted.

‎packages/spacecat-shared-data-access/CLAUDE.md‎

Lines changed: 204 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,204 @@
1+
# spacecat-shared-data-access
2+
3+
Shared data-access layer for SpaceCat services. Provides entity models/collections backed by PostgreSQL via PostgREST.
4+
5+
## Architecture
6+
7+
```
8+
Lambda/ECS service
9+
-> this package (@adobe/spacecat-shared-data-access)
10+
-> @supabase/postgrest-js
11+
-> mysticat-data-service (PostgREST + Aurora PostgreSQL)
12+
```
13+
14+
- **Database schema**: lives in [mysticat-data-service](https://github.com/adobe/mysticat-data-service) as dbmate SQL migrations
15+
- **This package**: JavaScript model/collection layer mapping camelCase entities to snake_case PostgREST API
16+
- **v2 (retired)**: ElectroDB -> DynamoDB. Published as `@adobe/spacecat-shared-data-access-v2`
17+
- **v3 (current)**: PostgREST client -> mysticat-data-service
18+
19+
## Key Files
20+
21+
| File | Purpose |
22+
|------|---------|
23+
| `src/index.js` | Default export: `dataAccessWrapper(fn)` for Helix/Lambda handlers |
24+
| `src/service/index.js` | `createDataAccess(config, log?, client?)` factory |
25+
| `src/models/base/schema.builder.js` | DSL for defining entity schemas (attributes, references, indexes) |
26+
| `src/models/base/base.model.js` | Base entity class (auto-generated getters/setters, save, remove) |
27+
| `src/models/base/base.collection.js` | Base collection class (findById, all, query, count) |
28+
| `src/models/base/entity.registry.js` | Registers all entity collections |
29+
| `src/util/postgrest.utils.js` | camelCase<->snake_case field mapping, query builders, cursor pagination |
30+
| `src/models/index.js` | Barrel export of all entity models |
31+
32+
## Entity Structure
33+
34+
Each entity lives in `src/models/<entity>/` with 4 files:
35+
36+
```
37+
src/models/site/
38+
site.schema.js # SchemaBuilder definition (attributes, references, indexes)
39+
site.model.js # Extends BaseModel (business logic, constants)
40+
site.collection.js # Extends BaseCollection (custom queries)
41+
index.js # Re-exports model, collection, schema
42+
```
43+
44+
### Schema Definition Pattern
45+
46+
```js
47+
const schema = new SchemaBuilder(Site, SiteCollection)
48+
.addReference('belongs_to', 'Organization') // FK -> organizations.id
49+
.addReference('has_many', 'Audits') // One-to-many relationship
50+
.addAttribute('baseURL', {
51+
type: 'string',
52+
required: true,
53+
validate: (value) => isValidUrl(value),
54+
})
55+
.addAttribute('deliveryType', {
56+
type: Object.values(Site.DELIVERY_TYPES), // Enum validation
57+
default: Site.DEFAULT_DELIVERY_TYPE,
58+
required: true,
59+
})
60+
.addAttribute('config', {
61+
type: 'any',
62+
default: DEFAULT_CONFIG,
63+
get: (value) => Config(value), // Transform on read
64+
})
65+
.addAllIndex(['imsOrgId']) // Query index
66+
.build();
67+
```
68+
69+
### Attribute Options
70+
71+
| Option | Purpose |
72+
|--------|---------|
73+
| `type` | `'string'`, `'number'`, `'boolean'`, `'any'`, `'map'`, or array of enum values |
74+
| `required` | Validation on save |
75+
| `default` | Default value or factory function |
76+
| `validate` | Custom validation function |
77+
| `readOnly` | No setter generated |
78+
| `postgrestField` | Custom DB column name (default: `camelToSnake(name)`) |
79+
| `postgrestIgnore` | Virtual attribute, not sent to DB |
80+
| `hidden` | Excluded from `toJSON()` |
81+
| `watch` | Array of field names that trigger this attribute's setter |
82+
| `set` | Custom setter `(value, allAttrs) => transformedValue` |
83+
| `get` | Custom getter `(value) => transformedValue` |
84+
85+
### Field Mapping
86+
87+
Models use camelCase, database uses snake_case. Mapping is automatic:
88+
89+
| Model field | DB column | Notes |
90+
|-------------|-----------|-------|
91+
| `siteId` (idName) | `id` | Primary key always maps to `id` |
92+
| `baseURL` | `base_url` | Auto camelToSnake |
93+
| `organizationId` | `organization_id` | FK from `belongs_to` reference |
94+
| `isLive` | `is_live` | Auto camelToSnake |
95+
96+
Override with `postgrestField: 'custom_name'` on the attribute.
97+
98+
## Changing Entities
99+
100+
Changes require **two repos**:
101+
102+
### 1. Database schema — [mysticat-data-service](https://github.com/adobe/mysticat-data-service)
103+
104+
```bash
105+
make migrate-new name=add_foo_to_sites
106+
# Edit the migration SQL (table, columns, indexes, grants, comments)
107+
make migrate && make test
108+
```
109+
110+
Every migration must include: indexes on FKs, `GRANT` to `postgrest_anon`/`postgrest_writer`, `COMMENT ON` for OpenAPI docs. See [mysticat-data-service CLAUDE.md](https://github.com/adobe/mysticat-data-service/blob/main/CLAUDE.md).
111+
112+
### 2. Model layer — this package
113+
114+
- Add attribute in `<entity>.schema.js` -> auto-generates getter/setter
115+
- Add business logic in `<entity>.model.js`
116+
- Add custom queries in `<entity>.collection.js`
117+
- New entity: create 4 files + register in `src/models/index.js`
118+
119+
### 3. Integration test
120+
121+
```bash
122+
npm run test:it # Spins up PostgREST via Docker, runs mocha suite
123+
```
124+
125+
## Testing
126+
127+
```bash
128+
npm test # Unit tests (mocha + sinon + chai)
129+
npm run test:debug # Unit tests with debugger
130+
npm run test:it # Integration tests (Docker: Postgres + PostgREST)
131+
npm run lint # ESLint
132+
npm run lint:fix # Auto-fix lint issues
133+
```
134+
135+
### Integration Test Setup
136+
137+
Integration tests pull the `mysticat-data-service` Docker image from ECR:
138+
139+
```bash
140+
# ECR login (one-time)
141+
aws ecr get-login-password --profile spacecat-dev --region us-east-1 \
142+
| docker login --username AWS --password-stdin 682033462621.dkr.ecr.us-east-1.amazonaws.com
143+
144+
# Override image tag
145+
export MYSTICAT_DATA_SERVICE_TAG=v1.13.0
146+
npm run test:it
147+
```
148+
149+
### Unit Test Conventions
150+
151+
- Tests in `test/unit/models/<entity>/`
152+
- PostgREST calls are stubbed via sinon
153+
- Each entity model and collection has its own test file
154+
155+
## Common Patterns
156+
157+
### Collection query with WHERE clause
158+
159+
```js
160+
// In a collection method
161+
async findByStatus(status) {
162+
return this.all(
163+
(attrs, op) => op.eq(attrs.status, status),
164+
{ limit: 100, order: { field: 'createdAt', direction: 'desc' } }
165+
);
166+
}
167+
```
168+
169+
### Reference traversal
170+
171+
```js
172+
// belongs_to: site.getOrganization() -> fetches parent org
173+
// has_many: organization.getSites() -> fetches child sites
174+
const site = await dataAccess.Site.findById(id);
175+
const org = await site.getOrganization();
176+
const audits = await site.getAudits();
177+
```
178+
179+
### PostgREST WHERE operators
180+
181+
`eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `is`, `in`, `contains`, `like`, `ilike`
182+
183+
```js
184+
// Usage in collection.all()
185+
const liveSites = await dataAccess.Site.all(
186+
(attrs, op) => op.eq(attrs.isLive, true)
187+
);
188+
```
189+
190+
## Environment Variables
191+
192+
| Variable | Required | Purpose |
193+
|----------|----------|---------|
194+
| `POSTGREST_URL` | Yes | PostgREST base URL (e.g. `http://data-svc.internal`) |
195+
| `POSTGREST_SCHEMA` | No | Schema name (default: `public`) |
196+
| `POSTGREST_API_KEY` | No | JWT for `postgrest_writer` role (enables UPDATE/DELETE) |
197+
| `S3_CONFIG_BUCKET` | No | Only for `Configuration` entity |
198+
| `AWS_REGION` | No | Only for `Configuration` entity |
199+
200+
## Special Entities
201+
202+
- **Configuration**: S3-backed (not PostgREST). Requires `S3_CONFIG_BUCKET`.
203+
- **KeyEvent**: Deprecated in v3. All methods throw.
204+
- **LatestAudit**: Virtual entity computed from `Audit` queries (no dedicated table).

‎packages/spacecat-shared-data-access/README.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,12 +134,88 @@ Current exported entities include:
134134
- `TrialUser`
135135
- `TrialUserActivity`
136136

137+
## Architecture
138+
139+
```
140+
Lambda / ECS service
141+
-> @adobe/spacecat-shared-data-access (this package)
142+
-> @supabase/postgrest-js
143+
-> mysticat-data-service (PostgREST + Aurora PostgreSQL)
144+
https://github.com/adobe/mysticat-data-service
145+
```
146+
147+
**v2 (retired):** ElectroDB -> DynamoDB (direct, schema-in-code)
148+
**v3 (current):** PostgREST client -> [mysticat-data-service](https://github.com/adobe/mysticat-data-service) (schema-in-database)
149+
150+
The database schema (tables, indexes, enums, grants) lives in **mysticat-data-service** as dbmate migrations.
151+
This package provides the JavaScript model/collection layer that maps camelCase entities to the snake_case PostgREST API.
152+
137153
## V3 Behavior Notes
138154

139155
- `Configuration` remains S3-backed in v3.
140156
- `KeyEvent` is deprecated in v3 and intentionally throws on access/mutation methods.
141157
- `LatestAudit` is virtual in v3 and derived from `Audit` queries (no dedicated table required).
142158

159+
## Changing Entities
160+
161+
Adding or modifying an entity now requires changes in **two repositories**:
162+
163+
### 1. Database schema — [mysticat-data-service](https://github.com/adobe/mysticat-data-service)
164+
165+
Create a dbmate migration for the schema change (table, columns, indexes, grants, enums):
166+
167+
```bash
168+
# In mysticat-data-service
169+
make migrate-new name=add_foo_column_to_sites
170+
# Edit db/migrations/YYYYMMDDHHMMSS_add_foo_column_to_sites.sql
171+
make migrate
172+
docker compose -f docker/docker-compose.yml restart postgrest
173+
make test
174+
```
175+
176+
See the [mysticat-data-service CLAUDE.md](https://github.com/adobe/mysticat-data-service/blob/main/CLAUDE.md) for migration conventions (required grants, indexes, comments, etc.).
177+
178+
### 2. Model/collection layer — this package
179+
180+
Update the entity schema, model, and/or collection in `src/models/<entity>/`:
181+
182+
| File | What to change |
183+
|------|---------------|
184+
| `<entity>.schema.js` | Add/modify attributes, references, indexes |
185+
| `<entity>.model.js` | Add business logic methods |
186+
| `<entity>.collection.js` | Add custom query methods |
187+
188+
**Adding a new attribute example:**
189+
190+
```js
191+
// In <entity>.schema.js, add to the SchemaBuilder chain:
192+
.addAttribute('myNewField', {
193+
type: 'string',
194+
required: false,
195+
// Optional: custom DB column name (default: camelToSnake)
196+
// postgrestField: 'custom_column_name',
197+
})
198+
```
199+
200+
This automatically generates `getMyNewField()` and `setMyNewField()` on the model.
201+
202+
**Adding a new entity:** Create 4 files following the pattern in any existing entity folder:
203+
- `<entity>.schema.js` — SchemaBuilder definition
204+
- `<entity>.model.js` — extends `BaseModel`
205+
- `<entity>.collection.js` — extends `BaseCollection`
206+
- `index.js` — re-exports model, collection, schema
207+
208+
Then register the entity in `src/models/index.js`.
209+
210+
### 3. Integration test the full stack
211+
212+
```bash
213+
# In this package — runs PostgREST in Docker
214+
npm run test:it
215+
```
216+
217+
Integration tests pull the `mysticat-data-service` Docker image from ECR, so new schema changes must be published as a new image tag first (or test against a local PostgREST).
218+
143219
## Migrating from V2
144220

145221
If you are upgrading from DynamoDB/ElectroDB-based v2:
@@ -154,6 +230,7 @@ If you are upgrading from DynamoDB/ElectroDB-based v2:
154230

155231
- Backing store is now Postgres via PostgREST, not DynamoDB/ElectroDB.
156232
- You must provide `postgrestUrl` (or `POSTGREST_URL` via wrapper env).
233+
- Schema changes now go through [mysticat-data-service](https://github.com/adobe/mysticat-data-service) migrations, not code.
157234
- `Configuration` remains S3-backed (requires `s3Bucket`/`S3_CONFIG_BUCKET` when used).
158235
- `KeyEvent` is deprecated in v3 and now throws.
159236
- `LatestAudit` is no longer a dedicated table and is computed from `Audit` queries.

0 commit comments

Comments
 (0)