feat: add ContextMemory entity (Context Center memories)#28224
Conversation
github-actions
Bot
added
Ingestion
safe to test
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
Upstream introduction of a new ContextMemory first-class entity for the Context Center: schema + create payload, repository/resource/mapper, CollectionDAO wiring, Entity.CONTEXT_MEMORY constant, MySQL/Postgres DDL in native/2.0.1, and a small unit test for status-transition rules. The entity is intentionally non-searchable and the resource exposes only standard CRUD at /v1/contextCenter/memories; AI/embedding concerns are left to downstream.
Changes:
- Add
entity/context/contextMemory.jsonandapi/context/createContextMemory.jsonplusEntity.CONTEXT_MEMORYregistration. - Add
ContextMemoryRepository,ContextMemoryResource,ContextMemoryMapper, andCollectionDAO.contextMemoryDAO(); relationships and lifecycle status enforcement live in the repository. - Add MySQL/Postgres DDL for
context_memoryin2.0.1/*/schemaChanges.sqland a unit test forvalidateStatusTransition.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| openmetadata-spec/src/main/resources/json/schema/entity/context/contextMemory.json | New entity schema with enums (type/scope/status/source/visibility/role), share config, machine representation, hierarchy fields, and standard entity properties. |
| openmetadata-spec/src/main/resources/json/schema/api/context/createContextMemory.json | Create payload; requires name, question, answer (inconsistent with the entity's required list). |
| openmetadata-service/src/main/java/org/openmetadata/service/Entity.java | Registers CONTEXT_MEMORY constant. |
| openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/CollectionDAO.java | Adds ContextMemoryDAO and contextMemoryDAO() accessor backed by context_memory table. |
| openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/ContextMemoryRepository.java | Implements FQN, prepare/store/relationships, status-transition validation, and a minimal updater (does not maintain relationships on updates). |
| openmetadata-service/src/main/java/org/openmetadata/service/resources/context/ContextMemoryResource.java | Thin EntityResource exposing standard CRUD/patch/version/restore endpoints. |
| openmetadata-service/src/main/java/org/openmetadata/service/resources/context/ContextMemoryMapper.java | Maps CreateContextMemory → ContextMemory, resolves domain FQNs and defaults owner to the caller. |
| bootstrap/sql/migrations/native/2.0.1/mysql/schemaChanges.sql | Creates context_memory table for MySQL (deliberately non-_entity suffix). |
| bootstrap/sql/migrations/native/2.0.1/postgres/schemaChanges.sql | Creates context_memory table and updatedAt index for Postgres. |
| openmetadata-service/src/test/java/org/openmetadata/service/resources/context/ContextMemoryResourceTest.java | Unit tests of validateStatusTransition only — does not exercise the REST resource or repository CRUD. |
Comments suppressed due to low confidence (1)
openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/ContextMemoryRepository.java:227
storeRelationshipsis only invoked once at create time byEntityRepository.createNewEntityFlush; it is not re-invoked during PUT/PATCH updates.ContextMemoryUpdater.entitySpecificUpdateonly callsrecordChangefor scalar fields and does not maintain the HAS/RELATED_TO relationships. As a result, when a client updatesprimaryEntity,relatedEntities,rootMemory, orparentMemory, the underlyingentity_relationshiprows are not updated: the old links remain and the new links are never persisted. The JSON payload appears updated but lineage / containment queries that rely on the relationships table will be incorrect. These fields need dedicated update handling inContextMemoryUpdater(deleting stale relationships and adding new ones) or, at minimum, a re-derivation step instoreRelationships-on-update fashion.
public class ContextMemoryUpdater extends EntityUpdater {
public ContextMemoryUpdater(
ContextMemory original, ContextMemory updated, Operation operation) {
super(original, updated, operation);
}
@Override
public void entitySpecificUpdate(boolean consolidatingChanges) {
recordChange("title", original.getTitle(), updated.getTitle());
recordChange("summary", original.getSummary(), updated.getSummary());
recordChange("question", original.getQuestion(), updated.getQuestion());
recordChange("answer", original.getAnswer(), updated.getAnswer());
recordChange("memoryType", original.getMemoryType(), updated.getMemoryType());
recordChange("memoryScope", original.getMemoryScope(), updated.getMemoryScope());
// Validate lifecycle transition before recording status change
if (original.getStatus() != null
&& updated.getStatus() != null
&& original.getStatus() != updated.getStatus()) {
validateStatusTransition(original.getStatus(), updated.getStatus());
}
recordChange("status", original.getStatus(), updated.getStatus());
recordChange("shareConfig", original.getShareConfig(), updated.getShareConfig());
}
}
|
…status msg, test name) - storeRelationships: rootMemory -> Relationship.CONTAINS, parentMemory -> Relationship.HAS so the root-ancestor and direct-parent hierarchies are distinguishable. - setFullyQualifiedName: derive from the immutable name only (drop mutable primaryEntity/owner derivation that destabilized nameHash on update). - validateStatusTransition: separate "no transitions defined" from "disallowed transition". - Rename ContextMemoryResourceTest -> ContextMemoryStatusTransitionTest (pure unit test). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
| public void entitySpecificUpdate(boolean consolidatingChanges) { | ||
| recordChange("title", original.getTitle(), updated.getTitle()); | ||
| recordChange("summary", original.getSummary(), updated.getSummary()); | ||
| recordChange("question", original.getQuestion(), updated.getQuestion()); | ||
| recordChange("answer", original.getAnswer(), updated.getAnswer()); | ||
| recordChange("memoryType", original.getMemoryType(), updated.getMemoryType()); | ||
| recordChange("memoryScope", original.getMemoryScope(), updated.getMemoryScope()); | ||
|
|
||
| // Validate lifecycle transition before recording status change | ||
| if (original.getStatus() != null | ||
| && updated.getStatus() != null | ||
| && original.getStatus() != updated.getStatus()) { | ||
| validateStatusTransition(original.getStatus(), updated.getStatus()); | ||
| } | ||
| recordChange("status", original.getStatus(), updated.getStatus()); | ||
|
|
||
| recordChange("shareConfig", original.getShareConfig(), updated.getShareConfig()); | ||
| } |
| public void storeRelationships(ContextMemory entity) { | ||
| if (entity.getPrimaryEntity() != null) { | ||
| addRelationship( | ||
| entity.getPrimaryEntity().getId(), | ||
| entity.getId(), | ||
| entity.getPrimaryEntity().getType(), | ||
| CONTEXT_MEMORY_ENTITY, | ||
| Relationship.HAS); | ||
| } | ||
|
|
||
| for (var relatedEntity : listOrEmpty(entity.getRelatedEntities())) { | ||
| addRelationship( | ||
| relatedEntity.getId(), | ||
| entity.getId(), | ||
| relatedEntity.getType(), | ||
| CONTEXT_MEMORY_ENTITY, | ||
| Relationship.RELATED_TO); | ||
| } | ||
|
|
||
| // Distinct relationship types so the root-ancestor and direct-parent hierarchies | ||
| // can be resolved independently when read back from the relationship table. | ||
| if (entity.getRootMemory() != null) { | ||
| addRelationship( | ||
| entity.getRootMemory().getId(), | ||
| entity.getId(), | ||
| CONTEXT_MEMORY_ENTITY, | ||
| CONTEXT_MEMORY_ENTITY, | ||
| Relationship.CONTAINS); | ||
| } | ||
|
|
||
| if (entity.getParentMemory() != null) { | ||
| addRelationship( | ||
| entity.getParentMemory().getId(), | ||
| entity.getId(), | ||
| CONTEXT_MEMORY_ENTITY, | ||
| CONTEXT_MEMORY_ENTITY, | ||
| Relationship.HAS); | ||
| } | ||
| } |
| if (!nullOrEmpty(owners)) { | ||
| return owners; | ||
| } | ||
| return List.of(getEntityReferenceByName(Entity.USER, user, Include.NON_DELETED)); |
| } | ||
| } | ||
| }, | ||
| "required": ["name", "question", "answer"], |
| @@ -0,0 +1,221 @@ | |||
| /* | |||
| * Copyright 2024 Collate | |||
| @Path("/v1/contextCenter/memories") | ||
| @Produces(MediaType.APPLICATION_JSON) | ||
| @Consumes(MediaType.APPLICATION_JSON) | ||
| @Collection(name = "contextMemories") | ||
| public class ContextMemoryResource extends EntityResource<ContextMemory, ContextMemoryRepository> { | ||
| public static final String COLLECTION_PATH = "v1/contextCenter/memories/"; |
| ContextMemory rootMemory = Entity.getEntity(entity.getRootMemory(), "", Include.NON_DELETED); | ||
| entity.setRootMemory(rootMemory.getEntityReference()); | ||
| } | ||
| if (entity.getParentMemory() != null) { | ||
| ContextMemory parentMemory = | ||
| Entity.getEntity(entity.getParentMemory(), "", Include.NON_DELETED); | ||
| entity.setParentMemory(parentMemory.getEntityReference()); |
| if (entity.getShareConfig() != null && entity.getShareConfig().getSharedWith() != null) { | ||
| entity | ||
| .getShareConfig() | ||
| .getSharedWith() | ||
| .forEach( | ||
| sharedPrincipal -> { | ||
| if (sharedPrincipal.getPrincipal() != null) { | ||
| EntityReference principal = | ||
| Entity.getEntityReference( | ||
| sharedPrincipal.getPrincipal(), Include.NON_DELETED); | ||
| sharedPrincipal.setPrincipal(principal); | ||
| } | ||
| }); | ||
| } |
| "description": "Role granted to the principal.", | ||
| "$ref": "#/definitions/shareRole" | ||
| } | ||
| }, |
| // FQN is the (immutable) memory name. Deriving it from mutable fields such as | ||
| // primaryEntity or owners would change nameHash on update, risking unique-constraint | ||
| // collisions and orphaned references. The link to primaryEntity/owners is captured | ||
| // via the relationship table instead. |
The context_memory table belongs in the 2.0.0 migration. Relocated the MySQL and Postgres DDL verbatim; the 2.0.1 schemaChanges.sql files are restored to their original task_migration_mapping-only content. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 15 changed files in this pull request and generated 4 comments.
Comments suppressed due to low confidence (2)
openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/ContextMemoryRepository.java:281
- The PUT
createOrUpdatepath goes throughContextMemoryMapper.createToEntity, which copies fields straight fromCreateContextMemorywithout consulting the persisted entity. Optional fields that the client omits in the PUT body (e.g.status,memoryType,memoryScope,shareConfig,primaryEntity,summary,title, …) will arrive asnullon theupdatedobject.entitySpecificUpdatethen callsrecordChange(..., updated.get*()), which records and persists those nulls — silently wiping previously-set values on any partial PUT. This is especially problematic forstatus: theoriginal != null && updated != nullguard aroundvalidateStatusTransitionmeans lifecycle enforcement is bypassed when the new status isnull, and the recordChange immediately after blanks out the status field. Consider either documenting that PUT requires the full payload (and forcing required defaults in the mapper, e.g. status), or coalescing each field againstoriginal.get*()when the create payload value is null.
public void entitySpecificUpdate(boolean consolidatingChanges) {
recordChange("title", original.getTitle(), updated.getTitle());
recordChange("summary", original.getSummary(), updated.getSummary());
recordChange("question", original.getQuestion(), updated.getQuestion());
recordChange("answer", original.getAnswer(), updated.getAnswer());
recordChange("memoryType", original.getMemoryType(), updated.getMemoryType());
recordChange("memoryScope", original.getMemoryScope(), updated.getMemoryScope());
recordChange("sourceType", original.getSourceType(), updated.getSourceType());
recordChange(
"sourceConversation", original.getSourceConversation(), updated.getSourceConversation());
recordChange(
"sourceHumanMessage", original.getSourceHumanMessage(), updated.getSourceHumanMessage());
recordChange(
"sourceAssistantMessage",
original.getSourceAssistantMessage(),
updated.getSourceAssistantMessage());
recordChange(
"machineRepresentation",
original.getMachineRepresentation(),
updated.getMachineRepresentation());
// Validate lifecycle transition before recording status change
if (original.getStatus() != null
&& updated.getStatus() != null
&& original.getStatus() != updated.getStatus()) {
validateStatusTransition(original.getStatus(), updated.getStatus());
}
recordChange("status", original.getStatus(), updated.getStatus());
recordChange("shareConfig", original.getShareConfig(), updated.getShareConfig());
openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/ContextMemoryRepository.java:142
setCreatorAsDefaultOwnercallsEntity.getEntityReferenceByName(Entity.USER, entity.getUpdatedBy(), Include.NON_DELETED)unconditionally on every create. If the request comes from a bot/service principal that is not registered as aUser(e.g., bot-only accounts, JWT subjects, or any caller whose principal name does not map 1:1 to ausersrow), this lookup will throwEntityNotFoundExceptionand the POST will fail. Other resources that auto-assign the creating user typically guard this lookup (e.g., wrap in try/catch and fall through to leaving owners null, or skip when the principal is a bot). Consider gracefully skipping default-owner assignment when the principal cannot be resolved to a user.
private void setCreatorAsDefaultOwner(ContextMemory entity, boolean update) {
if (update || !nullOrEmpty(entity.getOwners())) {
return;
}
entity.setOwners(
List.of(
Entity.getEntityReferenceByName(
Entity.USER, entity.getUpdatedBy(), Include.NON_DELETED)));
}
🟡 Playwright Results — all passed (12 flaky)✅ 4140 passed · ❌ 0 failed · 🟡 12 flaky · ⏭️ 86 skipped
🟡 12 flaky test(s) (passed on retry)
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
Explicit engine clause, consistent with the task/search-index tables in the same migration and robust to any server default change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 15 changed files in this pull request and generated 5 comments.
Comments suppressed due to low confidence (1)
openmetadata-service/src/main/java/org/openmetadata/service/resources/context/ContextMemoryMapper.java:57
.withOwners(create.getOwners())overwrites the validated owners list produced byEntityMapper.copy(...). SincestoreOwners(...)does not validate that owner IDs/types exist, this can persist relationships to non-existent/deleted principals. Similarly, the explicitdomainsmapping duplicates the domain lookups already done bycopy(...)(extra DB calls). Prefer relying on thecopy(...)-populatedowners/domains/tagsvalues instead of re-setting them here.
.withMachineRepresentation(create.getMachineRepresentation())
.withOwners(create.getOwners())
.withTags(create.getTags())
.withDomains(
nullOrEmpty(create.getDomains())
? null
: create.getDomains().stream()
.map(
domain ->
getEntityReferenceByName(Entity.DOMAIN, domain, Include.NON_DELETED))
.toList());
…latedEntities Review follow-ups: - ContextMemoryMapper no longer re-sets description/owners/domains/tags/displayName after copy(). copy() sanitizes description (stored-XSS) and validates owners and domains; re-setting the raw request values bypassed both. Only ContextMemory- specific fields are set now. - prepare() now assigns the result of EntityUtil.populateEntityReferences back onto relatedEntities so orphaned/invalid refs are filtered instead of persisted. - ContextMemoryIT Javadoc now references ContextMemoryRepository#setCreatorAsDefaultOwner (the defaultOwners mapper method no longer exists). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Code Review ✅ Approved 7 resolved / 7 findingsIntroduces the ContextMemory entity for standardized storage of Context Center memories, resolving issues with relationship integrity, FQN mutability, and status transition validation. Database schemas are updated for InnoDB compatibility and correct bootstrap migration. ✅ 7 resolved✅ Bug: rootMemory and parentMemory use same Relationship type, indistinguishable
✅ Bug: FQN derived from mutable fields can break on update
✅ Edge Case: validateStatusTransition NPE when 'from' has no map entry
✅ Quality: Test class should be in integration-tests module per conventions
✅ Bug: deleteTo with null fromEntityType may wipe domain relationships
...and 2 more resolved from earlier reviews OptionsDisplay: compact → Showing less information. Comment with these commands to change:
Was this helpful? React with 👍 / 👎 | Gitar |
Copilot review disposition (commit
|
| # | Item | Action |
|---|---|---|
| 1 | mapper re-set description/owners/domains/tags/displayName after copy() |
Fixed — security-relevant. copy() sanitizes description (stored-XSS) and validates owners/domains; the mapper now sets only ContextMemory-specific fields |
| 2 | populateEntityReferences(relatedEntities) return ignored |
Fixed — result assigned back so orphaned/invalid refs are filtered, not persisted |
| 3 / 4 | move DDL out of 2.0.0 into 2.0.1 |
Declined (replied in-thread) — 2.0.0 is the current unreleased in-development migration, not immutable yet; placing it here is deliberate and ensures every deployment applies it |
| 5 | IT Javadoc cited ContextMemoryMapper#defaultOwners |
Fixed — now references ContextMemoryRepository#setCreatorAsDefaultOwner |
| suppressed | owners/domains/tags re-set in mapper | Fixed — same root cause as #1 |
|
|
What
Adds
ContextMemoryas a first-class OpenMetadata entity: a reusable Context Center memory (question/answer/summary, lifecycle status, sharing/visibility, primary/related-entity links, parent/root memory hierarchy).openmetadata-spec:entity/context/contextMemory.json+api/context/createContextMemory.json(org.openmetadata.schema.entity.context.*/api.context.*,@om-entity-type).openmetadata-service:ContextMemoryRepository(cleanEntityRepository,supportsSearch=false), thinContextMemoryResource(standard CRUD at/v1/contextCenter/memories),ContextMemoryMapper,CollectionDAO.contextMemoryDAO(),Entity.CONTEXT_MEMORY.bootstrap:context_memorytable DDL (MySQL + Postgres) innative/2.0.1.ContextMemoryStatusTransitionTest(lifecycle status-transition rules) +ContextMemoryIT(integration CRUD).Why
This entity was previously implemented downstream and is being upstreamed so the core platform owns the storage/CRUD model. AI/semantic-retrieval, vector embedding, and agent-injection layers are intentionally not included here — they remain a downstream concern that composes on top of this entity (thin resource, no search wiring).
Notes for reviewers
context_memory(notcontext_memory_entity). Deliberate: it matches the existing downstream table so existing deployments need no data-rename migration. Happy to add the_entitysuffix + a rename migration if the project prefers strict convention.@Repository(name = "ContextMemoryRepository")(default priority) so downstream can override via the standard priority mechanism.supportsSearch=false.🤖 Generated with Claude Code
Summary by Gitar
countFailuresByJobIdtoCollectionDAOto excludeREADER_RELATIONSHIP_WARNINGevents from job failure counts.storeRelationshipsby removing blanket edge deletion to prevent accidental loss of domain-related relationships.updateFromRelationshipsinentitySpecificUpdateto perform surgical, idempotent updates for entity links while maintaining proper audit logging.This will update automatically on new commits.