Conversation
|
just a tiny bit worried about the overwrite of edge ids. I know ctkp likes those to be set internally? |
|
We confirmed with them that it was ok to overwrite the edge ids as long as
we preserved the other attributes which have the same ids as links.
|
|
Just to clarify, Pydantic validates association 'id' fields as present, so the ingests need to set them, but they are otherwise irrelevant in that they are simply reassigned (overwritten, idempotently and globally uniquely) as the various knowledge-specific ingest data are pulled into the merge? |
|
Based on our MUTT discussion today: @RichardBruskiewich the edge ids assigned in ingests won't be completely irrelevant, we'll use them to track edges through our system, and to better track merges, but most of them might also be overwritten during normalization or merging. I'll also implement a new section of merge metadata that includes the specific edge id mappings when mergers occur; and a way to let valid UUIDs from upstream "pass through" sources persist through the ingest, and even through normalization and merging if nothing changes. Let's hold off on this PR until then I guess. |
Roger that @EvanDietzMorris ... No change in any of the ingests (within which Pydantic enforces the setting of Association.id fields)
It's your PR so... this ball is in your court, LOL. |
4 participants
This bumps ORION to a new version which has a significantly improved node/edge merging algorithm. It's quite a bit faster (2-3x in benchmarks so far) and consumes less memory (half-ish? but not well tested).
It also adds a couple flags that tell ORION to add UUID edge ids to every edge during merging, overwriting existing ones.
The pinned version of robokop-orion was removed from the pyproject.toml recently but I'm not sure why, so this also sets one again (was that intentional @sierra-moxon?).