[backend] Adapt changes and build history on top (#8237) #13775
Conversation
Gwendoline-FAVRE-FELIX
added
the
filigran team
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #13775 +/- ##
==========================================
+ Coverage 31.12% 31.13% +0.01%
==========================================
Files 2923 2927 +4
Lines 193813 194227 +414
Branches 39564 39596 +32
==========================================
+ Hits 60317 60468 +151
- Misses 133496 133759 +263
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
richard-julien
force-pushed
the
issue/8237-jri
branch
from
9d54d10 to
c98855f
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
richard-julien
force-pushed
the
issue/8237-jri
branch
2 times, most recently
from
cbbaf80 to
199235e
Compare
There was a problem hiding this comment.
Pull request overview
This PR refactors the history and audit system to use a new change tracking structure (history_changes with history_added/history_removed) replacing the previous changes structure (previous/new/added/removed). The changes enable better tracking of modifications with both raw and translated values, improved permission filtering for history entries, and enhanced UI components for displaying audit details.
Key changes:
- Replaced
Changeinterface fields (previous,new,added,removed) withhistory_addedandhistory_removedarrays containingChangeValueobjects withrawand optionaltranslatedfields - Updated GraphQL schema to reflect new
HistoryChangeandDataChangetypes - Added permission-based filtering for history entries through
historyFilteringoption - Refactored frontend components to use new drawer components and query individual logs by ID
Reviewed changes
Copilot reviewed 41 out of 43 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| opencti-platform/opencti-graphql/src/types/event.d.ts | Updated Change interface structure |
| opencti-platform/opencti-graphql/src/database/middleware.js | Refactored buildChanges and removed generateUpdateMessage |
| opencti-platform/opencti-graphql/src/database/generate-message.ts | Completely rewritten message generation logic |
| opencti-platform/opencti-graphql/src/database/engine.ts | Added history filtering restrictions and inner hits support |
| opencti-platform/opencti-graphql/src/domain/log.ts | Added new query functions with history filtering |
| opencti-platform/opencti-graphql/config/schema/opencti.graphql | Updated GraphQL schema types |
| opencti-platform/opencti-front/src/private/components/common/drawer/HistoryDrawer.tsx | Complete rewrite to use lazy loading with new schema |
| opencti-platform/opencti-front/src/private/components/settings/activity/audit/AuditDrawer.tsx | New component for audit detail display |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
opencti-platform/opencti-graphql/src/database/generate-message.ts
Outdated
Show resolved
Hide resolved
richard-julien
force-pushed
the
issue/8237-jri
branch
2 times, most recently
from
bd69a54 to
6c93f1e
Compare
SamuelHassine
force-pushed
the
master
branch
7 times, most recently
from
1c222ef to
2cb4539
Compare
richard-julien
force-pushed
the
issue/8237-jri
branch
from
d893806 to
f37f5e5
Compare
richard-julien
requested review from
aHenryJard,
marieflorescontact and
xfournet
...rm/opencti-graphql/src/modules/attributes/stixSightingRelationship-registrationAttributes.ts
Outdated
Show resolved
Hide resolved
|
In Data > Activity, in a platform with only 'old events', I checked 'include knowledge' and add a filter, an error occurs. 
|
|
The list of events in Data > Activity is blocked after a few lines, we can't go below (it happens only if 'include knowledge' is checked): |
|
The messaging refacto is great ! It will improve lots of things and make readability better. We are ok that this PR does not fix any attribute access rights in history messages? |
|
'Shared with' history lines are visible for users without the 'Restrict organization access' capability 
|
richard-julien
force-pushed
the
issue/8237-jri
branch
2 times, most recently
from
7e4e3ab to
6c17d34
Compare
richard-julien
force-pushed
the
issue/8237-jri
branch
2 times, most recently
from
0815b29 to
f981f73
Compare
|
@Archidoit, global approach is now changed. You can review again. Thanks |
opencti-platform/opencti-graphql/src/modules/workflow/workflow-types.ts
Outdated
Show resolved
Hide resolved
|
(Only a UI small comment that is only my opinion): In knowledge log details, when we hover a line, its color changes whereas we can't click on it. I think the color shouln't change since a line is not clickable 
|
|
When we add a user in Assignee, the history line display 'adds...'. Whereas when we add an individual in Author, the history line display 'replaces...' even if there were no author before. Same for all the fields with single values (working well for fields with multiple values, like labels or markings). 
|
|
Boolean values history message are not correct. There are always 'No replaced by No'. Example in a malware, the 'is family' field was set to True and I turned it to False. The log displayed is : 
--> Same log display if I change the value from False to True |
| filters: FilterGroup | ||
| search: String | ||
| ): LogConnection @auth(for: [SETTINGS_SETACCESSES, SETTINGS_SECURITYACTIVITY, VIRTUAL_ORGANIZATION_ADMIN]) | ||
| audit(id: ID!): Log @auth(for: [SETTINGS_SETACCESSES, SETTINGS_SECURITYACTIVITY, VIRTUAL_ORGANIZATION_ADMIN]) |
There was a problem hiding this comment.
I think a user with SETACCESSES or ACTIVITY can't view audit logs (but only history logs) if he has not the SECURITYACTIVITY capability.
There was a problem hiding this comment.
I apply the same security as audits, i dont think there is any issue with that
richard-julien
force-pushed
the
issue/8237-jri
branch
from
d3f8f7b to
6bf46d8
Compare
|
For dates, if the field has no value before and we add a new value, a fake 'removed' value is displayed (origin 1970 or 5138 value) : 
|
Target
The primary target of this pull request is the History and Audit system within the OpenCTI backend and frontend. It aims to standardize how data changes are tracked, stored, and visualized, shifting the resolution of human-readable messages from write-time to read-time.
Improvements
logContextDataBatchLoader). This significantly improves write performance by removing the need to synchronously resolve entity names during ingestion.changes_addedandchanges_removed) replacing the legacy mix ofprevious,new,added, andremoved.generate-message.js.engine.ts) to support searching within the new history structure, including specific fields likechanges_added.rawandchanges_removed.raw.HistoryDrawerand related components to consume the new data structure, simplifying the view to "Added" vs "Removed" columns and using GraphQL queries for data fetching.attribute-definition.ts(e.g., specificIdAttribute,TextAttribute), enhancing code reliability.Impact Analysis
Backend
context_data.history_changesin logs now strictly follows the{ field, changes_added, changes_removed }format.context_data.history_changesstructure.Frontend
HistoryDrawerQueryusing the log ID, reducing initial load weight.