Audit log

[ypldan]

StatGPT Backend version

0.3.0

What is the problem this feature will solve?

Currently there's no way for admin to see what was happening with the configurations of different entities within StatGPT.

What is the proposed feature or solution?

Implement immutable audit log, that will be stored in Postgres and will capture the following actions:

1. Add/modify/delete data source
2. Add/modify/delete dataset
3. Add/modify/delete/export/import channel
4. Any reindexing & deduplication operations

Suggested audit log entries:

1. Entity type: data source, dataset, channel
2. Action type
3. Entity id & name
4. Who performed action (uuid)
5. Action trigger(manual, smth else)
6. State before change
7. State after change
8. Timestamp
9. Trace ID

REST API endpoints will be required for accessing audit log in read only mode.

What alternatives have you considered?

No response

[iliia-veselov]

I think we'll need to have trace_id/request_id in this audit.

Also I am wondering whether there will be something similar to rollback to some audit state or not? Because it looks like having "State before change" and "State after change" allows us to plan for it.

[ypldan]

@iliia-veselov

1. trace_id/request_id - agree on that
2. Rollback - definitely not in the initial release, that's outside of scope. However, later, it would be nice to have such feature.

[iliia-veselov]

@iliia-veselov

1. trace_id/request_id - agree on that

2. Rollback - definitely not in the initial release, that's outside of scope. However, later, it would be nice to have such feature.

ok. So if we are having states before and after only for admins to see changes, I think it may be a good idea to also store differences or updated fields, so admin could see changes easier

[Fedir-Yatsenko]

@iliia-veselov

1. trace_id/request_id - agree on that

2. Rollback - definitely not in the initial release, that's outside of scope. However, later, it would be nice to have such feature.

ok. So if we are having states before and after only for admins to see changes, I think it may be a good idea to also store differences or updated fields, so admin could see changes easier

I also thought about this, but it might be difficult to implement, since the most important and frequently changed field in the app is a JSONB object with a complex structure. Saving states before and after changes seems like a simpler solution. And in the future, we can implement dynamic calculation of the difference based on these states.

[Fedir-Yatsenko]

By the way, I just realized that we don't need to store the state before changes, since we can get it from the previous log entity. What do you think?

[Fedir-Yatsenko]

💡 Alternative solution can be using rules or triggers in the PostgreSQL. Here is an example from the PostgreSQL documentation of a rule that conditionally writes a log entry when an UPDATE is executed:
https://www.postgresql.org/docs/current/rules-update.html#RULES-UPDATE-HOW-FIRST

NOTE: Just saved this here, no action required for now, but it might come in handy in the future.