Project-level data retention settings

[sudhanshu746]

Describe the feature or potential improvement

Thanks for providing this great tool, a feature could be a retention settings for trace purge automatically like all traces > 30 days old.

Additional information

No response

[marcklingen]

Agree, with the migration to Langfuse v3, setting expiration dates on single rows will become simpler to handle data retention at scale in langfuse. Thanks for raising this!

[AlithAnar]

If there is a safe way to delete traces from PostgreSQL in current v2 setup could you please share from which tables can we manually delete records? are they connected with foreign keys with cascading deletetion feature or we have to manually delete some disconnected remnants?

[msabramo]

I am also interested in this. Folks in my org would like to be able to limit data storage to 24 hours to comply with regulations.

LangSmith has this feature: https://docs.smith.langchain.com/self_hosting/configuration/ttl

[marcklingen]

Quick update on this thread, retention capabilities will be added in the upcoming v3 of Langfuse as a project-level setting.

Langfuse Cloud

If you run on Langfuse Cloud, please reach out to us and we can manage retention settings for you project on our end internally.

Self-hosted

If you run self-hosted, you can safely delete data from the following tables. You can easily run these queries on a cron schedule. If you db is huge, please keep in mind that vacuuming might impact db performance while it is happening.

v3 (Clickhouse)

delete from traces
where project_id = ''
and timestamp < now() - interval N days

delete from observations
where project_id = ''
and start_time < now() - interval N days

delete from scores
where project_id = ''
and timestamp < now() - intervval N days 

v2 (Postgres)

DELETE FROM traces
WHERE "timestamp" < '2024-08-01'

DELETE FROM observations
WHERE "start_time" < '2024-08-01'

DELETE FROM scores
WHERE "timestamp" < '2024-08-01'

Q&A

are they connected with foreign keys with cascading deletetion feature or we have to manually delete some disconnected remnants?

No foreign key constraints are applied that would block a deletion of records. Background: Records need to be compliant with data privacy regulation as well as data needs to be ingested into langfuse without a constraint on the order of records.

[akashkaveti]

Hi @marcklingen, Instead of manual DELETE operations, can we implement ClickHouse TTL on these tables for automatic data cleanup?

ALTER TABLE traces MODIFY TTL timestamp + INTERVAL N DAY DELETE;
ALTER TABLE observations MODIFY TTL start_time + INTERVAL N DAY DELETE;
ALTER TABLE scores MODIFY TTL timestamp + INTERVAL N DAY DELETE;

[Steffen911]

@akashkaveti You can do this manually for self-hosted instances, yes.

[synv-2692]

Hi @Steffen911, Can I use ClickHouse’s TTL feature combined with conditions to delete old data

ALTER TABLE traces
MODIFY TTL
    timestamp + INTERVAL 30 DAY DELETE WHERE project_id = 'project_abc',
    timestamp + INTERVAL 90 DAY DELETE WHERE project_id = 'project_xyz';

[Steffen911]

@synv-2692 Yes, that should work too for a self-hosted deployment.

[Steffen911]

You may want to patch the observations and scores table as well.

[sudhanshu746]

Thank you, @marcklingen for sharing options.I am closing this thread for now.

[nitinkalra-bst]

are you not creating these tables partitioned by date? can't we just drop the older partitions?

[nitinkalra-bst]

@marcklingen please check this. It takes a lot of time to run delete command

[marcklingen]

Currently they are not partitioned b date but indexed by date. This will be changed in v3 as the data will be more immutable and thus easier to partition.

[bharath-muthineni]

@sudhanshu746 , is there any open request or github issue, just to know when this was implemented? 😅

[Steffen911]

@TheErk Yes, that would be possible and should work well.

[Pratish315]

Hi @marcklingen, are these ALTER TABLE statements built into Langfuse or would I need to exec into the pods and run these queries? (for self-hosted)

Is there any plan to build this into Langfuse so the engineer doesn't have to run this? (having a retention parameter in values file within in the helm chart)

[Steffen911]

@Pratish315 Those queries would run within ClickHouse.

There is a data retention feature within Langfuse, but it's part of the commercial offering. See https://langfuse.com/docs/data-retention for details.

[nkalra0123]

how to manage minio disks in the sekf hosted version? Do you have any suggestions on that? i think they take the most disk

[Steffen911]

@nkalra0123 With the Langfuse retention feature we should delete things from MinIO automatically.
For other tiers, minio lifecycle rules may work: https://min.io/docs/minio/linux/administration/object-management/object-lifecycle-management.html#object-expiration.