can we delete old json files in the transaction log

[djouallah]

currently I insert 288 files per day, which generate 288 json files in the transaction log, is there a way to delete old json ?

[dennyglee]

Hey @djouallah by default, the log retention is 30 days. You can certainly modify it for your scenario via the delta.logRetentionDuration property which you set with the ALTER TABLE SET TBLPROPERTIES SQL method (reference)

[yefetBenTili]

Is there a way to edit this configuration within delta-rs and not Databricks?

[ion-elgreco]

@yefetBenTili currently it's only possible to set during a create table operation or write operation when table does not exist yet. We still need to build this alter tbl properties operation in Rust and then expose it to Python.

[djouallah]

Thanks does this works with Delta Rust ?

[rtyler]

Unfortunately right now I don't believe delta-rs does any transaction log culling on reads or writes. I think that would be a good improvement though!

[yefetBenTili]

I had a similar issue and thanks to @ion-elgreco who pointed me to cleanup_metadata() but it requires an explicit call somewhere in your logic

[ion-elgreco]

@djouallah do you have an example of before and after

[djouallah]

never mind, I think it is triggered only with a massive number of logs entry, basically when the checkpoint become big enough

[djouallah]

Sorry to bother again but can you give an example how to define retention period at the table creation not very clear from the documentation

[djouallah]

@ion-elgreco Sorry to bother again but can you give an example how to define retention period at the table creation not very clear from the documentation

[ion-elgreco]

@djouallah
I believe you should do it like this:

DeltaTable.create(configuration = {"delta.logRetentionDuration": "<value>"})