support for add\remove file(s) write transactions

[maqister]

hi guys,

Like in the title I need support to commit new delta table version which removes some data files and adds some data files.

1. Are you guys in the process of making this capability available to delta kernel consumers?

2. If this is entirely out of your scope then my conclusion would be that I can't use delta-kernel,
   but if it is in the scope I would love to discuss timelines and if there is lack of the contributor available to tackle it I could help.

I found this but it is unclear if author works on it and what are the ETAs.
delta-io/delta#3988

thanks,
Krystian

[zachschuermann]

Hi Krystian! Thanks for the message! we already have support for appends (adding files) but not yet deletions.

1. Yes! Appends are done and updates/deletions will come in the future - would love to hear more about your use case and ideal API
2. For timelines I will do some sizing and report back here :) and yes we are always in need of contributors!

The issue you linked is from the delta repository which includes scala/spark and java kernel implementations. Which language/system do you intend to use the kernel in?

[maqister]

Sorry for confusing you with the linked item, I linked it because it extensively mentions delta-kernel-rs.

As far as my use-case goes I am implementing data layout optimizer of sorts in c++ so I am using ffi-headers to call into your delta-kernel rust dll. It will be pip installable python utility, that data engineers can run in their notebooks.

I need both remove files and add files.

For example

delta version 0
a.parquet
b.parquet
c.parquet
d.parquet
e.parquet

my tool moves table to version 1 (data moves around but actual content stays the same)

delta version 1
f.parquet (merge of a b c)
d.parquet // keep
e.parquet // keep

so in order to do that I would submit delta transaction that adds f.parquet, and removes a.parquet b.parquet c.parquet

I started building my prototype with this library, reading delta log is working, but I am stuck now.
I will be doing reading parquet and writing parquet on my own, but I want to use delta-kernel to manage delta protocol.

[maqister]

hi @zachschuermann , @sebastiantia

looks like this PR will deliver what I need?

#779

I see e.g. following

    // 1st commit: adds `fake_path_2` & removes `fake_path_1`
    write_commit_to_store(
        &store,
        vec![
            Action::Add(Add {
                path: "fake_path_2".into(),
                data_change: true,
                ..Default::default()
            }),
            Action::Remove(Remove {
                path: "fake_path_1".into(),
                data_change: true,
                ..Default::default()
            }),
        ],
        1,
    )?;

This delivers exact API that I need,
I am just not sure how the API will look for C++ via ffi-headers, but this looks very promising?!

Could you kindly confirm if my understanding is accurate?