Need help for On Prem object stores (MinIO) ingestion

[siva-vunet]

Hello,

Currently I am trying to utilize the KDI application to ingest data from kafka into a delta table in our MinIO object store. But I am facing some issues here.

Issue:

When i use the MinIO port as the endpoint url for KDI it runs fine for a while. I can see the delta table being created in my object store as expected but after one ingestion the application throws the following error and quits,

I don't know much about dynamoDB and locking but I followed my idea about this error arising due to the application not being able to create a locks table using the MinIO port as the endpoint url.

Ideas

At this point after discussing with my team I saw two paths:

A) Add a separate endpoint url for the dynamoDB service into the application
B) Bypass the dynamoDB requirement altogether

For both of these options I have to make modifications to the source code itself even though I didn't know anything about rust. Regardless I powered through all the code lines and was able to identify the dynamoDB service related processes and calls. I made a few additions which i thought would make the application work and when I try to do the cargo build step I keep encountering these errors;

Later changed the dependencies in the KDI Cargo.toml file to point to the correct crates, like so (last two lines);

(Also I kept encountering dependency version conflicts the first time I ran cargo build, I just made the required changes to the versions in the toml file, thats why you see the version as "22" for arrow and parquet here)

And now I came across this error,

So, I need some real help on what I am doing wrong here. I could be completely off the track here by following these ideas to make the application work. Is there a better way to do this ? Or is there a mode where KDI works for on prem object stores already ?

Any help would be deeply appreciated.