Enabling delta-rs logging

[coccor]

I am trying to investigate why it takes so long to write just a few rows inside an existing Delta Lake.
For that I want to see the logs generated by delta-rs, but I couldn't find any way of doing that.

So far I tried setting RUST_LOG environment variable to debug, that that doesn't seem to have any effect at all.

Is there a recommended way of doing this ? I haven't been able to find anything.

[mightyshazam]

There is no way to do this by default because delta-rs uses the tracing crate. The place to implement this is here where we create the runtime. The reason I did not do this already is because delta-rs wasn't using the tracing crate when I initially developed this library. The most analogous code is in the delta python code.
Basically, we just need to update the runtime options to take optional tracing parameters and construct the tracing subscriber. Then logging/tracing from delta-rs will propagate to dotnet. The only question is whether to configure tracing directly in rust or create a bridge subscriber and integrate it with System.Diagnostics.

[coccor]

I followed the instructions and I got it to work, but now I'm faced with a different problem.
I can't find a way to configure data fusion.
I would like to set:
datafusion.execution.target_partitions
datafusion.execution.batch_size
datafusion.optimizer.enable_round_robin_repartition
datafusion.execution.coalesce_batches

Is there a place to set these ?

[mightyshazam]

I don't know of any place to set these values without writing the rust code specifically.