Delta Table Loading Issue in Azure Function App

[kimzed]

We're experiencing an issue loading Delta tables in an Azure Function App using delta-dotnet. Key details:
Working scenario:

Works locally on Windows
Using path format:

filePath = $"abfss://{containerName}@{storageAccountName}.dfs.core.windows.net/{path-to-delta-table}"

Using access_key for storage authentication

var options = new TableOptions
{
};
options.StorageOptions["access_key"] = [access key];

Issue:

Same code fails when deployed to Azure Function App

Error: DeltaLake.Errors.DeltaRuntimeException: Failed to read delta log object: Generic HTTP client error: builder error
Error occurs during table loading operation

For this line:

DeltaRuntime runtime = new DeltaRuntime(RuntimeOptions.Default);
_table = DeltaTable.LoadAsync(runtime,
    filePath,
    tableOptions,
    CancellationToken.None).Result;

Has anyone encountered similar issues or have suggestions ?

[kimzed]

we are using 0.1.0 for deltalake.dotnet

failing method is

    internal unsafe async Task<Table> LoadTableAsync(Memory<byte> tableUri, DeltaLake.Table.TableOptions options, System.Threading.CancellationToken cancellationToken)
    {
        TaskCompletionSource<Table> tsc = new TaskCompletionSource<Table>();
        using Scope scope = new Scope();
        DeltaLake.Bridge.Interop.TableOptions tableOptions = default(DeltaLake.Bridge.Interop.TableOptions);
        tableOptions.version = (options.Version.HasValue ? ((long)options.Version.Value) : (-1L));
        tableOptions.without_files = (options.WithoutFiles ? ((byte)1) : ((byte)0));
        tableOptions.log_buffer_size = options.LogBufferSize ?? 0;
        tableOptions.storage_options = ((options.StorageOptions != null) ? scope.Dictionary(this, options.StorageOptions) : null);
        DeltaLake.Bridge.Interop.TableOptions value = tableOptions;
        Methods.table_new(Ptr, scope.Pointer(scope.ByteArray(tableUri)), scope.Pointer(value), scope.CancellationToken(cancellationToken), scope.FunctionPointer<TableNewCallback>(delegate (RawDeltaTable* success, DeltaTableError* fail)
        {
            if (cancellationToken.IsCancellationRequested)
            {
                Task.Run(() => tsc.TrySetCanceled(cancellationToken));
            }
            else if (fail != null)
            {
                Task.Run(() => tsc.TrySetException(DeltaRuntimeException.FromDeltaTableError(Ptr, fail)));
            }
            else
            {
                Task.Run(() => tsc.TrySetResult(new Table(this, success)));
            }
        }));
        return await tsc.Task.ConfigureAwait(continueOnCapturedContext: false);
    }

[mightyshazam]

@kimzed Check your environment variables in both environments. It is possible something about the function environment is triggering different behavior. The process appears to be failing before ever attempts a request because it can't build an http client.

[kimzed]

What environment variables are you talking about exactly?
for credentials we checked and everything is correct.
for other environment variables we checked in settings but nothing related pops out. I looked for proxy or http related variable

We tried to lower security rules in the azure function app settings to try to isolate the error more but we do not get a different result so far.

im wondering as well when the package is going to get a new release in nuget? what we have seems quite outdated

[mightyshazam]

I don't know which environment variable would cause the build statement to fail. I was just suggesting looking at differences. Are the OS and .net version the same in the function as the local environment?
Something is different, but without knowing what, it's hard to figure out.
We were waiting for the kernel to gain write support and/or come out of the incubator. Now that it is out and the default engine has write support, we can move forward. I think it should be possible to get a new release out next month.

[kimzed]

thanks for your answer

the function app works on ".NET 8 (LTS), isolated worker model" on a linux environment

we work on a windows 10

I could try to replicate that on a linux OS

[kimzed]

I also tried on a linux machine but did not get any issue

[kimzed]

does someone know on which version the used rust packages are?

[kimzed]

so we tried a couple of things, identified the differences in env variables
I tried to remove all managed identities related variables on the function app to try to force it to use the access key
also tried additional managed identities options in table options but not working either

so we are thinking of switching to another tech stack to solve this issue

[mightyshazam]

@kimzed Since this is still experimental, the CI didn't establish any tags for releases. However, I pushed a tag so that people can see the current nuget version. The current 0.1.0 version references delta-rs rust-v0.20.0.

[mdrakiburrahman]

@kimzed - just curious, can you repro this (on the old delta-dotnet Nuget you originally tried in Cloud) using Azure Function Emulator locally?

https://learn.microsoft.com/en-us/azure/azure-functions/functions-develop-local
https://learn.microsoft.com/en-us/azure/azure-functions/functions-run-local?tabs=windows%2Cisolated-process%2Cnode-v4%2Cpython-v2%2Chttp-trigger%2Ccontainer-apps&pivots=programming-language-csharp

Or does it only repro in their Cloud runtime?

[mdrakiburrahman]

Also, if it's only in Cloud, could you paste a JSON representation of the Azure Function? (i.e. what the ARM props say):

[dmunch]

I'm planning to use this lib in the mid-term on Azure Functions as well, so got interested in this and had a quick look.

It's a long shot, but since delta-dotnet links to native code, I followed the dependency chain and found that the http client library reqwest used by the rust implementation depends in turn on native-tls, which on Linux depends on a shared library to be installed on the system. Since it's notoriously tricky to get the right version of OpenSSL on a Linux system (at least in my experience), this might be the cause of the issue: The Azure Function host on Linux might no have the right version installed.

As a solution it seems to be possible to statically compile OpenSSL by setting the vendored feature of native-tls (https://docs.rs/native-tls/latest/native_tls/#cargo-features) - I've got no experience in Rust, but maybe it's as simple as adding the following line (or something similar) to this projects Cargo.toml ?

[dependencies]
native-tls = { version = "0.2", features = ["vendored"] }

[dmunch]

It seems that most recent Microsoft Docker Images (and AFAIR Azure Functions run in a Docker Container) have removed OpenSSL 1.X (dotnet/dotnet-docker#4740) due to it being EOL. It's not entirely sure to me though if reqwest supports OpenSSL 3? seanmonstar/reqwest#1978

[mightyshazam]

@dmunch It is true that the openssl dependency is usually a problem. However, the rust dependencies all use rustls to avoid that pitfall. The latest version includes delta-kernel-rs, and that use reqwest with default features, so that may be an issue.