Azure Storage Gen2 (hierarchical namespaces) - use DFS endpoint to improve performance

[keen85]

Hi,
Azure Storage comes in different flavors. A "regular" Azure Storage account is a classic object store and it comes with blob endpoint (blob.core.windows.net, blob.fabric.windows.com).

Azure Storage Account with hierarchical namespaces enabled (aka. Azure Data Lake Gen2, Fabric OneLake) come with a second "DFS" (Distributed Filesystem) endpoint: dfs.core.windows.net (dfs.fabric.windows.com) that has it's own REST-API: https://learn.microsoft.com/en-us/rest/api/storageservices/data-lake-storage-gen2

This endpoint allows for potentially massive performance improvements in the following scenarios:

• recursive directory listings
• renaming or moving files or folders
• deleting directories with many files

These are single atomic operations via DFS. With the Blob endpoint they require many individual blob operations and are therefore slower.

To my understanding, DFS-URLs are already accepted but ultimately only the blob endpoint is used by object_store.

I propose implementing the DFS endpoint in object_store and introducing a feature-flag allowing user to specify if DFS endpoint should be used.