Missing ReadOnlySpan<char> overloads for APIs in System.IO.Path, System.IO.Directory and System.IO.File

[Aniobodo]

Some APIs like .Combine, .Exists(), .GetRelativePath(), etc. currently do not have direct ReadOnlySpan-of-char overloads.
Is there any reason behind the missing overloads or is it just a matter of time?

[huoyaoyuan]

Using spans are only meaningful when the API is using a truncation of path.

Combine can operate on parent directory path that may be a span from GetDirectoryName, however there isn't a way to represent undetermined number of spans (Span<Span>).

Exists is a syscall, which requires null termination.

GetRelativePath needs concatenation of relative components.

[colejohnson66]

Even then, concatenation of spans into a singular string would avoid temporaries. Path.Combine(a.ToString(), b.ToString()) compared to Path.Combine(a, b) (assuming a and b are spans) would remove two temporary string allocations.

[huoyaoyuan]

The filename for File.Open will almost certainly be passed to P/Invoke, which usually uses null-terminated string. A span doesn't help much for that.

[ufcpp]

Here is a benchmark:

benchmark code

using BenchmarkDotNet.Attributes;
using Microsoft.Win32.SafeHandles;
using System.Buffers;
using System.IO.Enumeration;
using System.Runtime.InteropServices;

#if DEBUG
var b = new Benchmark();
try
{
    b.Setup();

    await b.FileOpenRead();
    await b.Win32CreateFile();
}
catch { }
finally { b.Dispose(); }
#else
BenchmarkDotNet.Running.BenchmarkRunner.Run<Benchmark>();
#endif

[MemoryDiagnoser]
public class Benchmark
{
    DirectoryInfo? _temp;

    [GlobalSetup]
    public void Setup()
    {
        _temp = Directory.CreateTempSubdirectory();
        var d = _temp.FullName;

        for (int i = 0; i < 1000; i++)
        {
            File.WriteAllText($"{d}/{i}.txt", $"{i}");
        }
    }

    [GlobalCleanup]
    public void Dispose()
    {
        _temp?.Delete(true);
    }

    [Benchmark]
    public Task FileOpenRead() => RunAsync(_temp!.FullName, new ReaderWithFileOpenRead());

    [Benchmark]
    public Task Win32CreateFile() => RunAsync(_temp!.FullName, new ReaderWithWin32CreateFile());

    private static async Task RunAsync(string path, Reader reader)
    {
        var sum = 0;
        await reader.ReadFilesAsync(
            path,
            data =>
            {
                if (int.TryParse(data, out var i)) sum += i;
            });
    }
}

abstract class Reader
{
    public async ValueTask ReadFilesAsync(string directoryPath, Action<ReadOnlySpan<byte>> action)
    {
        foreach (var fs in new FileSystemEnumerable<FileStream>(
            directoryPath,
            OpenFile))
        {
            var len = (int)fs.Length;
            var rent = ArrayPool<byte>.Shared.Rent(len);
            try
            {
                var m = rent.AsMemory(0, len);
                await fs.ReadExactlyAsync(m).ConfigureAwait(false);
                action(m.Span);
            }
            finally
            {
                ArrayPool<byte>.Shared.Return(rent);
                fs.Dispose();
            }
        }
    }

    protected abstract FileStream OpenFile(ref FileSystemEntry entry);
}

class ReaderWithFileOpenRead : Reader
{
    protected override FileStream OpenFile(ref FileSystemEntry entry) => File.OpenRead(Path.Combine(entry.Directory.ToString(), entry.FileName.ToString()));
}

partial class ReaderWithWin32CreateFile : Reader
{
    protected override FileStream OpenFile(ref FileSystemEntry entry)
    {
        return OpenFile(entry.Directory, entry.FileName);
    }

    private static FileStream OpenFile(ReadOnlySpan<char> directory, ReadOnlySpan<char> fileName)
    {
        Span<char> path = stackalloc char[directory.Length + fileName.Length + 2];
        directory.CopyTo(path);
        path[directory.Length] = Path.DirectorySeparatorChar;
        fileName.CopyTo(path[(directory.Length + 1)..]);
        path[^1] = '\0';

        return new(OpenHandle(path), FileAccess.Read, 0);
    }

    private static SafeFileHandle OpenHandle(ReadOnlySpan<char> path)
    {
        if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
        {
            var handle = CreateFile(path, 0x80000000, 1, 0, 3, 0x80, 0);
            return new(handle, handle != 0);
        }
        else
        {
            return File.OpenHandle(path.ToString());
        }
    }

    [LibraryImport("kernel32.dll", EntryPoint = "CreateFileW", StringMarshalling = StringMarshalling.Utf16)]
    private static partial nint CreateFile(
        ReadOnlySpan<char> lpFileName,
        uint dwDesiredAccess,
        uint dwShareMode,
        nint lpSecurityAttributes,
        uint dwCreationDisposition,
        uint dwFlagsAndAttributes,
        nint hTemplateFile);

    [LibraryImport("kernel32.dll")]
    [return: MarshalAs(UnmanagedType.Bool)]
    private static partial bool CloseHandle(nint hObject);

}

Method	Mean	Error	StdDev	Gen0	Allocated
FileOpenRead	30.41 ms	0.432 ms	0.404 ms	343.7500	4633.08 KB
Win32CreateFile	29.02 ms	0.463 ms	0.433 ms	-	164.95 KB

The string overload results in a very large number of allocations.

[jilles-sg]

When opening many files in a directory on Unix-like systems, it would be better to use functions like openat and fstatat that take a file descriptor to a directory and a relative path. This is both faster (not only because of less string manipulation, but more importantly because it relieves the kernel from looking up the directory pathname repeatedly) and necessary to descend into hierarchies deeper than the maximum pathname length accepted by system calls (although implementing this without consuming one file descriptor per directory level is not trivial). It is also necessary but not sufficient to defend against race conditions where an attacker replaces a directory with a symlink (attempting to cause the traversal to leave the original directory). Historically (before openat existed), this had to be done by changing the current directory, which is a non-starter in a multithreaded process.

[ufcpp]

Ideally, FileSystemEntry would have OpenFile method to directly obtain SafeFileHandle or Stream.

[jilles-sg]

If you can make do with Path.Join instead of Path.Combine, it has overloads that accept spans. Note that it is sour if you have a string containing an absolute path, convert it to a span and then pass it as the last parameter to Path.Combine: this unnecessarily copies a string.

[ufcpp]

the Join overload which has ReadOnlySpan arguments also returns a string, and does the allocation occur?