Introduce an interim 'Utf8String' ref struct type for UTF-8 string literals for type (pin) safety

[dracocephalum]

I love the new "u8" string literals and they are very efficient to work with when comes to interop with old unmanaged code that takes ASCII strings. I believe these utf8 string are allocated in the .data section, so GC won't touch them and they are pretty much already "pinned", so I came up with code like below to get the pointer address of these literals:

public static byte* AddrOf(ReadOnlySpan<byte> utf8StringLiteral)
    => (byte*)Unsafe.AsPointer(ref Unsafe.AsRef(in MemoryMarshal.AsRef<byte>(utf8StringLiteral)));

As you can see, however, since all utf-8 string literals are ReadOnlySpan<byte>, this method is very dangerous, because we cannot tell if the parameter passed in is a real utf-8 string literal that's already pinned, or just some managed, unpinned ReadOnlySpan<byte> memory blocks.

I know this is a very corner case and rare usage, but it would be nice to have an intermediate type like Utf8String so that I could at least constraint the passed in parameter a bit for type (pin) safety.

Also hopefully I got it right about the non-alloc and no-GC characteristics of the utf-8 string literals :)

Thanks.

[huoyaoyuan]

but it would be nice to have an intermediate type like Utf8String so that I could at least constraint the passed in parameter a bit for type (pin) safety.

You don't get pin safety if there is a string-like Utf8String. Strings are allocated on heap and are unsafe to be operated like that. You code is already creating a GC hole if it accepts a (non-literal) string.

Indeed, the pattern you are using that accepting a memory-safe object and returning a pointer is never GC-safe. If all you need is to pass the string to an unmanaged function, you can just use fixed on the ReadOnlySpan<byte>, or relies on LibraryImport generator to do that for you.

Or, you can write an analyzer at C# level to force only literals can be passed into your function. This also helps string.

[dracocephalum]

How about a ref struct Utf8String or ref struct Utf8StringLiteral instead of heap string-like? I think currently the compiler will eventually turn "my string"u8 into something like new ReadOnlySpan<byte>(0x12345678 /* memory address */, 9). So if we have a ref struct like Utf8String then it could become new Utf8String(0x12345678 /* memory address */, 9), which can be further cast to ReadOnlySpan via operator overload and potentially compiler can simplify this so no performance loss is caused?

I think the analyzer option is acceptable, but if this is needed to be exposed in a library then it would be better to guard it via type safety.

One thing I would like to make sure is, are all utf-8 string literals stored in the .data sections and are directly referenced so no alloc/GC involved? e.g. GC are not moving utf-8 string literals (not those wrapped around byte[]) around? I would like to avoid fixed keyword because it is generating extra local variables on stack, and from the assembly generated it cannot be simplified to a single load instruction. I have code that are directly translated from C/C++ into C# via automatic translator, so appending the "u8" suffix is a lot easier than updating all the const strings to be wrapped in fixed blocks, and I would expected performance can be faster without fixed keyword.

[huoyaoyuan]

How about a ref struct Utf8String or ref struct Utf8StringLiteral instead of heap string-like?

It will effectively be an alias of ROS<byte>. Note that there's no dedicated representation for string literals either.

One thing I would like to make sure is, are all utf-8 string literals stored in the .data sections and are directly referenced so no alloc/GC involved?

Yes, as an implementation detail. The spec doesn't force that.
Also note that assemblies can be collectible. Thus, everything embedded in the assembly can be collected.

I would like to avoid fixed keyword because it is generating extra local variables on stack, and from the assembly generated it cannot be simplified to a single load instruction.

and I would expected performance can be faster without fixed keyword.

The performance difference is negligible in this case. The base overhead of doing a P/Invoke is much higher than that.
Even if the callee is managed code, the overhead of fixed is just a single MOV if there's nothing to pin.

I have code that are directly translated from C/C++ into C# via automatic translator, so appending the "u8" suffix is a lot easier than updating all the const strings to be wrapped in fixed blocks

If your callee is managed code, updating them to accept ROS<byte> instead of byte* will results in better performance and safety.

If your callee is native (P/Invoke), updating them to LibraryImport with ROS<byte> will solve the complexity for you.

[neon-sunset]

As @huoyaoyuan noted, pinning itself is effectively free.

In fact, even when calculating a pointer offset into an array and then pinning that, it's still almost free aside from the array being, well, pinned.

If you want "as cheap as it gets" example - compile this https://github.com/U8String/U8String/blob/main/Examples/Interop/CSharp with -p:StripSymbols=false and then disassemble it with Ghidra/IDA Pro/objdump (I had problems with it though). If you look at the first call, you'll notice that it's a very cheap build reference -> add 16B offset -> spill the byref to the stack -> make a direct call -> after call, check if GC poll is needed.

[huoyaoyuan]

See also #101937 (comment)

While it is safe in this case, it is generally recommended to avoid relying on such details and express intentions directly via fixed. Unsafe.AsPointer is one of the most dangerous API and, sadly, too often introduces so called gc holes.

[hez2010]

I think the right way is to introduce an explicit extension for ROS<byte> called Utf8String.

[huoyaoyuan]

That only solves a minor part of the problem. There still needs to be an analyzer to enforce the parameter to be literal.