Safety of type punning for the runtime

[GeeLaw]

Hello, I have a few questions regarding the safety of type punning as far as the runtime is concerned.

// Case 0: Happens-to-be-OK variant array access
string[]    array = new string[2];
Span<object> span = MemoryMarshal.CreateSpan(
  ref Unsafe.As<string, object>(ref array[0]),
  2);
span[0] = "hello,";
span[1] = null;

1. Does the above code excerpt contain undefined behavior (spec-wise) and destabilizing behavior (implementation-wise)? I think not.
2. I believe span[0] = (object)123; would be undefined behavior. Is this correct?
3. Suppose the array is actually object[] and fill with IEnumerable and null, and I reinterpret the array storage as ReadOnlySpan<IEnumerable> and read from it. Is there undefined behavior? I think not.

// Case 1: Reinterpreting wrapper value type as its reference field type
public readonly struct StringRef
{
  public readonly string Referent;
  public StringRef(string referent) { Referent = referent; }
}
if (sizeof(StringRef) != sizeof(string)) Environment.FailFast(null);
StringRef[] array = new StringRef[2];
Span<string> span = MemoryMarshal.CreateSpan(
  ref Unsafe.As<StringRef, string>(ref array[0]),
  2);
span[0] = "hello,";
span[1] = null;

1. Does the code excerpt contain undefined behavior? I believe not (calling Environment.FailFast is desired in case there's a layout mismatch; otherwise, I expect things to work out).
2. Suppose instead the reinterpreted type is object, but only string and null are stored. Is there undefined behavior? I think not.

// Case 2: Reinterpreting wrapper value type as its value field type
public readonly struct IntWrapper
{
  public readonly int Value;
  public IntWrapper(int value) { Value = value; }
}
if (sizeof(IntWrapper) != sizeof(int)) Environment.FailFast(null);
IntWrapper[] array = new IntWrapper[2];
Span<int>     span = MemoryMarshal.CreateSpan(
  ref Unsafe.As<IntWrapper, int>(ref array[0]),
  2);
span[0] = 123;
span[1] = 456;

1. Does the code excerpt contain undefined behavior? I believe not.
2. Suppose instead the reinterpreted type is any unmanaged type (e.g., uint, nint, float, double, StringSplitOptions) and no memory is access beyond the storage of the array. Is there undefined behavior? I believe not and what happens is just the bits are tweaked.

General.

1. Does any of the answer change if I use ref and Unsafe.Add instead of Span?
2. What is the general rule regarding type safety of type punning?

I tried reading the docs of .NET memory model but it only deals with multithreaded access, not type punning.

[EgorBo]

ref Unsafe.As<StringRef, string>

All kinds of struct <-> class reinterpretation-like casts are UB and may lead to hard-to-reproduce crashes/gc holes.

[GeeLaw]

Thanks for the answer. I will avoid that. What about

public struct StringRef
{
  public string Referent; // NO LONGER READONLY
  public StringRef(string referent) { Referent = referent; }
}
if (sizeof(StringRef) != sizeof(string)) Environment.FailFast(null);
StringRef[] array = new StringRef[2];
Span<string> span = MemoryMarshal.CreateSpan(
  ref array[0].Referent,
  2);
span[1] = "world!"; // ?

This way, there is no reinterpretation, but the pointer arithmetic might be dubious. (I removed readonly out of the fear of something similar to controlled mutability, e.g., it might be bad to overwrite individual fields although overwriting the StringRef wholesale is completely fine.)

[GeeLaw]

After reading the spec for a while, I don't think struct <-> class reinterpretation in my case is UB. I'll post a detailed analysis below and would appreciate your opinion.

[tannergooding]

The spec isn't necessarily the source of truth and in many cases is out dated. It must be taken in context with the official augments made since the last published version: https://github.com/dotnet/runtime/blob/main/docs/design/specs/Ecma-335-Augments.md, taken with the guidance of experts from the .NET team, such as @EgorBo and @jkotas, and taken with the nuance of long standing behaviors that exist in the "de facto" implementation that typical programs silently depend on.

[GeeLaw]

I think the ECMA spec is the minimum one can rely on, i.e., any code correct per the ECMA spec, would be correct in Microsoft implementations and future specs (barring "very breaking" changes). Consulting the ECMA is a conservative approach, and consulting the augments is a great addition for making more code correct. Is that right?

[dongle-the-gadget]

As mentioned, Microsoft does not fully follow the ECMA spec, and you should consider Microsoft’s implementation as the source of truth, not ECMA.

[GeeLaw]

For case 2, #76169 shows a way to do it using MemoryMarshal.Cast. The implementation of MemoryMarshal.Cast boils down to Unsafe.As and recomputing the size, but framework code might have special status and might not be replicatable by user code, so the question still stands.

[teo-tsirpanis]

The answer to question 1 is yes. The constructor of Span<T> checks that the passed array is of the exact type, and your snippet sidesteps this check. If you change it to Span<object> span = array it will throw. Converting a string[] to a ReadOnlySpan<object> is supported BTW.

The answer to question 2 is yes as well, because you assign a boxed int to an element of a string[]. If you had an array the assignment would check for it and throw, but this doesn't happen with span assignments because the check (that your code sidesteps) is done at the time of the span's construction.

[GeeLaw]

I understand Span is guarded against array variance, but the reasoning seems a little bit off that circumventing the guard without writing object references of incorrect dynamic type is UB.

string s = null;
// The answer to question 1 (case 1) appears to imply
// that reinterpreting at all is UB?
Unsafe.As<string, object>(ref s);
// If the above is not true, then
// writing object reference of correct dynamic type,
// via incorrectly typed lvalue, is UB?
Unsafe.As<string, object>(ref s) = "world!";

[jilles-sg]

At least in the current versions of the runtime, creating Span<T> from a covariant array is acceptable (provided no object references of incorrect dynamic type are written), since the implementation of Array.Sort<T>(T[]) does it.

However, this still leaves the possibility that a future runtime will add optimizations that assume a Span<T> cannot point to a covariant array and add a slow path for Array.Sort<T>(T[]) on a covariant array.

[GeeLaw]

Re: Definedness of reinterpreting struct whose only field is object, as object reference.

// C# defaults to sequential layout.
public readonly struct Seq { public readonly object O; public Seq(object o) { O = o; } }
[StructLayout(LayoutKind.Explicit)]
public readonly struct Expl
{
  [FieldOffset(0)]
  public readonly object O;
  public Expl(object o) { O = o; }
}

if (sizeof(Seq) != sizeof(object)) Environment.FailFast(null);
if (sizeof(Expl) != sizeof(object)) Environment.FailFast(null);

Seq s = new Seq();
Expl[] e = new Expl[2];
Unsafe.As<Seq, object>(ref s) = new object();  // S
Unsafe.Add(
  ref Unsafe.As<Expl, object>(ref e[0]),
  1) = new object(); // E

Per ECMA-335 edition 6, I.12.1.1 (Native size: native int, native unsigned int, O and &):

... field and stack frame offsets are also not known at compile time... field offsets are not computed early anyway...

This suggests each field of each type as a fixed offset once JIT-ted.

Per II.10.7 (Controlling instance layout):

Fields can be accessed using pointer arithmetic and ldind to load the field indirectly or stind to store
the field indirectly (see Partition III). See §II.22.16 and §II.22.18 for encoding of this information. For
explicit layout, every field shall be assigned an offset.

There are 3 possible interpretations of the boldfaced sentence:

1. Fields can be accessed by pointer arithmetic only if it is a field of an explicit-layout type.
2. ... only if it is a field of a sequential- or explicit-layout type (because the section only talks about types having layout).
3. ... for all types, as long as the correct offset is used.

In interpretations 2 and 3, the line with comment S should be correct, because the correct address of the O field is obtained (the first sizeof check ensures the offset of O is 0).

In all interpretations, the line with comment E should be correct:

• FieldOffset ensures the address of e[0] is the same as that of e[0].O.
• The second sizeof check ensures adding 1 * sizeof(object) bytes is equivalent to adding 1 * sizeof(Expl) bytes. The guarantee that arrays have contiguous storage, together with the guarantee of fixed offset, means the corresponding fields of two instances in the same array, have exactly the same distance as the two instances themselves.

---

OTOH, it's surprising that the spec does not say that a sequential-layout type has its first field at offset 0 (making the sizeof check for S logically necessary). I wouldn't expect any implementation to not do this, but...

[MichalPetryka]

OTOH, it's surprising that the spec does not say that a sequential-layout type has its first field at offset 0 (making the sizeof check for S logically necessary). I wouldn't expect any implementation to not do this, but...

It's cause there's no such guarantee.
Sequential layout is only guaranteed to match the layout of an equivalent C type for interop, which in turn will match the standard ABI used by the platform.
As such, the exact layout used then is provided by the external specification for the ABI, which can (or not) provide such behaviour.
A theoretical ABI could for example require all structs to be padded to 16B with padding inserted at the start and dotnet would need to follow such behaviour if it wanted to run there.

Additionally, it's worth noting that Sequential layout is only documented to affect marshalling behaviour, not managed layouts. As such, it's only guaranteed to affect managed layouts for blittable types (technically it does so for all unmanaged ones) and all types that contain managed fields ignore it and behave like Auto in the managed world (do note that while not guaranteed, Explicit layouts ARE respected for managed types).

[tannergooding]

Additionally, it's worth noting that Sequential layout is only documented to affect marshalling behaviour, not managed layouts.

This isn't correct.

Sequential is explicitly documented to control both the managed and unmanaged layout for blittable types (see https://learn.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.structlayoutattribute#remarks). For non-blittable types it does only control the unmanaged layout and the managed layout is instead treated as Auto. Explicit will likewise control the layout for both managed and unmanaged memory regardless of whether the type is blittable or not; however, if the layout is incompatible with the GC requirements, such as by overlapping managed and unmanaged fields or causing managed fields to be misaligned, then it will result in a TypeLoadException or similar being raised.