[Proposal]: Expand NotNullWhen attribute to support Enum types as Generic parameter

[YIShikunov]

NotNullWhen attribute

 * [x]  Proposed

 * [ ]  Prototype: Not Started

 * [ ]  Implementation: Not Started

 * [ ]  Specification: Not Started

Summary

A nullable attribute specifies that a method return value is not null if Enum return value equals a given value

Motivation

One of the possible uses can be implementation of discriminated unions.

public enum OneOfDefined
{
    Item1 = 1,
    Item2 = 2,
    Item3 = 3
}

public sealed class OneOf<TItem1, TItem2, TItem3>
    where TItem1 : notnull
    where TItem2 : notnull
    where TItem3 : notnull
{
   public OneOfDefined TryGetItem([NotNullWhen(OneOfDefined.Item1)] out TItem1 item1,  
                                 [NotNullWhen(OneOfDefined.Item2)] out TItem2 item2,  
                                 [NotNullWhen(OneOfDefined.Item3)] out TItem3 item3)  
}

Design

A new generic attribute will be added and supported by the Code Analysis

namespace System.Diagnostics.CodeAnalysis
{
    [AttributeUsage(AttributeTargets.Parameter, AllowMultiple = true)]
    internal sealed class NotNullWhenAttribute<TEnum> : Attribute
    where TEnum: enum
    {
        public NotNullWhenAttribute(TEnum returnValue);

        public TEnum ReturnValue { get; }
    }
}

Originally posted by @YIShikunov in #5600 (comment)

[CyrusNajmabadi]

It would be good to point at some real world apis, including those shipped either in the BCL or in very popular APIs (like Json.net) that would benefit from this.

[dahall]

Native Win32 API and COM interface methods regularly use a pattern of returning a status code to indicate success or failure rather than exceptions. Basing the NotNullWhenAttribute on an inheritable attribute class is the recommendation I would propose rather than assuming that the return value is an enumerated type or a boolean value.

For example, let's look at IBackgroundCopyManager::CreateJob. The C# imported definition for this method is:

[PreserveSig]
int CreateJob([In, MarshalAs(UnmanagedType.LPWStr)] string DisplayName, [In] int Type, out Guid pJobId,
  out IBackgroundCopyJob ppJob);

While not documented, this method returns NULL for the ppJob parameter on failure (HRESULT != S_OK) and ideally, for null checking analysis, should be marked as nullable, but only under failure condition. Being able to extend the base class for NotNullWhenAttribute could provide accurate compiler warnings, as follows:

namespace System.Diagnostics.CodeAnalysis
{
  [AttributeUsage(AttributeTargets.Parameter)]
  public abstract class NotNullWhenConditionAttribute<TRet> : Attribute
  {
    /// <summary>Validates the return value condition to determine if targeted parameter should be non-null.</summary>
    /// <param name="returnValue">The return value to validate.</param>
    /// <returns><see langword="true"/> when the return value has met the condition; otherwise <see langword="false"/></returns>
    protected abstract bool ValidateReturnValue(TRet returnValue);
  }

  [AttributeUsage(AttributeTargets.Parameter)]
  public class NotNullWhenEqualsAttribute<T> : NotNullWhenConditionAttribute<T> where T : IEquatable<T>
  {
    /// <summary>Initializes the attribute with the specified return value condition.</summary>
    /// <param name="returnValue">
    /// The return value condition. If the method returns this value, the associated parameter will not be null.
    /// </param>
    public NotNullWhenEqualsAttribute(T returnValue) => ReturnValue = returnValue;

    /// <summary>Gets the return value condition.</summary>
    public T ReturnValue { get; }

    /// <summary>Validates the return value condition to determine if targeted parameter should be non-null.</summary>
    protected override bool ValidateReturnValue(T returnValue) => ReturnValue.Equals(returnValue);
  }

  [AttributeUsage(AttributeTargets.Parameter)]
  public sealed class NotNullWhenAttribute : NotNullWhenEqualsAttribute<bool>
  {
    /// <summary>Initializes the attribute with the specified return value condition.</summary>
    /// <param name="returnValue">
    /// The return value condition. If the method returns this value, the associated parameter will not be null.
    /// </param>
    public NotNullWhenAttribute(bool returnValue) : base(returnValue) { }
  }
}

Then the P/Invoke method could be expressed at:

[PreserveSig]
int CreateJob([In, MarshalAs(UnmanagedType.LPWStr)] string DisplayName, [In] int Type, out Guid pJobId,
  [NotNullWhenEquals(0)] out IBackgroundCopyJob? ppJob);