[FEATURE REQ] Missing data from `GeneratedImage` type

[ericstj]

Describe the bug

The OpenAI API describes the following properties on the Image response https://platform.openai.com/docs/api-reference/images/object

• background
• created
• data
• output_format
• quality
• size
• usage

The only property exposed by the GeneratedImage type is data. output_format in particular is necessary for determining mime-type, but it seems like all the properties should be exposed.

Steps to reproduce

Use ImageClient to generate an image.

Code snippets

OS

Any

.NET version

.NET 8

Library version

2.2.0

[ericstj]

#259 would help here since the data is in that dictionary, but it's not exposed.

[jsquire]

Hi @ericstj. Thanks for reaching out and for your request. The Images feature in the REST API spec that the current release is based on lagged a bit from the most recent service version. The new properties will be included in a future release as the spec is updated. The work for exposing a global additional properties collection on models is also underway.

In the meantime, you can make use of the newer properties by parsing the response JSON directly and bypassing the model. One thing that I'll note is that many of the fields that you're interested in are conditional, based on the model that you're using to generate. The output_format, for example, is only populated when using gpt-image-1 to generate, which is true of most of the missing properties.

An example of grabbing the full set or properties using custom output models would look something like:

Creating the image

var client = new ImageClient("dall-e-3", "<< YOUR API KEY >>");

var response = await client.GenerateImageAsync(
    "Krzysztof Cwalina looking at my code with disapproval.",
    new ImageGenerationOptions
    {
        Size = GeneratedImageSize.W1024xH1024
    });    
      
var generatedImage = JsonSerializer.Deserialize<UpdatedImage>(
    response.GetRawResponse().Content,
    new JsonSerializerOptions { PropertyNameCaseInsensitive = true });

Console.WriteLine($"Image Uri: {generatedImage.Data.Single().Uri}");
Console.WriteLine();
Console.WriteLine(response.GetRawResponse().Content);

Custom output models

public class UpdatedImage
{
    [JsonPropertyName("created")]
    public long Created { get; set; }

    [JsonPropertyName("data")]
    public List<ImageData> Data { get; set; }

    [JsonPropertyName("background")]
    public string Background { get; set; }

    [JsonPropertyName("output_format")]
    public string OutputFormat { get; set; }

    [JsonPropertyName("size")]
    public string Size { get; set; }

    [JsonPropertyName("quality")]
    public string Quality { get; set; }

    [JsonPropertyName("usage")]
    public Usage Usage { get; set; }
}

public class ImageData
{
    [JsonPropertyName("b64_json")]
    public string B64Json { get; set; }

    [JsonPropertyName("revised_prompt")]
    public string RevisedPrompt { get; set; }

    [JsonPropertyName("url")]
    public Uri Uri { get; set; }
}

public class Usage
{
    [JsonPropertyName("total_tokens")]
    public int TotalTokens { get; set; }

    [JsonPropertyName("input_tokens")]
    public int InputTokens { get; set; }

    [JsonPropertyName("output_tokens")]
    public int OutputTokens { get; set; }

    [JsonPropertyName("input_tokens_details")]
    public InputTokensDetails InputTokensDetails { get; set; }
}

public class InputTokensDetails
{
    [JsonPropertyName("text_tokens")]
    public int TextTokens { get; set; }

    [JsonPropertyName("image_tokens")]
    public int ImageTokens { get; set; }
}

[ericstj]

I think we would be doubling deserialization of very large payloads by doing this. Seems this API already takes care to go from JSON directly to bytes --

openai-dotnet/src/Generated/Models/Images/GeneratedImage.Serialization.cs

Line 105 in 5b5f6f6

imageBytes = BinaryData.FromBytes(prop.Value.GetBytesFromBase64("D"));

.

The uri is not something the new model returns - only the old dall-e models - so I suspect most folks will have to use the data and not uri going forward.

Is there a way to leverage the well-tuned client API for most of the serialization, and just deserialize a couple additional properties without generating a bunch of additional objects/strings?

[KrzysztofCwalina]

@ericstj, as a workaround, you can call the protocol version of the GenerateImageAsync, and do deserialization yourself. See https://learn.microsoft.com/en-us/dotnet/azure/sdk/protocol-convenience-methods?tabs=convenience-methods%2Csystem-clientmodel

Longer term, we are thinking about adding public "additional properties" to models to access such properties.

[jsquire]

@ericstj: That's a fair observation. For the scenario where you're looking to extract just the image and avoid the serialization cost, you'd want to handle the deserialization manually and extract just the fields that you're interested in. You'll likely also want to turn off response buffering. In this case, you mentioned wanting to grab just the image and format, which may look something like:

Create and do something with the image.

var client = new ImageClient("dall-e-3", "<<YOUR API KEY >>");

// Invoke the request with response buffering disabled.
var response = await client.GenerateImagesAsync(
    BinaryContent.Create(BinaryData.FromObjectAsJson(new
    {
        prompt = "Krzysztof Cwalina looking at my code with disapproval.",
        model = "dall-e-3",
        n = 1,
        response_format = "b64_json"
    })), 
    new RequestOptions() { BufferResponse = false });

// Perform a custom deserialization on the content stream which extracts 
// only the "output_format" and a stream version of the base64 image.
(string? outputFormat, Stream imageStream) = ExtractImageFromStream(response.GetRawResponse().ContentStream!);

try
{
    // For the example, write the image out to the current user's desktop.
    var fileName = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop), $"generated-image.{format ?? "png"}");

    using var fileStream = new FileStream(fileName, FileMode.Create, FileAccess.Write);
    imageStream.CopyTo(fileStream);
}
finally
{
    // Ownership of the image stream was transferred, so it should be 
    // cleaned up here.
    imageStream.Dispose();    
}

Parse the response to extract the image.

public static (string? OutputFormat, Stream ImageStream) ExtractImageFromStream(Stream jsonStream)
{
    const int BufferSize = 4096;
    var buffer = new byte[BufferSize];
    var pooledBuffer = new ArrayBufferWriter<byte>();

    JsonReaderState state = new JsonReaderState();
    string? outputFormat = null;
    Stream? imageStream = null;

    string? currentProperty = null;
    bool inDataArray = false;
    bool inFirstObject = false;
    bool firstObjectCompleted = false;

    while (!firstObjectCompleted)
    {
        int bytesRead = jsonStream.Read(buffer, 0, buffer.Length);
        if (bytesRead == 0)
        {
            break;
        }

        pooledBuffer.Write(buffer.AsSpan(0, bytesRead));

        var span = pooledBuffer.WrittenSpan;
        var reader = new Utf8JsonReader(span, isFinalBlock: false, state);

        int consumed = 0;

        while (reader.Read())
        {
            consumed = (int)reader.BytesConsumed;

            switch (reader.TokenType)
            {
                case JsonTokenType.PropertyName:
                    if (!inDataArray && reader.ValueTextEquals("data"))
                    {
                        if (!reader.Read() || reader.TokenType != JsonTokenType.StartArray)
                        {
                            throw new JsonException("Expected start of 'data' array.");
                        }

                        inDataArray = true;
                    }
                    else if (inFirstObject)
                    {
                        currentProperty = reader.GetString();
                    }
                    break;

                case JsonTokenType.StartObject:
                    if (inDataArray && !inFirstObject)
                    {
                        inFirstObject = true;
                    }
                    break;

                case JsonTokenType.EndObject:
                    if (inFirstObject)
                    {
                        firstObjectCompleted = true;
                    }
                    break;

                case JsonTokenType.String:
                    if (inFirstObject && currentProperty != null)
                    {
                        if (currentProperty == "b64_json")
                        {
                            string base64 = reader.GetString() ?? throw new JsonException("'b64_json' is null.");
                            byte[] decoded = Convert.FromBase64String(base64);
                            imageStream = new MemoryStream(decoded, writable: false);
                        }
                        else if (currentProperty == "output_format")
                        {
                            outputFormat = reader.GetString();
                        }

                        currentProperty = null;
                    }
                    break;
            }

            state = reader.CurrentState;
        }

        // Discard consumed bytes and keep unprocessed data
        if (consumed > 0)
        {
            int remaining = pooledBuffer.WrittenCount - consumed;
            var newBuffer = new ArrayBufferWriter<byte>(remaining + BufferSize);
            span.Slice(consumed, remaining).CopyTo(newBuffer.GetSpan(remaining));
            newBuffer.Advance(remaining);
            pooledBuffer = newBuffer;
        }
    }

    if (imageStream == null)
    {
        throw new InvalidOperationException("Required field 'b64_json' was not found.");
    }

    return (outputFormat, imageStream);
}