Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

I am migrating from Swashbuckle to the new built-in OpenAPI source generator in ASP.NET Core (no Swagger UI). Previously, my endpoint for downloading files was documented with [ProducesResponseType(StatusCodes.Status200OK, Type = typeof(FileContentResult))], which correctly generated a binary file response in the OpenAPI spec as:

"200": {
  "description": "OK",
  "content": {
    "text/plain": {
      "schema": {
        "type": "string",
        "format": "binary"
      }
    },
    "application/json": {
      "schema": {
        "type": "string",
        "format": "binary"
      }
    }
  }
}

After migrating to the new OpenAPI document generation, the same endpoint generates this in the spec:

"200": {
  "description": "OK",
  "content": {
    "text/plain": {
      "schema": {
        "$ref": "#/components/schemas/FileContentResult"
      }
    },
    "application/json": {
      "schema": {
        "$ref": "#/components/schemas/FileContentResult"
      }
    }
  }
}

This causes issues because it does not mark the response as binary data but as a serialized object. Also, the [Produces("application/octet-stream")] attribute does not influence the schema properly — it only produces an empty schema without type: string and format: binary.

I have not found a way to explicitly specify in the new OpenAPI generator that the response content is binary. I want to do this preferably with attributes or minimal manual schema mapping, but there appears to be no direct support for this.

Expected Behavior

The OpenAPI document generator should support explicitly marking responses as binary file content, ideally by honoring [Produces("application/octet-stream")] with a schema of:

  "schema": {
    "type": "string",
    "format": "binary"
  }

The generated OpenAPI spec for file downloads should not reference the FileContentResult schema, but instead describe the response as binary data.

Ideally, this could be controlled by attributes or minimal manual mapping without complex workarounds.

Steps To Reproduce

1. Create an ASP.NET Core Web API project.
2. Define a controller endpoint returning a FileContentResult or similar.
3. Add the attribute [Produces("application/octet-stream")].
4. Add [ProducesResponseType(StatusCodes.Status200OK, Type = typeof(FileContentResult))].
5. Use the new OpenAPI document generation (e.g., via Microsoft.AspNetCore.OpenApi package, not Swashbuckle).
6. Generate the OpenAPI JSON spec.
7. Observe that the 200 response references the FileContentResult schema rather than a binary string schema.

Exceptions (if any)

No exceptions thrown, but the generated OpenAPI document is semantically incorrect and causes serialization or client errors when interpreting the API response as a file download.

.NET Version

9.0.102

Anything else?

No response