Skip to content

Commit 7a45929

Browse files
victordibiamarkwallace-microsoftReubenBond
authored
Python: .Net: Dotnet devui compatibility fixes (microsoft#2026)
* DevUI: Add OpenAI Responses API proxy support with enhanced UI features This commit adds support for proxying requests to OpenAI's Responses API, allowing DevUI to route conversations to OpenAI models when configured to enable testing. Backend changes: - Add OpenAI proxy executor with conversation routing logic - Enhance event mapper to support OpenAI Responses API format - Extend server endpoints to handle OpenAI proxy mode - Update models with OpenAI-specific response types - Remove emojis from logging and CLI output for cleaner text Frontend changes: - Add settings modal with OpenAI proxy configuration UI - Enhance agent and workflow views with improved state management - Add new UI components (separator, switch) for settings - Update debug panel with better event filtering - Improve message renderers for OpenAI content types - Update types and API client for OpenAI integration * update ui, settings modal and workflow input form, add register cleanup hooks. * add workflow HIL support, user mode, other fixes * feat(devui): add human-in-the-loop (HIL) support with dynamic response schemas Implement HIL workflow support allowing workflows to pause for user input with dynamically generated JSON schemas based on response handler type hints. Key Features: - Automatic response schema extraction from @response_handler decorators - Dynamic form generation in UI based on Pydantic/dataclass response types - Checkpoint-based conversation storage for HIL requests/responses - Resume workflow execution after user provides HIL response Backend Changes: - Add extract_response_type_from_executor() to introspect response handlers - Enrich RequestInfoEvent with response_schema via _enrich_request_info_event_with_response_schema() - Map RequestInfoEvent to response.input.requested OpenAI event format - Store HIL responses in conversation history and restore checkpoints Frontend Changes: - Add HILInputModal component with SchemaFormRenderer for dynamic forms - Support Pydantic BaseModel and dataclass response types - Render enum fields as dropdowns, strings as text/textarea, numbers, booleans, arrays, objects - Display original request context alongside response form Testing: - Add tests for checkpoint storage (test_checkpoints.py) - Add schema generation tests for all input types (test_schema_generation.py) - Validate end-to-end HIL flow with spam workflow sample This enables workflows to seamlessly pause execution and request structured user input with type-safe, validated forms generated automatically from response type annotations. * improve HIL support, improve workflow execution view * ui updates * ui updates * improve HIL for workflows, add auth and view modes * update workflow * security improvements , ui fixes * fix mypy error * update loading spinner in ui * DevUI: Serialize workflow input as string to maintain conformance with OpenAI Responses format * Phase 1: Add /meta endpoint and fix workflow event naming for .NET DevUI compatibility * additional fixes for .NET DevUI workflow visualization item ID tracking **Problem:** .NET DevUI was generating different item IDs for ExecutorInvokedEvent and ExecutorCompletedEvent, causing only the first executor to highlight in the workflow graph. Long executor names and error messages also broke UI layout. **Changes:** - Add ExecutorActionItemResource to match Python DevUI implementation - Track item IDs per executor using dictionary in AgentRunResponseUpdateExtensions - Reuse same item ID across invoked/completed/failed events for proper pairing - Add truncateText() utility to workflow-utils.ts - Truncate executor names to 35 chars in execution timeline - Truncate error messages to 150 chars in workflow graph nodes ** Details:** - ExecutorActionItemResource registered with JSON source generation context - Dictionary cleaned up after executor completion/failure to prevent memory leaks - Frontend item tracking by unique item.id supports multiple executor runs - All changes follow existing codebase patterns and conventions Tested with review-workflow showing correct executor highlighting and state transitions for sequential and concurrent executors. * format fixes, remove cors tests * remove unecessary attributes --------- Co-authored-by: Mark Wallace <127216156+markwallace-microsoft@users.noreply.github.com> Co-authored-by: Reuben Bond <reuben.bond@gmail.com>
1 parent c0c12df commit 7a45929

File tree

22 files changed

+483
-159
lines changed

22 files changed

+483
-159
lines changed

‎dotnet/src/Microsoft.Agents.AI.DevUI/DevUIExtensions.cs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ public static IEndpointConventionBuilder MapDevUI(
3232
{
3333
var group = endpoints.MapGroup("");
3434
group.MapDevUI(pattern: "/devui");
35+
group.MapMeta();
3536
group.MapEntities();
3637
return group;
3738
}

‎dotnet/src/Microsoft.Agents.AI.DevUI/Entities/EntitiesJsonContext.cs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,10 +15,12 @@ namespace Microsoft.Agents.AI.DevUI.Entities;
1515
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull)]
1616
[JsonSerializable(typeof(EntityInfo))]
1717
[JsonSerializable(typeof(DiscoveryResponse))]
18+
[JsonSerializable(typeof(MetaResponse))]
1819
[JsonSerializable(typeof(EnvVarRequirement))]
1920
[JsonSerializable(typeof(List<EntityInfo>))]
2021
[JsonSerializable(typeof(List<JsonElement>))]
2122
[JsonSerializable(typeof(Dictionary<string, JsonElement>))]
23+
[JsonSerializable(typeof(Dictionary<string, bool>))]
2224
[JsonSerializable(typeof(JsonElement))]
2325
[ExcludeFromCodeCoverage]
2426
internal sealed partial class EntitiesJsonContext : JsonSerializerContext;

‎dotnet/src/Microsoft.Agents.AI.DevUI/Entities/MetaResponse.cs‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
// Copyright (c) Microsoft. All rights reserved.
2+
3+
using System.Text.Json.Serialization;
4+
5+
namespace Microsoft.Agents.AI.DevUI.Entities;
6+
7+
/// <summary>
8+
/// Server metadata response for the /meta endpoint.
9+
/// Provides information about the DevUI server configuration, capabilities, and requirements.
10+
/// </summary>
11+
/// <remarks>
12+
/// This response is used by the frontend to:
13+
/// - Determine the UI mode (developer vs user interface)
14+
/// - Check server capabilities (tracing, OpenAI proxy support)
15+
/// - Verify authentication requirements
16+
/// - Display framework and version information
17+
/// </remarks>
18+
internal sealed record MetaResponse
19+
{
20+
/// <summary>
21+
/// Gets the UI interface mode.
22+
/// "developer" shows debug tools and advanced features, "user" shows a simplified interface.
23+
/// </summary>
24+
[JsonPropertyName("ui_mode")]
25+
public string UiMode { get; init; } = "developer";
26+
27+
/// <summary>
28+
/// Gets the DevUI version string.
29+
/// </summary>
30+
[JsonPropertyName("version")]
31+
public string Version { get; init; } = "0.1.0";
32+
33+
/// <summary>
34+
/// Gets the backend framework identifier.
35+
/// Always "agent_framework" for Agent Framework implementations.
36+
/// </summary>
37+
[JsonPropertyName("framework")]
38+
public string Framework { get; init; } = "agent_framework";
39+
40+
/// <summary>
41+
/// Gets the backend runtime/language.
42+
/// "dotnet" for .NET implementations, "python" for Python implementations.
43+
/// Used by frontend for deployment guides and feature availability.
44+
/// </summary>
45+
[JsonPropertyName("runtime")]
46+
public string Runtime { get; init; } = "dotnet";
47+
48+
/// <summary>
49+
/// Gets the server capabilities dictionary.
50+
/// Key-value pairs indicating which optional features are enabled.
51+
/// </summary>
52+
/// <remarks>
53+
/// Standard capability keys:
54+
/// - "tracing": Whether trace events are emitted for debugging
55+
/// - "openai_proxy": Whether the server can proxy requests to OpenAI
56+
/// </remarks>
57+
[JsonPropertyName("capabilities")]
58+
public Dictionary<string, bool> Capabilities { get; init; } = new();
59+
60+
/// <summary>
61+
/// Gets a value indicating whether Bearer token authentication is required for API access.
62+
/// When true, clients must include "Authorization: Bearer {token}" header in requests.
63+
/// </summary>
64+
[JsonPropertyName("auth_required")]
65+
public bool AuthRequired { get; init; }
66+
}

‎dotnet/src/Microsoft.Agents.AI.DevUI/MetaApiExtensions.cs‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
// Copyright (c) Microsoft. All rights reserved.
2+
3+
using Microsoft.Agents.AI.DevUI.Entities;
4+
5+
namespace Microsoft.Agents.AI.DevUI;
6+
7+
/// <summary>
8+
/// Provides extension methods for mapping the server metadata endpoint to an <see cref="IEndpointRouteBuilder"/>.
9+
/// </summary>
10+
internal static class MetaApiExtensions
11+
{
12+
/// <summary>
13+
/// Maps the HTTP API endpoint for retrieving server metadata.
14+
/// </summary>
15+
/// <param name="endpoints">The <see cref="IEndpointRouteBuilder"/> to add the route to.</param>
16+
/// <returns>The <see cref="IEndpointConventionBuilder"/> for method chaining.</returns>
17+
/// <remarks>
18+
/// This extension method registers the following endpoint:
19+
/// <list type="bullet">
20+
/// <item><description>GET /meta - Retrieve server metadata including UI mode, version, capabilities, and auth requirements</description></item>
21+
/// </list>
22+
/// The endpoint is compatible with the Python DevUI frontend and provides essential
23+
/// configuration information needed for proper frontend initialization.
24+
/// </remarks>
25+
public static IEndpointConventionBuilder MapMeta(this IEndpointRouteBuilder endpoints)
26+
{
27+
return endpoints.MapGet("/meta", GetMeta)
28+
.WithName("GetMeta")
29+
.WithSummary("Get server metadata and configuration")
30+
.WithDescription("Returns server metadata including UI mode, version, framework identifier, capabilities, and authentication requirements. Used by the frontend for initialization and feature detection.")
31+
.Produces<MetaResponse>(StatusCodes.Status200OK, contentType: "application/json");
32+
}
33+
34+
private static IResult GetMeta()
35+
{
36+
// TODO: Consider making these configurable via IOptions<DevUIOptions>
37+
// For now, using sensible defaults that match Python DevUI behavior
38+
39+
var meta = new MetaResponse
40+
{
41+
UiMode = "developer", // Could be made configurable to support "user" mode
42+
Version = "0.1.0", // TODO: Extract from assembly version attribute
43+
Framework = "agent_framework",
44+
Runtime = "dotnet", // .NET runtime for deployment guides
45+
Capabilities = new Dictionary<string, bool>
46+
{
47+
// Tracing capability - will be enabled when trace event support is added
48+
["tracing"] = false,
49+
50+
// OpenAI proxy capability - not currently supported in .NET DevUI
51+
["openai_proxy"] = false,
52+
53+
// Deployment capability - not currently supported in .NET DevUI
54+
["deployment"] = false
55+
},
56+
AuthRequired = false // Could be made configurable based on authentication middleware
57+
};
58+
59+
return Results.Json(meta, EntitiesJsonContext.Default.MetaResponse);
60+
}
61+
}

‎dotnet/src/Microsoft.Agents.AI.Hosting.OpenAI/OpenAIHostingJsonUtilities.cs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,7 @@ private static JsonSerializerOptions CreateDefaultOptions()
109109
[JsonSerializable(typeof(MCPApprovalRequestItemResource))]
110110
[JsonSerializable(typeof(MCPApprovalResponseItemResource))]
111111
[JsonSerializable(typeof(MCPCallItemResource))]
112+
[JsonSerializable(typeof(ExecutorActionItemResource))]
112113
[JsonSerializable(typeof(List<ItemResource>))]
113114
// ItemParam types
114115
[JsonSerializable(typeof(ItemParam))]

‎dotnet/src/Microsoft.Agents.AI.Hosting.OpenAI/Responses/AgentRunResponseUpdateExtensions.cs‎

Lines changed: 89 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,9 @@ public static async IAsyncEnumerable<StreamingResponseEvent> ToStreamingResponse
4545
var updateEnumerator = updates.GetAsyncEnumerator(cancellationToken);
4646
await using var _ = updateEnumerator.ConfigureAwait(false);
4747

48+
// Track active item IDs by executor ID to pair invoked/completed/failed events
49+
Dictionary<string, string> executorItemIds = [];
50+
4851
AgentRunResponseUpdate? previousUpdate = null;
4952
StreamingEventGenerator? generator = null;
5053
while (await updateEnumerator.MoveNextAsync().ConfigureAwait(false))
@@ -55,7 +58,92 @@ public static async IAsyncEnumerable<StreamingResponseEvent> ToStreamingResponse
5558
// Special-case for agent framework workflow events.
5659
if (update.RawRepresentation is WorkflowEvent workflowEvent)
5760
{
58-
yield return CreateWorkflowEventResponse(workflowEvent, seq.Increment(), outputIndex);
61+
// Convert executor events to standard OpenAI output_item events
62+
if (workflowEvent is ExecutorInvokedEvent invokedEvent)
63+
{
64+
var itemId = IdGenerator.NewId(prefix: "item");
65+
// Store the item ID for this executor so we can reuse it for completion/failure
66+
executorItemIds[invokedEvent.ExecutorId] = itemId;
67+
68+
var item = new ExecutorActionItemResource
69+
{
70+
Id = itemId,
71+
ExecutorId = invokedEvent.ExecutorId,
72+
Status = "in_progress",
73+
CreatedAt = DateTimeOffset.UtcNow.ToUnixTimeSeconds()
74+
};
75+
76+
yield return new StreamingOutputItemAdded
77+
{
78+
SequenceNumber = seq.Increment(),
79+
OutputIndex = outputIndex,
80+
Item = item
81+
};
82+
}
83+
else if (workflowEvent is ExecutorCompletedEvent completedEvent)
84+
{
85+
// Reuse the item ID from the invoked event, or generate a new one if not found
86+
var itemId = executorItemIds.TryGetValue(completedEvent.ExecutorId, out var existingId)
87+
? existingId
88+
: IdGenerator.NewId(prefix: "item");
89+
90+
// Remove from tracking as this executor run is now complete
91+
executorItemIds.Remove(completedEvent.ExecutorId);
92+
JsonElement? resultData = null;
93+
if (completedEvent.Data != null && JsonSerializer.IsReflectionEnabledByDefault)
94+
{
95+
resultData = JsonSerializer.SerializeToElement(
96+
completedEvent.Data,
97+
OpenAIHostingJsonUtilities.DefaultOptions.GetTypeInfo(typeof(object)));
98+
}
99+
100+
var item = new ExecutorActionItemResource
101+
{
102+
Id = itemId,
103+
ExecutorId = completedEvent.ExecutorId,
104+
Status = "completed",
105+
Result = resultData,
106+
CreatedAt = DateTimeOffset.UtcNow.ToUnixTimeSeconds()
107+
};
108+
109+
yield return new StreamingOutputItemDone
110+
{
111+
SequenceNumber = seq.Increment(),
112+
OutputIndex = outputIndex,
113+
Item = item
114+
};
115+
}
116+
else if (workflowEvent is ExecutorFailedEvent failedEvent)
117+
{
118+
// Reuse the item ID from the invoked event, or generate a new one if not found
119+
var itemId = executorItemIds.TryGetValue(failedEvent.ExecutorId, out var existingId)
120+
? existingId
121+
: IdGenerator.NewId(prefix: "item");
122+
123+
// Remove from tracking as this executor run has now failed
124+
executorItemIds.Remove(failedEvent.ExecutorId);
125+
126+
var item = new ExecutorActionItemResource
127+
{
128+
Id = itemId,
129+
ExecutorId = failedEvent.ExecutorId,
130+
Status = "failed",
131+
Error = failedEvent.Data?.ToString(),
132+
CreatedAt = DateTimeOffset.UtcNow.ToUnixTimeSeconds()
133+
};
134+
135+
yield return new StreamingOutputItemDone
136+
{
137+
SequenceNumber = seq.Increment(),
138+
OutputIndex = outputIndex,
139+
Item = item
140+
};
141+
}
142+
else
143+
{
144+
// For other workflow events (not executor-specific), keep the old format as fallback
145+
yield return CreateWorkflowEventResponse(workflowEvent, seq.Increment(), outputIndex);
146+
}
59147
continue;
60148
}
61149

‎dotnet/src/Microsoft.Agents.AI.Hosting.OpenAI/Responses/Converters/ItemResourceConverter.cs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ internal sealed class ItemResourceConverter : JsonConverter<ItemResource>
4545
MCPApprovalRequestItemResource.ItemType => doc.Deserialize(OpenAIHostingJsonContext.Default.MCPApprovalRequestItemResource),
4646
MCPApprovalResponseItemResource.ItemType => doc.Deserialize(OpenAIHostingJsonContext.Default.MCPApprovalResponseItemResource),
4747
MCPCallItemResource.ItemType => doc.Deserialize(OpenAIHostingJsonContext.Default.MCPCallItemResource),
48+
ExecutorActionItemResource.ItemType => doc.Deserialize(OpenAIHostingJsonContext.Default.ExecutorActionItemResource),
4849
_ => null
4950
};
5051
}
@@ -106,6 +107,9 @@ public override void Write(Utf8JsonWriter writer, ItemResource value, JsonSerial
106107
case MCPCallItemResource mcpCall:
107108
JsonSerializer.Serialize(writer, mcpCall, OpenAIHostingJsonContext.Default.MCPCallItemResource);
108109
break;
110+
case ExecutorActionItemResource executorAction:
111+
JsonSerializer.Serialize(writer, executorAction, OpenAIHostingJsonContext.Default.ExecutorActionItemResource);
112+
break;
109113
default:
110114
throw new JsonException($"Unknown item type: {value.GetType().Name}");
111115
}

‎dotnet/src/Microsoft.Agents.AI.Hosting.OpenAI/Responses/Models/ItemResource.cs‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -888,3 +888,47 @@ internal sealed class MCPCallItemResource : ItemResource
888888
[JsonPropertyName("error")]
889889
public string? Error { get; init; }
890890
}
891+
892+
/// <summary>
893+
/// An executor action item resource for workflow execution visualization.
894+
/// </summary>
895+
internal sealed class ExecutorActionItemResource : ItemResource
896+
{
897+
/// <summary>
898+
/// The constant item type identifier for executor action items.
899+
/// </summary>
900+
public const string ItemType = "executor_action";
901+
902+
/// <inheritdoc/>
903+
public override string Type => ItemType;
904+
905+
/// <summary>
906+
/// The executor identifier.
907+
/// </summary>
908+
[JsonPropertyName("executor_id")]
909+
public required string ExecutorId { get; init; }
910+
911+
/// <summary>
912+
/// The execution status: "in_progress", "completed", "failed", or "cancelled".
913+
/// </summary>
914+
[JsonPropertyName("status")]
915+
public required string Status { get; init; }
916+
917+
/// <summary>
918+
/// The executor result data (for completed status).
919+
/// </summary>
920+
[JsonPropertyName("result")]
921+
public JsonElement? Result { get; init; }
922+
923+
/// <summary>
924+
/// The error message (for failed status).
925+
/// </summary>
926+
[JsonPropertyName("error")]
927+
public string? Error { get; init; }
928+
929+
/// <summary>
930+
/// The creation timestamp.
931+
/// </summary>
932+
[JsonPropertyName("created_at")]
933+
public long CreatedAt { get; init; }
934+
}

‎dotnet/src/Microsoft.Agents.AI.Hosting.OpenAI/Responses/Models/StreamingResponseEvent.cs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -565,7 +565,7 @@ internal sealed class StreamingWorkflowEventComplete : StreamingResponseEvent
565565
/// <summary>
566566
/// The constant event type identifier for workflow event events.
567567
/// </summary>
568-
public const string EventType = "response.workflow_event.complete";
568+
public const string EventType = "response.workflow_event.completed";
569569

570570
/// <inheritdoc/>
571571
[JsonIgnore]

‎python/packages/devui/agent_framework_devui/_server.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -397,6 +397,7 @@ async def get_meta() -> MetaResponse:
397397
ui_mode=self.mode, # type: ignore[arg-type]
398398
version=__version__,
399399
framework="agent_framework",
400+
runtime="python", # Python DevUI backend
400401
capabilities={
401402
"tracing": os.getenv("ENABLE_OTEL") == "true",
402403
"openai_proxy": openai_executor.is_configured,

0 commit comments

Comments
 (0)