feat: Add MCP server to Game Studio & Stride Engine for AI agent integration#3094
Draft
madsiberian wants to merge 41 commits intostride3d:masterfrom
Draft
feat: Add MCP server to Game Studio & Stride Engine for AI agent integration#3094madsiberian wants to merge 41 commits intostride3d:masterfrom
madsiberian wants to merge 41 commits intostride3d:masterfrom
Conversation
Implement a minimal viable MCP (Model Context Protocol) server as an editor plugin, enabling AI assistants to inspect the editor state via SSE/HTTP transport on localhost:5271. New project Stride.GameStudio.Mcp with: - McpEditorPlugin: plugin lifecycle (init/dispose) with McpServerService - McpServerService: embedded Kestrel/ASP.NET Core hosting SSE transport - DispatcherBridge: thread marshalling from HTTP threads to WPF dispatcher - GetEditorStatusTool: first read-only tool returning project info, asset count, and scene list Tested end-to-end: SSE handshake, MCP initialize, and tools/call all return correct data from a live Game Studio session. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ne 1.2) Implement the three read-only tools for asset and scene inspection: - query_assets: enumerate/filter assets by name, type, or URL folder prefix with pagination support - get_scene_tree: return full entity hierarchy for a scene with component type lists at each node - get_entity: return detailed entity info including all component properties serialized via [DataMember] reflection, with proper handling of Stride math types (Vector3, Quaternion, Color, etc.) All tools dispatch to the WPF UI thread via DispatcherBridge and return structured JSON. Invalid IDs return descriptive error messages. Tested end-to-end against FirstPersonShooter sample: 735 assets, 304 entities in MainScene, full component property serialization including physics, camera, and transform data. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…property to 'currentProject' for consistency
Adds integration tests for all 4 MCP tools (get_editor_status, query_assets, get_scene_tree, get_entity) plus tool listing. Tests connect to a running Game Studio instance via the MCP client SDK and are skipped unless STRIDE_MCP_INTEGRATION_TESTS=true is set. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Covers AI agent configuration for Claude Code, Claude Desktop, Continue, Junie, and Cursor, plus integration test instructions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…n tests - bootstrap.ps1: automates GameStudio build (VS MSBuild), pruned DLL workaround, NuGet pack, and test project build - GameStudioFixture: xUnit collection fixture that launches GameStudio, polls MCP endpoint for readiness, and kills the process after tests - Move integration test docs from MCP plugin README to test project README - Tests still disabled by default (STRIDE_MCP_INTEGRATION_TESTS=true) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…e 2.1) Navigation & Selection tools for the embedded MCP server: - open_scene: Opens a scene asset in the editor via IAssetEditorsManager - select_entity: Selects entities in the scene hierarchy (supports multi-select and add-to-selection) - focus_entity: Centers the viewport camera on an entity via FocusOnEntityCommand Also adds a non-generic InvokeTaskOnUIThread overload to DispatcherBridge for async void dispatcher calls, and 7 new integration tests covering all navigation tools (15 total tests, all passing). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…m MCP tools (Milestone 3.1 + 3.2) Add scene modification tools with full undo/redo support: - create_entity: Creates entities with optional parent, using AssetHierarchyPropertyGraph - delete_entity: Deletes entities and children via DeleteParts with tracking - reparent_entity: Reparents via clone→remove→re-add with circular reference check - set_transform: Sets position/rotation(Euler)/scale via Quantum property graph nodes All operations go through the Stride property graph system for proper undo/redo and UI binding updates. Integration tests added (25 total, all passing). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements the modify_component tool that allows AI agents to add, remove, or update components on entities through the MCP interface. Supports type resolution from short names (e.g. 'ModelComponent') or fully qualified names, JSON property updates with type conversion, and undo/redo. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ests Adds async project build support through build_project (triggers MSBuild via VSProjectHelper) and get_build_status (polls completion, returns errors/warnings). Also adds integration tests for modify_component and build tools (30 total tests, all passing), and updates README with the complete tool listing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a new capture_viewport tool that captures the 3D viewport as a PNG screenshot, enabling AI agents to visually inspect scenes. Uses GPU readback of the DirectX backbuffer via a new EditorGameScreenshotService, following the existing EditorGameCubemapService pattern. Changes in Stride.Assets.Presentation: - IEditorGameScreenshotService: public interface for viewport capture - EditorGameScreenshotService: captures backbuffer on game thread - GameEditorViewModel.GetEditorGameService<T>(): public service accessor - EntityHierarchyEditorController: registers screenshot service Changes in Stride.GameStudio.Mcp: - CaptureViewportTool: returns ImageContentBlock with base64 PNG - Integration test for error case, updated ListTools assertion (31 tests) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 6 new MCP tools for comprehensive asset management: - get_asset_details: Deep inspection of asset properties via [DataMember] reflection - get_asset_dependencies: Reference graph (inbound/outbound/broken links) - create_asset: Create new assets using factory system with type resolution - manage_asset: Rename, move, or delete assets with reference safety checks - set_asset_property: Modify asset properties via Quantum property graph navigation - save_project: Save all changes to disk Extract shared JsonTypeConverter utility from GetEntityTool and ModifyComponentTool to eliminate code duplication for serialization and type conversion. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
save_project persists all editor changes (scenes, entities, assets, etc.), not just asset modifications. Moved it out of Asset Management into a dedicated Project section in the README. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…dance
- JsonTypeConverter: new ConvertJsonToType overload with SessionViewModel
that resolves asset references via ContentReferenceHelper. Accepts
{"assetId":"GUID"}, {"assetRef":"GUID"}, "GUID", or null formats.
- ModifyComponentTool: pass session for asset refs, better error message
when script component types aren't found (suggests build_project).
- SetAssetPropertyTool: pass session for asset refs, updated descriptions.
- CaptureViewportTool: IMPORTANT prefix emphasizing primary verification.
- SaveProjectTool: WARNING about overwriting external file changes.
- ReloadSceneTool: new tool — closes/reopens scene editor tab.
- ReloadProjectTool: new tool — triggers full GameStudio restart via
ReloadSessionCommand reflection.
- Tests: new tests for reload_scene, asset reference update; fixed flaky
SetAssetProperty test (filter by MaterialAsset); fixture now copies the
sample project to a temp directory so tests don't pollute the source tree.
- README: asset reference docs, reload behavior section.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 8 new MCP tools for UI page editing (get_ui_tree, get_ui_element, add_ui_element, remove_ui_element, set_ui_element_property), sprite sheet management (add_sprite_frame, remove_sprite_frame), and project control (set_active_project). Update get_editor_status to include project details. Fix JsonTypeConverter to handle NaN/Infinity float values in serialization. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Widen EditorGameScreenshotService to accept GameEditorViewModel (common base) and register it in UIEditorController so UI page editors support viewport capture. Add open_ui_page tool mirroring open_scene. Update capture_viewport to accept both scene and UI page asset IDs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Enable AI agents to set interface/abstract-typed properties like
LightComponent.Type (ILight) and MaterialAttributes.Diffuse
(IMaterialDiffuseFeature) via modify_component and set_asset_property.
JsonTypeConverter now resolves concrete types by DataContract alias,
class name, or fully qualified name, and instantiates them using
ObjectFactoryRegistry — the same mechanism the editor property grid uses.
Two JSON formats supported:
- String: "LightPoint" (creates default instance)
- Object: {"$type": "LightPoint", "Radius": 5.0} (with inline props)
Invalid type names produce error messages listing all available types.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add manage_root_assets tool (list/add/remove) so agents can mark assets for build inclusion. Uses the editor's undo/redo pattern via AssetDependenciesViewModel.IsRoot for proper model sync. Enhance get_editor_status with isExecutable, recommended, and rootAssetCount fields so agents can identify the correct build target. Update set_active_project to warn when a Library project is selected, since Library projects cannot produce a runnable game. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds an embeddable MCP server that runs inside a live Stride game, enabling AI agents to observe and interact with the running game instance. Double opt-in: game must reference the package AND enable it via env var or McpSettings. Includes 7 MCP tools: get_game_status, get_scene_entities, get_entity, capture_screenshot, get_logs, simulate_input, focus_element. The focus_element tool maps UI element WorldMatrix positions through Stride's centered UI coordinate system (Panel.PanelArrangeMatrix + virtualResolution/2 offset) to screen-space normalized coordinates. Integration tests (26 total) verify all tools including precise UI coordinate validation with elements at top-left, center, and bottom-right positions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Enables setting file/directory path properties (like model Source) via the set_asset_property MCP tool. Both types are constructed from JSON string values using their string constructors. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds deserialization support for all Stride value types commonly used in asset properties: - Numeric primitives: byte, short, ushort, uint, ulong, decimal - Integer vectors: Int2, Int3, Int4 - Float vectors: Vector4 (was missing from deserialization) - Color (byte RGBA, was only serialized) - Geometry: Rectangle, RectangleF, Size2, Size2F, Size3, AngleSingle - System types: Guid, TimeSpan - TypeConverter fallback for any remaining types with registered converters Also improves serialization output for Int2/3/4, Rectangle/F, Size2/2F/3, and AngleSingle with structured JSON instead of ToString() fallback. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
# Conflicts: # build/Stride.sln
…servers Helps LLM agents understand what's rendered in the viewport even when materials or lighting make the captured image hard to interpret. Returns all entities with their projected screen coordinates (normalized 0-1), world positions, distances to camera, and visibility flags. Editor version reconstructs camera matrices from ViewModel settings; game-runtime version uses live CameraComponent matrices directly. Also fixes duplicate PackageVersion entries in Directory.Packages.props from the branch merge. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add NavigateViewportTool with 5 actions: set_orientation, set_position, set_projection, set_field_of_view, get_camera_state - Add dictionary key support to SetAssetPropertyTool (bracket notation) - Add dictionary/list entry support to ModifyComponentTool (bracket notation and whole-dictionary JSON objects) - Add auto-save to ReloadProjectTool before restarting Game Studio - Update CaptureViewportTool description with camera preview tip - Add 12 navigate_viewport integration tests (77/77 passing) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
create_asset now accepts an optional 'source' parameter for importing files (FBX, GLTF, OBJ, PNG, JPG, WAV, etc.) using Stride's asset importer pipeline. The importer auto-detects the asset type and creates all dependent assets (e.g. model + materials + textures from an FBX). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
manage_asset now supports action 'reimport' to reload an asset from its original source file on disk (e.g. FBX, PNG, WAV). Uses Stride's built-in reimport pipeline which preserves user-modified properties while updating data from the source. Supports undo/redo. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ResolveComponentType now uses GetInheritedInstantiableTypes() — the same mechanism as the editor's "Add component" dropdown — to discover all available EntityComponent types including user game scripts. Error messages now list available user script types when a component type is not found. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
After build_project completes, game assemblies need to be reloaded for user-defined script types (e.g. PlayerController) to become available. This tool exposes the editor's "Reload game assemblies" button via MCP. - New reload_assemblies tool with 'status' and 'reload' actions - get_build_status now includes assemblyReloadPending field - modify_component error messages guide users through the full build_project → get_build_status → reload_assemblies workflow Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename the tool to clearly differentiate it from reload_assemblies: - restart_game_studio: full editor restart, drops MCP connection (for .csproj/.sln changes) - reload_assemblies: hot-reloads game scripts without restart (for post-build script type loading) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
build_project now auto-saves before building, matching the editor's Build button behavior (PrepareBuild → SaveSession). This eliminates the error-prone manual save_project → build_project two-step. Updated save_project description to focus on its actual use case: checkpointing work and preparing for external tool access, rather than as a build prerequisite. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
get_entity was missing asset reference properties (Model, Material,
Texture, etc.) because these are stored as proxy objects with metadata
attached via AttachedReferenceManager, not as IReference instances.
Added AttachedReferenceManager.GetAttachedReference() check to
SerializeValue so these properties now appear as {assetRef, url}.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Port improvements from the editor's JsonTypeConverter to
RuntimeEntitySerializer:
- Add AttachedReferenceManager check for asset reference properties
(Model, Material, Texture, etc.) — same fix as editor
- Add Int2/3/4, RectangleF, Rectangle, Size2/2F/3, AngleSingle support
- Add NaN/Infinity handling for float/double
- Add EntityComponent reference serialization
- Align output format with editor ({assetRef, url} instead of {type, url})
- Increase collection limit from 10 to 20 (matching editor)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add editor settings to enable/disable the MCP server and configure its port. The server is now off by default (experimental). When port is set to 0 (default), auto-selects an available port starting from 5271, allowing multiple GameStudio instances to each run their own MCP server. Environment variables STRIDE_MCP_ENABLED and STRIDE_MCP_PORT override settings for CI/testing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… .sdpkg.user MCP server enabled/port settings are now stored per-project in the .sdpkg.user file instead of the global editor settings. This allows each project to independently configure its MCP server, and the settings appear in the per-package properties panel alongside other project-level settings like effect compilation mode. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ution root MCP server settings were stored per-package in .sdpkg.user files, but the server is per-editor-instance. Replace with a .stride/mcp.json file at the solution root that serves as both user config and runtime discovery for agents. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Without this, the file never appeared unless the server was already enabled, making it impossible to discover and enable via the config file. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DialogService now has a [ThreadStatic] SuppressDialogs flag. When set, dialog methods skip showing UI and record messages instead. DispatcherBridge sets this flag around all MCP-dispatched work so modal popups don't block the UI thread and the MCP tool response. Also fixes a permanent hang in InvokeTaskOnUIThread: the upstream DispatcherService.InvokeTask uses a TaskCompletionSource but only calls SetResult on success — if the async lambda throws, the TCS never completes and the caller hangs forever. DispatcherBridge now uses its own TCS with ExecuteAndComplete that guarantees result/cancellation/exception propagation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Prevents NullReferenceException when CheckSkeleton is called before a model has been assigned. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove the per-asset 'reimport' action from manage_asset (unreliable — UI kept blinking, assets didn't update). Instead, both save_project and build_project now automatically reimport all assets whose source files have changed on disk before proceeding, via a shared AssetReimportHelper. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Author
|
@dotnet-policy-service agree |
Collaborator
|
I'm not sure this fits within the engine; 15000 LOC for a niche tool, and from those, maybe 20 lines that touch the rest of the codebase. I would much rather we setup an experimental surface for thirdparty dlls to be loaded alongside the engine and have this live in its own repo. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Details
Summary
This PR adds a Model Context Protocol (MCP) server embedded directly into Stride Game Studio, enabling AI agents to interact with the running editor programmatically: creating entities, editing scenes, managing assets, building projects, and more.
A secondary game-runtime MCP server (
Stride.Engine.Mcp) is also included for inspecting and controlling running games.Description
What's included
Stride.GameStudio.Mcp); disabled by default;Stride.Engine.Mcp); disabled by default;.stride/mcp.jsonat the solution root, auto-created on editor launch.Tools
get_editor_status,query_assets,get_scene_tree,get_entityopen_scene,open_ui_page,select_entity,focus_entitycreate_entity,delete_entity,reparent_entity,set_transform,modify_componentget_asset_details,get_asset_dependencies,create_asset,manage_asset,set_asset_property,manage_root_assets,add_sprite_frame,remove_sprite_frameget_ui_tree,get_ui_element,add_ui_element,remove_ui_element,set_ui_element_propertycapture_viewport,navigate_viewport,describe_viewportbuild_project,get_build_status,reload_assemblies,reload_scene,save_project,restart_game_studio,set_active_projectRelated Issue
I haven't created an issue to discuss it first. I'm sorry =(
Types of changes
Checklist