-
Notifications
You must be signed in to change notification settings - Fork 126
apps MCP: restructure context management #4073
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
598ddbc
apps-mcp: add context management design doc
arsenyinfo 8a8e27b
upd
arsenyinfo 590043a
check in
arsenyinfo c97c7a3
add tests
arsenyinfo 5ca293e
rollback prompt change
arsenyinfo 600a9c1
Merge branch 'main' into apps-mcp-context-management
arsenyinfo 30dbdf8
address PR comments
arsenyinfo File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| <!-- DO NOT MODIFY: This documentation defines the context management architecture for Databricks MCP --> | ||
| # Context Management for Databricks MCP | ||
|
|
||
| ## Goals | ||
|
|
||
| - Universal MCP for any coding agent (Claude, Cursor, etc.) | ||
| - Support multiple target types: apps, jobs, pipelines | ||
| - Support multiple templates per target type | ||
| - Clean separation of context layers | ||
| - Detect existing project context automatically | ||
|
|
||
| ## Context Layers | ||
|
|
||
| | Layer | Content | When Injected | | ||
| |-------|---------|---------------| | ||
| | **L0: Tools** | Databricks MCP tool names and descriptions | Always (MCP protocol) | | ||
| | **L1: Flow** | Universal workflow, available tools, CLI patterns | Always (via `databricks_discover`) | | ||
| | **L2: Target** | Target-specific: validation, deployment, constraints | When target type detected or after `init-template` | | ||
| | **L3: Template** | SDK/language-specific: file structure, commands, patterns | After `init-template`. For existing projects, agent reads CLAUDE.md. | | ||
|
|
||
| L0 is implicit - tool descriptions guide agent behavior before any tool is called (e.g., `databricks_discover` description tells agent to call it first during planning). | ||
|
|
||
| ### Examples | ||
|
|
||
| **L1 (universal):** "validate before deploying", "use invoke_databricks_cli for all commands" | ||
|
|
||
| **L2 (apps):** app naming constraints, deployment consent requirement, app-specific validation | ||
|
|
||
| **L3 (appkit-typescript):** npm scripts, tRPC patterns, useAnalyticsQuery usage, TypeScript import rules | ||
|
|
||
| ## Flows | ||
|
|
||
| ### New Project | ||
|
|
||
| ``` | ||
| Agent MCP | ||
| │ │ | ||
| ├─► databricks_discover │ | ||
| │ {working_directory: "."} │ | ||
| │ ├─► Run detectors (nothing found) | ||
| │ ├─► Return L1 only | ||
| │◄─────────────────────────────┤ | ||
| │ │ | ||
| ├─► invoke_databricks_cli │ | ||
| │ ["...", "init-template", ...] | ||
| │ ├─► Scaffold project | ||
| │ ├─► Return L2[apps] + L3 | ||
| │◄─────────────────────────────┤ | ||
| │ │ | ||
| ├─► (agent now has L1 + L2 + L3) | ||
| ``` | ||
|
|
||
| ### Existing Project | ||
|
|
||
| ``` | ||
| Agent MCP | ||
| │ │ | ||
| ├─► databricks_discover │ | ||
| │ {working_directory: "./my-app"} | ||
| │ ├─► BundleDetector: found apps + jobs | ||
| │ ├─► Return L1 + L2[apps] + L2[jobs] | ||
| │◄─────────────────────────────┤ | ||
| │ │ | ||
| ├─► Read CLAUDE.md naturally │ | ||
| │ (agent learns L3 itself) │ | ||
| ``` | ||
|
|
||
| ### Combined Bundles | ||
|
|
||
| When `databricks.yml` contains multiple resource types (e.g., app + job), all relevant L2 layers are injected together. | ||
|
|
||
| ## Extensibility | ||
|
|
||
| New target types can be added by: | ||
| 1. Creating `target_<type>.tmpl` in `lib/prompts/` | ||
| 2. Adding detection logic to recognize the target type from `databricks.yml` | ||
|
|
||
| New templates can be added by: | ||
| 1. Creating template directory with CLAUDE.md (L3 guidance) | ||
| 2. Adding detection logic to recognize the template from project files | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| package detector | ||
|
|
||
| import ( | ||
| "context" | ||
| "os" | ||
| "path/filepath" | ||
|
|
||
| "github.com/databricks/cli/bundle" | ||
| "github.com/databricks/cli/bundle/phases" | ||
| "github.com/databricks/cli/libs/logdiag" | ||
| ) | ||
|
|
||
| // BundleDetector detects Databricks bundle configuration. | ||
| type BundleDetector struct{} | ||
|
|
||
| // Detect loads databricks.yml with all includes and extracts target types. | ||
| func (d *BundleDetector) Detect(ctx context.Context, workDir string, detected *DetectedContext) error { | ||
igrekun marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| bundlePath := filepath.Join(workDir, "databricks.yml") | ||
| if _, err := os.Stat(bundlePath); err != nil { | ||
| // no bundle file - not an error, just not a bundle project | ||
| return nil | ||
| } | ||
|
|
||
| // use full bundle loading to get all resources including from includes | ||
| ctx = logdiag.InitContext(ctx) | ||
| b, err := bundle.Load(ctx, workDir) | ||
| if err != nil || b == nil { | ||
| return nil | ||
| } | ||
|
|
||
| phases.Load(ctx, b) | ||
| if logdiag.HasError(ctx) { | ||
| return nil | ||
| } | ||
|
|
||
| detected.InProject = true | ||
| detected.BundleInfo = &BundleInfo{ | ||
| Name: b.Config.Bundle.Name, | ||
| RootDir: workDir, | ||
| } | ||
|
|
||
| // extract target types from fully loaded resources | ||
| if len(b.Config.Resources.Apps) > 0 { | ||
| detected.TargetTypes = append(detected.TargetTypes, "apps") | ||
| } | ||
| if len(b.Config.Resources.Jobs) > 0 { | ||
| detected.TargetTypes = append(detected.TargetTypes, "jobs") | ||
| } | ||
| if len(b.Config.Resources.Pipelines) > 0 { | ||
| detected.TargetTypes = append(detected.TargetTypes, "pipelines") | ||
| } | ||
|
|
||
| return nil | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| // Package detector provides project context detection for Databricks MCP. | ||
| package detector | ||
|
|
||
| import ( | ||
| "context" | ||
| ) | ||
|
|
||
| // BundleInfo contains information about a detected Databricks bundle. | ||
| type BundleInfo struct { | ||
| Name string | ||
| Target string | ||
| RootDir string | ||
| } | ||
|
|
||
| // DetectedContext represents the detected project context. | ||
| type DetectedContext struct { | ||
| InProject bool | ||
| TargetTypes []string // ["apps", "jobs"] - supports combined bundles | ||
| Template string // "appkit-typescript", "python", etc. | ||
| BundleInfo *BundleInfo | ||
| Metadata map[string]string | ||
| } | ||
|
|
||
| // Detector detects project context from a working directory. | ||
| type Detector interface { | ||
| // Detect examines the working directory and updates the context. | ||
| Detect(ctx context.Context, workDir string, detected *DetectedContext) error | ||
| } | ||
|
|
||
| // Registry manages a collection of detectors. | ||
| type Registry struct { | ||
| detectors []Detector | ||
| } | ||
|
|
||
| // NewRegistry creates a new detector registry with default detectors. | ||
| func NewRegistry() *Registry { | ||
| return &Registry{ | ||
| detectors: []Detector{ | ||
| &BundleDetector{}, | ||
| &TemplateDetector{}, | ||
| }, | ||
| } | ||
| } | ||
|
|
||
| // Detect runs all detectors and returns the combined context. | ||
| func (r *Registry) Detect(ctx context.Context, workDir string) *DetectedContext { | ||
| detected := &DetectedContext{ | ||
| Metadata: make(map[string]string), | ||
| } | ||
|
|
||
| for _, d := range r.detectors { | ||
| // ignore errors - detectors should be resilient | ||
| _ = d.Detect(ctx, workDir, detected) | ||
| } | ||
|
|
||
| return detected | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| package detector_test | ||
|
|
||
| import ( | ||
| "context" | ||
| "os" | ||
| "path/filepath" | ||
| "testing" | ||
|
|
||
| "github.com/databricks/cli/experimental/apps-mcp/lib/detector" | ||
| "github.com/stretchr/testify/assert" | ||
| "github.com/stretchr/testify/require" | ||
| ) | ||
|
|
||
| func TestDetectorRegistry_EmptyDir(t *testing.T) { | ||
| dir := t.TempDir() | ||
| ctx := context.Background() | ||
|
|
||
| registry := detector.NewRegistry() | ||
| detected := registry.Detect(ctx, dir) | ||
|
|
||
| assert.False(t, detected.InProject) | ||
| assert.Empty(t, detected.TargetTypes) | ||
| assert.Empty(t, detected.Template) | ||
| } | ||
|
|
||
| func TestDetectorRegistry_BundleWithApps(t *testing.T) { | ||
| dir := t.TempDir() | ||
| ctx := context.Background() | ||
|
|
||
| bundleYml := `bundle: | ||
| name: my-app | ||
| resources: | ||
| apps: | ||
| my_app: {} | ||
| ` | ||
| require.NoError(t, os.WriteFile(filepath.Join(dir, "databricks.yml"), []byte(bundleYml), 0o644)) | ||
|
|
||
| registry := detector.NewRegistry() | ||
| detected := registry.Detect(ctx, dir) | ||
|
|
||
| assert.True(t, detected.InProject) | ||
| assert.Equal(t, []string{"apps"}, detected.TargetTypes) | ||
| assert.Equal(t, "my-app", detected.BundleInfo.Name) | ||
| } | ||
|
|
||
| func TestDetectorRegistry_BundleWithJobs(t *testing.T) { | ||
| dir := t.TempDir() | ||
| ctx := context.Background() | ||
|
|
||
| bundleYml := `bundle: | ||
| name: my-job | ||
| resources: | ||
| jobs: | ||
| daily_job: {} | ||
| ` | ||
| require.NoError(t, os.WriteFile(filepath.Join(dir, "databricks.yml"), []byte(bundleYml), 0o644)) | ||
|
|
||
| registry := detector.NewRegistry() | ||
| detected := registry.Detect(ctx, dir) | ||
|
|
||
| assert.True(t, detected.InProject) | ||
| assert.Equal(t, []string{"jobs"}, detected.TargetTypes) | ||
| assert.Equal(t, "my-job", detected.BundleInfo.Name) | ||
| } | ||
|
|
||
| func TestDetectorRegistry_CombinedBundle(t *testing.T) { | ||
| dir := t.TempDir() | ||
| ctx := context.Background() | ||
|
|
||
| bundleYml := `bundle: | ||
| name: my-project | ||
| resources: | ||
| apps: | ||
| my_app: {} | ||
| jobs: | ||
| daily_job: {} | ||
| ` | ||
| require.NoError(t, os.WriteFile(filepath.Join(dir, "databricks.yml"), []byte(bundleYml), 0o644)) | ||
|
|
||
| registry := detector.NewRegistry() | ||
| detected := registry.Detect(ctx, dir) | ||
|
|
||
| assert.True(t, detected.InProject) | ||
| assert.Contains(t, detected.TargetTypes, "apps") | ||
| assert.Contains(t, detected.TargetTypes, "jobs") | ||
| assert.Equal(t, "my-project", detected.BundleInfo.Name) | ||
| } | ||
|
|
||
| func TestDetectorRegistry_AppkitTemplate(t *testing.T) { | ||
| dir := t.TempDir() | ||
| ctx := context.Background() | ||
|
|
||
| // bundle + package.json with appkit marker | ||
| bundleYml := `bundle: | ||
| name: my-app | ||
| resources: | ||
| apps: | ||
| my_app: {} | ||
| ` | ||
| packageJson := `{ | ||
| "name": "my-app", | ||
| "dependencies": { | ||
| "@databricks/sql": "^1.0.0" | ||
| } | ||
| }` | ||
| require.NoError(t, os.WriteFile(filepath.Join(dir, "databricks.yml"), []byte(bundleYml), 0o644)) | ||
| require.NoError(t, os.WriteFile(filepath.Join(dir, "package.json"), []byte(packageJson), 0o644)) | ||
|
|
||
| registry := detector.NewRegistry() | ||
| detected := registry.Detect(ctx, dir) | ||
|
|
||
| assert.True(t, detected.InProject) | ||
| assert.Equal(t, []string{"apps"}, detected.TargetTypes) | ||
| assert.Equal(t, "appkit-typescript", detected.Template) | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lets make the comment there for Claude and Cursor not to modify