aitools: Add integrated skills system for domain-specific guidance (#4199)

## Changes

This adds a **skills system** to apps-mcp that provides domain-specific
implementation guides to AI agents. It follows up on
#4183. Skills are organized by
resource type and delivered through the discover tool. They work for any
agent that supports MCP.

Detailed overview:
* Skills live in `lib/skills/{apps,jobs,pipelines,...}` and include a
standard `SKILL.md` file with a skill front matter.
* The list of skills is shared via the `databricks_discover` tool.
* Skills can be read using the `read_skill_file` tool (which has a
precedent in many other MCPs an in our first-party agent).
* Adding useful skills is considered followup work. The present PR
includes one sample skill for doing auto-CDC with pipelines.

- [x] **Stacked PR**: this branch is based on [the `lakeflow-mcp`
branch](#4183); before merging it
needs to be based on `main` instead

## Why

* Supporting skills enables us to extend the set of domain-specific
capabilities of agents
* We want skills to be the universal format for these capabilities
across different agent implementations.
* Not all agents natively support skills (Codex, Cursor); the
`read_skill_file` tool helps these agents use skills.

## Tests

- Extended detector tests to validate resource type detection
- Updated acceptance tests for all init-template commands
- Manual validation with agents using the skills system

---------

Co-authored-by: Claude <[email protected]>

Author: lennartkats-db

acceptance/apps/init-template/app/output.txt

@@ -1 +1,139 @@
-✓ Template instantiation succeeded
+--
+## Databricks Apps Development
+
+### Validation
+⚠️ Always validate before deploying:
+  invoke_databricks_cli 'experimental aitools tools validate ./'
+
+This is battle-tested to catch common issues before deployment. Prefer using this over manual checks (e.g. `npm run lint`), as it covers more ground specific to Databricks Apps.
+
+### Deployment
+⚠️ USER CONSENT REQUIRED: Only deploy with explicit user permission.
+  invoke_databricks_cli 'experimental aitools tools deploy'
+
+### View and Manage
+  invoke_databricks_cli 'bundle summary'
+
+### View App Logs
+To troubleshoot deployed apps, view their logs:
+  invoke_databricks_cli 'apps logs <app-name> --tail-lines 100'
+
+### Local Development vs Deployed Apps
+
+During development:
+- Start template-specific dev server (see project's CLAUDE.md for command and port)
+- Use localhost URL shown when dev server starts
+
+After deployment:
+- Get URL from: invoke_databricks_cli 'bundle summary'
+
+Decision tree:
+- "open the app" + not deployed → localhost
+- "open the app" + deployed → ask which environment
+- "localhost"/"local" → always localhost
+
+
+## Skills
+
+You have access to modular Skills for domain-specific expertise knowledge.
+
+### Skill Selection & Loading
+* When a user request matches a skill's scope description, select that Skill
+* Load skills using the MCP tool: `read_skill_file(file_path: "category/skill-name/SKILL.md")`
+* Example: `read_skill_file(file_path: "pipelines/materialized-view/SKILL.md")`
+* Skills may contain links to sub-sections (e.g., "category/skill-name/file.md")
+* If no Skill is suitable, continue with your base capabilities
+* Never mention or reference skills to the user, only use them internally
+
+### Skill Registry (names + brief descriptors)
+
+
+**Note**: The following skills are for other resource types and may not be directly relevant to this project.
+
+* **pipelines/auto-cdc/SKILL.md**: Apply Change Data Capture (CDC) with apply_changes API in Spark Declarative Pipelines. Use when user needs to process CDC feeds from databases, handle upserts/deletes, maintain slowly changing dimensions (SCD Type 1 and Type 2), synchronize data from operational databases, or process merge operations.
+
+
+
+=== CLAUDE.md ===
+TypeScript full-stack template powered by **Databricks AppKit** with tRPC for additional custom API endpoints.
+
+- server/: Node.js backend with App Kit and tRPC
+- client/: React frontend with App Kit hooks and tRPC client
+- config/queries/: SQL query files for analytics
+- shared/: Shared TypeScript types
+- docs/: Detailed documentation on using App Kit features
+
+## Quick Start: Your First Query & Chart
+
+Follow these 3 steps to add data visualization to your app:
+
+**Step 1: Create a SQL query file**
+
+```sql
+-- config/queries/my_data.sql
+SELECT category, COUNT(*) as count, AVG(value) as avg_value
+FROM my_table
+GROUP BY category
+```
+
+**Step 2: Define the schema**
+
+```typescript
+// config/queries/schema.ts
+export const querySchemas = {
+  my_data: z.array(
+    z.object({
+      category: z.string(),
+      count: z.number(),
+      avg_value: z.number(),
+    })
+  ),
+};
+```
+
+**Step 3: Add visualization to your app**
+
+```typescript
+// client/src/App.tsx
+import { BarChart } from '@databricks/appkit-ui/react';
+
+<BarChart queryKey="my_data" parameters={{}} />
+```
+
+**That's it!** The component handles data fetching, loading states, and rendering automatically.
+
+**To refresh TypeScript types after adding queries:**
+- Run `npm run typegen` OR run `npm run dev` - both auto-generate type definitions in `client/src/appKitTypes.d.ts`
+- DO NOT manually edit `appKitTypes.d.ts`
+
+## Installation
+
+**IMPORTANT**: When running `npm install`, always use `required_permissions: ['all']` to avoid sandbox permission errors.
+
+## NPM Scripts
+
+### Development
+- `npm run dev` - Start dev server with hot reload (**ALWAYS use during development**)
+
+### Testing and Code Quality
+See the databricks experimental aitools tools validate instead of running these individually.
+
+### Utility
+- `npm run clean` - Remove all build artifacts and node_modules
+
+**Common workflows:**
+- Development: `npm run dev` → make changes → `npm run typecheck` → `npm run lint:fix`
+- Pre-deploy: Validate with `databricks experimental aitools tools validate .`
+
+## Documentation
+
+**IMPORTANT**: Read the relevant docs below before implementing features. They contain critical information about common pitfalls (e.g., SQL numeric type handling, schema definitions, Radix UI constraints).
+
+- [SQL Queries](docs/sql-queries.md) - query files, schemas, type handling, parameterization
+- [App Kit SDK](docs/appkit-sdk.md) - TypeScript imports, server setup, useAnalyticsQuery hook
+- [Frontend](docs/frontend.md) - visualization components, styling, layout, Radix constraints
+- [tRPC](docs/trpc.md) - custom endpoints for non-SQL operations (mutations, Databricks APIs)
+- [Testing](docs/testing.md) - vitest unit tests, Playwright smoke/E2E tests
+
+=================
+

acceptance/apps/init-template/app/script

@@ -1,4 +1,3 @@
 #!/bin/bash
-$CLI experimental aitools tools init-template app --name test_app --sql-warehouse-id abc123 --output-dir output > /dev/null 2>&1
-echo "✓ Template instantiation succeeded"
+$CLI experimental aitools tools init-template app --name test_app --sql-warehouse-id abc123 --output-dir output 2>&1 | grep -A 9999 "^--$"
 rm -rf output

acceptance/apps/init-template/empty/output.txt

@@ -1 +1,66 @@
-✓ Template instantiation succeeded
+--
+## Adding Databricks Resources
+
+Add resources by creating YAML files in resources/:
+
+**Jobs** - `resources/my_job.job.yml`:
+```yaml
+resources:
+  jobs:
+    my_job:
+      name: my_job
+      tasks:
+        - task_key: main
+          notebook_task:
+            notebook_path: ../src/notebook.py
+```
+
+**Pipelines** (Lakeflow Declarative Pipelines) - `resources/my_pipeline.pipeline.yml`:
+```yaml
+resources:
+  pipelines:
+    my_pipeline:
+      name: my_pipeline
+      catalog: ${var.catalog}
+      target: ${var.schema}
+      libraries:
+        - notebook:
+            path: ../src/pipeline.py
+```
+
+**Dashboards** - `resources/my_dashboard.dashboard.yml`
+**Alerts** - `resources/my_alert.alert.yml`
+**Model Serving** - `resources/my_endpoint.yml`
+**Apps** - `resources/my_app.app.yml`
+
+**Other resource types**: clusters, schemas, volumes, registered_models, experiments, quality_monitors
+
+### Deployment
+For dev targets you can deploy without user consent. This allows you to run resources on the workspace too!
+
+  invoke_databricks_cli 'bundle deploy --target dev'
+  invoke_databricks_cli 'bundle run <resource_name> --target dev'
+
+View status with `invoke_databricks_cli 'bundle summary'`.
+
+### Documentation
+- Resource types reference: https://docs.databricks.com/dev-tools/bundles/resources
+- YAML examples: https://docs.databricks.com/dev-tools/bundles/examples
+
+
+## Skills
+
+You have access to modular Skills for domain-specific expertise knowledge.
+
+### Skill Selection & Loading
+* When a user request matches a skill's scope description, select that Skill
+* Load skills using the MCP tool: `read_skill_file(file_path: "category/skill-name/SKILL.md")`
+* Example: `read_skill_file(file_path: "pipelines/materialized-view/SKILL.md")`
+* Skills may contain links to sub-sections (e.g., "category/skill-name/file.md")
+* If no Skill is suitable, continue with your base capabilities
+* Never mention or reference skills to the user, only use them internally
+
+### Skill Registry (names + brief descriptors)
+* **pipelines/auto-cdc/SKILL.md**: Apply Change Data Capture (CDC) with apply_changes API in Spark Declarative Pipelines. Use when user needs to process CDC feeds from databases, handle upserts/deletes, maintain slowly changing dimensions (SCD Type 1 and Type 2), synchronize data from operational databases, or process merge operations.
+
+

acceptance/apps/init-template/empty/script

@@ -1,4 +1,3 @@
 #!/bin/bash
-$CLI experimental aitools tools init-template empty --name test_empty --catalog main --output-dir output > /dev/null 2>&1
-echo "✓ Template instantiation succeeded"
+$CLI experimental aitools tools init-template empty --name test_empty --catalog main --output-dir output 2>&1 | grep -A 9999 "^--$"
 rm -rf output

acceptance/apps/init-template/job/output.txt

@@ -1 +1,120 @@
-✓ Template instantiation succeeded
+--
+## Lakeflow Jobs Development
+
+This guidance is for developing jobs in this project.
+
+### Project Structure
+- `src/` - Python notebooks (.ipynb) and source code
+- `resources/` - Job definitions in databricks.yml format
+
+### Configuring Tasks
+Edit `resources/<job_name>.job.yml` to configure tasks:
+
+```yaml
+tasks:
+  - task_key: my_notebook
+    notebook_task:
+      notebook_path: ../src/my_notebook.ipynb
+  - task_key: my_python
+    python_wheel_task:
+      package_name: my_package
+      entry_point: main
+```
+
+Task types: `notebook_task`, `python_wheel_task`, `spark_python_task`, `pipeline_task`, `sql_task`
+
+### Job Parameters
+Parameters defined at job level are passed to ALL tasks (no need to repeat per task). Example:
+```yaml
+resources:
+  jobs:
+    my_job:
+      parameters:
+        - name: catalog
+          default: ${var.catalog}
+        - name: schema
+          default: ${var.schema}
+```
+
+### Writing Notebook Code
+- Use `spark.read.table("catalog.schema.table")` to read tables
+- Use `spark.sql("SELECT ...")` for SQL queries
+- Use `dbutils.widgets` for parameters
+
+### Unit Testing
+Run unit tests locally with: `uv run pytest`
+
+### Documentation
+- Lakeflow Jobs: https://docs.databricks.com/jobs
+- Task types: https://docs.databricks.com/jobs/configure-task
+- Databricks Asset Bundles / yml format examples: https://docs.databricks.com/dev-tools/bundles/examples
+
+## Adding Databricks Resources
+
+Add resources by creating YAML files in resources/:
+
+**Jobs** - `resources/my_job.job.yml`:
+```yaml
+resources:
+  jobs:
+    my_job:
+      name: my_job
+      tasks:
+        - task_key: main
+          notebook_task:
+            notebook_path: ../src/notebook.py
+```
+
+**Pipelines** (Lakeflow Declarative Pipelines) - `resources/my_pipeline.pipeline.yml`:
+```yaml
+resources:
+  pipelines:
+    my_pipeline:
+      name: my_pipeline
+      catalog: ${var.catalog}
+      target: ${var.schema}
+      libraries:
+        - notebook:
+            path: ../src/pipeline.py
+```
+
+**Dashboards** - `resources/my_dashboard.dashboard.yml`
+**Alerts** - `resources/my_alert.alert.yml`
+**Model Serving** - `resources/my_endpoint.yml`
+**Apps** - `resources/my_app.app.yml`
+
+**Other resource types**: clusters, schemas, volumes, registered_models, experiments, quality_monitors
+
+### Deployment
+For dev targets you can deploy without user consent. This allows you to run resources on the workspace too!
+
+  invoke_databricks_cli 'bundle deploy --target dev'
+  invoke_databricks_cli 'bundle run <resource_name> --target dev'
+
+View status with `invoke_databricks_cli 'bundle summary'`.
+
+### Documentation
+- Resource types reference: https://docs.databricks.com/dev-tools/bundles/resources
+- YAML examples: https://docs.databricks.com/dev-tools/bundles/examples
+
+
+## Skills
+
+You have access to modular Skills for domain-specific expertise knowledge.
+
+### Skill Selection & Loading
+* When a user request matches a skill's scope description, select that Skill
+* Load skills using the MCP tool: `read_skill_file(file_path: "category/skill-name/SKILL.md")`
+* Example: `read_skill_file(file_path: "pipelines/materialized-view/SKILL.md")`
+* Skills may contain links to sub-sections (e.g., "category/skill-name/file.md")
+* If no Skill is suitable, continue with your base capabilities
+* Never mention or reference skills to the user, only use them internally
+
+### Skill Registry (names + brief descriptors)
+
+
+**Note**: The following skills are for other resource types and may not be directly relevant to this project.
+
+* **pipelines/auto-cdc/SKILL.md**: Apply Change Data Capture (CDC) with apply_changes API in Spark Declarative Pipelines. Use when user needs to process CDC feeds from databases, handle upserts/deletes, maintain slowly changing dimensions (SCD Type 1 and Type 2), synchronize data from operational databases, or process merge operations.
+
+

acceptance/apps/init-template/job/script

@@ -1,4 +1,3 @@
 #!/bin/bash
-$CLI experimental aitools tools init-template job --name test_job --catalog main --output-dir output > /dev/null 2>&1 || exit 1
-echo "✓ Template instantiation succeeded"
+$CLI experimental aitools tools init-template job --name test_job --catalog main --output-dir output 2>&1 | grep -A 9999 "^--$"
 rm -rf output