validate_manifest fails with $ref but runtime works

[devin-ai-integration]

Summary

The validate_manifest tool rejects manifests that use $ref to reference other parts of the manifest, but the runtime successfully executes these manifests. This appears to be a global issue with $ref validation, not specific to any particular component.

Reproduction

Example: Using $ref in SubstreamPartitionRouter

streams:
  - type: DeclarativeStream
    name: repositories
    # ... parent stream definition ...
  
  - type: DeclarativeStream
    name: issues
    retriever:
      type: SimpleRetriever
      partition_router:
        - type: SubstreamPartitionRouter
          parent_stream_configs:
            - type: ParentStreamConfig
              stream:
                $ref: "#/streams/0"  # Reference to repositories stream

Error from validate_manifest:

JSON schema validation failed at field 'retriever': {...} is not valid under any of the given schemas

The error message shows the $ref in the output, indicating the validator is seeing the unresolved reference.

Runtime behavior:

execute_stream_test_read works perfectly with the same manifest - all streams read data successfully.

Suspected Root Cause

1. JSON schema doesn't know how to properly validate manifests with $refs - The schema validator is running on the raw manifest before $refs are resolved
2. Validation tool calls the JSON schema validator directly - Without resolving $refs first

Proposed Mitigations

Option 1 (Best): Fix JSON schema to respect $refs

Come up with a version of the JSON schema which correctly respects the $refs syntax.

Option 2 (Workaround): Resolve manifest before validation

Within the validate_manifest tool, run "resolve manifest" or "fully resolve manifest" inline before JSON schema validation.

Impact

Without this fix, developers must either:

1. Inline definitions everywhere (massive duplication)
2. Skip validation and rely only on runtime testing

This significantly impacts maintainability for complex connectors.

Context

• Manifest version: 6.42.1
• CDK version: 6.61.6
• The connector builder UI itself generates manifests with $ref patterns
• Multiple existing connectors use $ref patterns

Requested By

@aaronsteers while working on source-github-gen2: https://app.devin.ai/sessions/175c32a9b87647a3830163fba2a3a55e

[devin-ai-integration]

Minimal Reproduction

I've created a minimal manifest that demonstrates the issue. The manifest uses $ref: "#/streams/0" to reference the parent stream in SubstreamPartitionRouter.

Files

manifest.yaml:

version: 6.42.1

type: DeclarativeSource

check:
  type: CheckStream
  stream_names:
    - repositories

definitions:
  base_requester:
    type: HttpRequester
    url_base: "https://api.github.com"
    http_method: "GET"
    authenticator:
      type: BearerAuthenticator
      api_token: "{{ config['access_token'] }}"
    request_headers:
      User-Agent: "Airbyte"
      Accept: "application/vnd.github+json"
      X-GitHub-Api-Version: "2022-11-28"

  base_paginator:
    type: DefaultPaginator
    page_token_option:
      type: RequestOption
      inject_into: request_parameter
      field_name: page
    page_size_option:
      type: RequestOption
      inject_into: request_parameter
      field_name: per_page
    pagination_strategy:
      type: PageIncrement
      page_size: 100
      start_from_page: 1

streams:
  - type: DeclarativeStream
    name: repositories
    primary_key: id
    retriever:
      type: SimpleRetriever
      requester:
        $ref: "#/definitions/base_requester"
        path: "/repos/{{ stream_slice.repository }}"
      record_selector:
        type: RecordSelector
        extractor:
          type: DpathExtractor
          field_path: []
      paginator:
        type: NoPagination
      partition_router:
        - type: ListPartitionRouter
          values: "{{ config.repositories }}"
          cursor_field: repository

  - type: DeclarativeStream
    name: issues
    primary_key: id
    retriever:
      type: SimpleRetriever
      requester:
        $ref: "#/definitions/base_requester"
        path: "/repos/{{ stream_slice.repository }}/issues"
      record_selector:
        type: RecordSelector
        extractor:
          type: DpathExtractor
          field_path: []
      paginator:
        $ref: "#/definitions/base_paginator"
      partition_router:
        - type: SubstreamPartitionRouter
          parent_stream_configs:
            - type: ParentStreamConfig
              parent_key: full_name
              partition_field: repository
              stream:
                $ref: "#/streams/0"  # <-- This $ref causes validation to fail

spec:
  type: Spec
  connection_specification:
    type: object
    $schema: http://json-schema.org/draft-07/schema#
    required:
      - access_token
      - repositories
    properties:
      access_token:
        type: string
        title: Personal Access Token
        airbyte_secret: true
        order: 0
      repositories:
        type: array
        items:
          type: string
        title: GitHub Repositories
        description: List of repositories in owner/repo format
        examples:
          - airbytehq/airbyte
        order: 1
    additionalProperties: true

config.json:

{
  "access_token": "YOUR_GITHUB_TOKEN_HERE",
  "repositories": ["airbytehq/integration-test"]
}

Reproduction Steps

Step 1: Validation FAILS ❌

Using the MCP tool validate_manifest:

from mcp_tool import call_tool

result = call_tool(
    server="connector-builder-mcp",
    tool_name="validate_manifest",
    tool_args={"manifest": "/path/to/manifest.yaml"}
)

Result:

{
  "is_valid": false,
  "errors": [
    "JSON schema validation failed at field 'retriever': {...'partition_router': [{'type': 'SubstreamPartitionRouter', 'parent_stream_configs': [{'type': 'ParentStreamConfig', 'parent_key': 'full_name', 'partition_field': 'repository', 'stream': {'$ref': '#/streams/0'}}]}]} is not valid under any of the given schemas"
  ]
}

Step 2: Runtime SUCCEEDS ✅

Using the MCP tool execute_stream_test_read:

# Test repositories stream
result = call_tool(
    server="connector-builder-mcp",
    tool_name="execute_stream_test_read",
    tool_args={
        "manifest": "/path/to/manifest.yaml",
        "config": {"repositories": ["airbytehq/integration-test"], "access_token": "ghp_..."},
        "stream_name": "repositories",
        "max_records": 1
    }
)
# Result: {"success": true, "records_read": 1}

# Test issues stream (uses $ref to parent)
result = call_tool(
    server="connector-builder-mcp",
    tool_name="execute_stream_test_read",
    tool_args={
        "manifest": "/path/to/manifest.yaml",
        "config": {"repositories": ["airbytehq/integration-test"], "access_token": "ghp_..."},
        "stream_name": "issues",
        "max_records": 3
    }
)
# Result: {"success": true, "records_read": 3}

Both streams work perfectly at runtime despite validation failure.

Key Observation

The error message shows the unresolved $ref in the output:

'stream': {'$ref': '#/streams/0'}

This indicates the validator is running on the raw manifest before $ref resolution, while the runtime properly resolves $ref before execution.