feat: Add metadata models package with dynamic schema download

- Add airbyte_cdk.metadata_models package with auto-generated Pydantic models
- Models are generated from JSON schemas in airbytehq/airbyte repository
- Schemas are downloaded on-demand during build process (no submodules)
- Uses pydantic.v1 compatibility layer for consistency with declarative models
- Includes comprehensive README with usage examples
- Adds py.typed marker for type hint compliance

The metadata models enable validation of connector metadata.yaml files
using the same schemas maintained in the main Airbyte repository.

Co-Authored-By: AJ Steers <[email protected]>

Author: devin-ai-integration[bot]

airbyte_cdk/metadata_models/README.md

@@ -0,0 +1,79 @@
+# Airbyte Metadata Models
+
+This package contains Pydantic models for validating Airbyte connector `metadata.yaml` files.
+
+## Overview
+
+The models are automatically generated from JSON Schema YAML files maintained in the [airbytehq/airbyte](https://github.com/airbytehq/airbyte) repository at:
+```
+airbyte-ci/connectors/metadata_service/lib/metadata_service/models/src/
+```
+
+During the CDK build process (`poetry run poe build`), these schemas are downloaded from GitHub and used to generate Pydantic models via `datamodel-code-generator`.
+
+## Usage
+
+### Validating a metadata.yaml file
+
+```python
+from pathlib import Path
+import yaml
+from airbyte_cdk.metadata_models import ConnectorMetadataDefinitionV0
+
+# Load metadata.yaml
+metadata_path = Path("path/to/metadata.yaml")
+metadata_dict = yaml.safe_load(metadata_path.read_text())
+
+# Validate using Pydantic
+try:
+    metadata = ConnectorMetadataDefinitionV0(**metadata_dict)
+    print("✓ Metadata is valid!")
+except Exception as e:
+    print(f"✗ Validation failed: {e}")
+```
+
+### Accessing metadata fields
+
+```python
+from airbyte_cdk.metadata_models import ConnectorMetadataDefinitionV0
+
+metadata = ConnectorMetadataDefinitionV0(**metadata_dict)
+
+# Access fields with full type safety
+print(f"Connector: {metadata.data.name}")
+print(f"Docker repository: {metadata.data.dockerRepository}")
+print(f"Docker image tag: {metadata.data.dockerImageTag}")
+print(f"Support level: {metadata.data.supportLevel}")
+```
+
+### Available models
+
+The main model is `ConnectorMetadataDefinitionV0`, which includes nested models for:
+
+- `ConnectorType` - Source or destination
+- `ConnectorSubtype` - API, database, file, etc.
+- `SupportLevel` - Community, certified, etc.
+- `ReleaseStage` - Alpha, beta, generally_available
+- `ConnectorBreakingChanges` - Breaking change definitions
+- `ConnectorReleases` - Release information
+- `AllowedHosts` - Network access configuration
+- And many more...
+
+## Regenerating Models
+
+Models are regenerated automatically when you run:
+
+```bash
+poetry run poe build
+```
+
+This command:
+1. Downloads the latest schema YAML files from the airbyte repository
+2. Generates Pydantic models using `datamodel-code-generator`
+3. Outputs models to `airbyte_cdk/metadata_models/generated/`
+
+## Schema Source
+
+The authoritative schemas are maintained in the [airbyte monorepo](https://github.com/airbytehq/airbyte/tree/master/airbyte-ci/connectors/metadata_service/lib/metadata_service/models/src).
+
+Any changes to metadata validation should be made there, and will be automatically picked up by the CDK build process.

airbyte_cdk/metadata_models/__init__.py

@@ -0,0 +1 @@
+from .generated import *

airbyte_cdk/metadata_models/generated/ActorDefinitionResourceRequirements.py

@@ -0,0 +1,48 @@
+# generated by datamodel-codegen:
+#   filename:  ActorDefinitionResourceRequirements.yaml
+
+from __future__ import annotations
+
+from enum import Enum
+from typing import List, Optional
+
+from pydantic.v1 import BaseModel, Extra, Field
+
+
+class ResourceRequirements(BaseModel):
+    class Config:
+        extra = Extra.forbid
+
+    cpu_request: Optional[str] = None
+    cpu_limit: Optional[str] = None
+    memory_request: Optional[str] = None
+    memory_limit: Optional[str] = None
+
+
+class JobType(Enum):
+    get_spec = "get_spec"
+    check_connection = "check_connection"
+    discover_schema = "discover_schema"
+    sync = "sync"
+    reset_connection = "reset_connection"
+    connection_updater = "connection_updater"
+    replicate = "replicate"
+
+
+class JobTypeResourceLimit(BaseModel):
+    class Config:
+        extra = Extra.forbid
+
+    jobType: JobType
+    resourceRequirements: ResourceRequirements
+
+
+class ActorDefinitionResourceRequirements(BaseModel):
+    class Config:
+        extra = Extra.forbid
+
+    default: Optional[ResourceRequirements] = Field(
+        None,
+        description="if set, these are the requirements that should be set for ALL jobs run for this actor definition.",
+    )
+    jobSpecific: Optional[List[JobTypeResourceLimit]] = None

airbyte_cdk/metadata_models/generated/AirbyteInternal.py

@@ -0,0 +1,39 @@
+# generated by datamodel-codegen:
+#   filename:  AirbyteInternal.yaml
+
+from __future__ import annotations
+
+from enum import Enum
+from typing import Optional
+
+from pydantic.v1 import BaseModel, Extra, Field
+
+
+class Sl(Enum):
+    integer_0 = 0
+    integer_100 = 100
+    integer_200 = 200
+    integer_300 = 300
+
+
+class Ql(Enum):
+    integer_0 = 0
+    integer_100 = 100
+    integer_200 = 200
+    integer_300 = 300
+    integer_400 = 400
+    integer_500 = 500
+    integer_600 = 600
+
+
+class AirbyteInternal(BaseModel):
+    class Config:
+        extra = Extra.allow
+
+    sl: Optional[Sl] = None
+    ql: Optional[Ql] = None
+    isEnterprise: Optional[bool] = False
+    requireVersionIncrementsInPullRequests: Optional[bool] = Field(
+        True,
+        description="When false, version increment checks will be skipped for this connector",
+    )

airbyte_cdk/metadata_models/generated/AllowedHosts.py

@@ -0,0 +1,18 @@
+# generated by datamodel-codegen:
+#   filename:  AllowedHosts.yaml
+
+from __future__ import annotations
+
+from typing import List, Optional
+
+from pydantic.v1 import BaseModel, Extra, Field
+
+
+class AllowedHosts(BaseModel):
+    class Config:
+        extra = Extra.allow
+
+    hosts: Optional[List[str]] = Field(
+        None,
+        description="An array of hosts that this connector can connect to.  AllowedHosts not being present for the source or destination means that access to all hosts is allowed.  An empty list here means that no network access is granted.",
+    )

airbyte_cdk/metadata_models/generated/ConnectorBreakingChanges.py

@@ -0,0 +1,70 @@
+# generated by datamodel-codegen:
+#   filename:  ConnectorBreakingChanges.yaml
+
+from __future__ import annotations
+
+from datetime import date
+from enum import Enum
+from typing import Any, Dict, List, Optional
+
+from pydantic.v1 import AnyUrl, BaseModel, Extra, Field, constr
+
+
+class DeadlineAction(Enum):
+    auto_upgrade = "auto_upgrade"
+    disable = "disable"
+
+
+class StreamBreakingChangeScope(BaseModel):
+    class Config:
+        extra = Extra.forbid
+
+    scopeType: Any = Field("stream", const=True)
+    impactedScopes: List[str] = Field(
+        ...,
+        description="List of streams that are impacted by the breaking change.",
+        min_items=1,
+    )
+
+
+class BreakingChangeScope(BaseModel):
+    __root__: StreamBreakingChangeScope = Field(
+        ...,
+        description="A scope that can be used to limit the impact of a breaking change.",
+    )
+
+
+class VersionBreakingChange(BaseModel):
+    class Config:
+        extra = Extra.forbid
+
+    upgradeDeadline: date = Field(
+        ...,
+        description="The deadline by which to upgrade before the breaking change takes effect.",
+    )
+    message: str = Field(
+        ..., description="Descriptive message detailing the breaking change."
+    )
+    deadlineAction: Optional[DeadlineAction] = Field(
+        None, description="Action to do when the deadline is reached."
+    )
+    migrationDocumentationUrl: Optional[AnyUrl] = Field(
+        None,
+        description="URL to documentation on how to migrate to the current version. Defaults to ${documentationUrl}-migrations#${version}",
+    )
+    scopedImpact: Optional[List[BreakingChangeScope]] = Field(
+        None,
+        description="List of scopes that are impacted by the breaking change. If not specified, the breaking change cannot be scoped to reduce impact via the supported scope types.",
+        min_items=1,
+    )
+
+
+class ConnectorBreakingChanges(BaseModel):
+    class Config:
+        extra = Extra.forbid
+
+    __root__: Dict[constr(regex=r"^\d+\.\d+\.\d+$"), VersionBreakingChange] = Field(
+        ...,
+        description="Each entry denotes a breaking change in a specific version of a connector that requires user action to upgrade.",
+        title="ConnectorBreakingChanges",
+    )

airbyte_cdk/metadata_models/generated/ConnectorBuildOptions.py

@@ -0,0 +1,15 @@
+# generated by datamodel-codegen:
+#   filename:  ConnectorBuildOptions.yaml
+
+from __future__ import annotations
+
+from typing import Optional
+
+from pydantic.v1 import BaseModel, Extra
+
+
+class ConnectorBuildOptions(BaseModel):
+    class Config:
+        extra = Extra.forbid
+
+    baseImage: Optional[str] = None

airbyte_cdk/metadata_models/generated/ConnectorIPCOptions.py

@@ -0,0 +1,36 @@
+# generated by datamodel-codegen:
+#   filename:  ConnectorIPCOptions.yaml
+
+from __future__ import annotations
+
+from enum import Enum
+from typing import List
+
+from pydantic.v1 import BaseModel, Extra
+
+
+class SupportedSerializationEnum(Enum):
+    JSONL = "JSONL"
+    PROTOBUF = "PROTOBUF"
+    FLATBUFFERS = "FLATBUFFERS"
+
+
+class SupportedTransportEnum(Enum):
+    STDIO = "STDIO"
+    SOCKET = "SOCKET"
+
+
+class DataChannel(BaseModel):
+    class Config:
+        extra = Extra.forbid
+
+    version: str
+    supportedSerialization: List[SupportedSerializationEnum]
+    supportedTransport: List[SupportedTransportEnum]
+
+
+class ConnectorIPCOptions(BaseModel):
+    class Config:
+        extra = Extra.forbid
+
+    dataChannel: DataChannel