Local runtime schema serialization endpoints (#2386)

yarikoptic · claude · mvandenburgh · web-flow · commit c14ec261251b · 2025-10-07T11:33:45.000-04:00
* Design document with a diagram of metadata life cycle Metadata lifecycle inspired by the one we created for BIDS: see bids-standard/bids-website#626 * Remove duplication with now present vendorization issue * Address all questions and detail implementation more * Implement local schema serialization endpoints - Add new API endpoints to serialize JSONSchema at runtime - Update info endpoint to use local schema URL instead of GitHub - Use TypeAdapter for proper Pydantic schema generation - Add tests for schema endpoints - Add implementation documentation 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * doc: Add mermaid diagram for updated schema architecture - Add visual representation of the new schema flow - Highlight key differences from the current approach - Show elimination of dependency on external schema repository 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Provide custom name for django testserver instance The default name django uses is "testserver" and then URL becomes http://testserver, but DJANGO itself does not consider it to be a valid URL because it lacks TLD. So whenever we add "schema_url" pointing to our server (in tests just "testserver") DJANGO starts raising validation errors. See encode/django-rest-framework#9705 for more information/rationale. * RF: refactor auto-AI-generated tests into parametrized ones to concetrate the testing logic and also to accent on the differences (parameters of the test) * Add GitHub schema comparison to test_schema_latest Extends test to compare local runtime-generated schema against static GitHub schema content. This test is expected to fail, demonstrating that vendorized schemas differ from static schemas due to runtime customizations. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Use TransitionalGenerateJsonSchema with proper type annotations Refactors schema generation to match dandischema approach and adds type safety. * Fix schema endpoints to use regular Dandiset/Asset models Changes from Published models to regular models to match GitHub static schemas: - Use Dandiset instead of PublishedDandiset - Use Asset instead of PublishedAsset - Update test parameterization accordingly Test now shows minimal vendorization difference (repository default value) demonstrating the expected runtime customization vs static schema difference. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Adjust test to handle vendorization differences Add logic to normalize vendorization differences before schema comparison: - Adjust repository default value from runtime config to match GitHub schema - Test now passes, confirming schemas are equivalent after vendorization adjustment - Demonstrates successful runtime schema generation with expected customizations 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Add published schema endpoints and refactor to reduce code duplication Extends the schema API to support both regular and published models: New endpoints: - /api/schema/latest/published-dandiset/ - /api/schema/<version>/published-dandiset/ - /api/schema/latest/published-asset/ - /api/schema/<version>/published-asset/ Refactoring improvements: - Extract common logic into _schema_view_impl() helper function - Update URL patterns and view exports Testing enhancements: - Extend parametrized tests to cover published schema endpoints 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fixup to mermaid diagram In d41f24e I committed a little rushed tune up by claude and it replaced "current state" diagram with the actually implemented, and then for implemented it just removed having jsonschema serializations altogether. Since we did merges into this branch, rebasing is "tricky", so I decided to just push a fixup commit. * Remove portion of the test checking for identity to released schemas * chore: remove unused import This resolve a complain from ruff * style: use `settings.DANDI_API_URL` to construct schema URL There are already consistent uses of `settings.DANDI_API_URL` for building URLs in the codebase. Check out #2386 (comment) detailed rationale behind this change. Co-authored-by: Mike VanDenburgh <37340715+mvandenburgh@users.noreply.github.com> * Don't re-implement get_schema_url in tests * Unify schema endpoints * Remove version query param from schema endpoint * Update wording Co-authored-by: Isaac To <candleindark@users.noreply.github.com> * Add test against unsupported model values * Update schema endpoint swagger docs * Add endpoint for listing available models * Fix view import/export * feat: change endpoints related to serving schemas * doc: update design docs regarding endpoint changes. * docs: remove mention of `TypeAdapter` in `vendored-schema-1-implementation.md` Pydantic models can generate their own JSONSchema, and we do generate the JSONSchema directly from the models. TypeAdapter in Pydantic has other uses, but we are not using it here. * docs: consolidate vendor-configurable metadata models design docs into one file --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Mike VanDenburgh <michael.vandenburgh@kitware.com> Co-authored-by: Isaac To <isaac.chun.to@gmail.com> Co-authored-by: Isaac To <candleindark@users.noreply.github.com> Co-authored-by: Mike VanDenburgh <37340715+mvandenburgh@users.noreply.github.com> Co-authored-by: Jacob Nesbitt <jjnesbitt2@gmail.com>
diff --git a/dandiapi/api/tests/test_info.py b/dandiapi/api/tests/test_info.py
@@ -3,12 +3,16 @@
 from django.conf import settings
 
 from dandiapi import __version__
-from dandiapi.api.views.info import schema_url
+from dandiapi.api.views.info import get_schema_url
 
 
 def test_rest_info(api_client):
     resp = api_client.get('/api/info/')
     assert resp.status_code == 200
+
+    # Get the expected schema URL
+    schema_url = get_schema_url()
+
     assert resp.json() == {
         'schema_version': settings.DANDI_SCHEMA_VERSION,
         'schema_url': schema_url,
diff --git a/dandiapi/api/tests/test_schema.py b/dandiapi/api/tests/test_schema.py
@@ -0,0 +1,36 @@
+from __future__ import annotations
+
+from dandischema.models import Asset, CommonModel, Dandiset, PublishedAsset, PublishedDandiset
+from dandischema.utils import TransitionalGenerateJsonSchema
+import pytest
+
+
+@pytest.mark.parametrize(
+    'model',
+    [
+        Dandiset,
+        Asset,
+        PublishedDandiset,
+        PublishedAsset,
+    ],
+)
+def test_schema_latest(api_client, model: CommonModel):
+    """Test that the schema endpoints return valid schemas."""
+    resp = api_client.get('/api/schemas/', {'model': model.__name__})
+    assert resp.status_code == 200
+
+    # Verify that the schema is json and has core properties
+    schema = resp.json()
+    assert isinstance(schema, dict)
+    assert 'properties' in schema
+    assert 'title' in schema
+
+    # Compare with expected schema from pydantic using same generator as dandischema
+    expected_schema = model.model_json_schema(schema_generator=TransitionalGenerateJsonSchema)
+    assert schema == expected_schema
+
+
+def test_schema_unsupported_model(api_client):
+    """Test that the schema endpoint returns an error when passed invalid choice."""
+    resp = api_client.get('/api/schemas/', {'model': 'NotAValidModel'})
+    assert resp.status_code == 400
diff --git a/dandiapi/api/views/__init__.py b/dandiapi/api/views/__init__.py
@@ -8,6 +8,7 @@
 from .info import info_view
 from .robots import robots_txt_view
 from .root import root_content_view
+from .schema import schema_list_view, schema_view
 from .stats import stats_view
 from .upload import (
     blob_read_view,
@@ -29,13 +30,11 @@
     'authorize_view',
     'blob_read_view',
     'info_view',
-    'info_view',
     'mailchimp_csv_view',
     'robots_txt_view',
-    'robots_txt_view',
     'root_content_view',
-    'root_content_view',
-    'stats_view',
+    'schema_list_view',
+    'schema_view',
     'stats_view',
     'upload_complete_view',
     'upload_initialize_view',
diff --git a/dandiapi/api/views/info.py b/dandiapi/api/views/info.py
@@ -1,17 +1,30 @@
 from __future__ import annotations
 
+from urllib.parse import ParseResult, urlencode, urlparse, urlunparse
+
 from django.conf import settings
+from django.urls import reverse
 from drf_yasg.utils import no_body, swagger_auto_schema
 from rest_framework import serializers
 from rest_framework.decorators import api_view
 from rest_framework.response import Response
 
 from dandiapi import __version__
 
-schema_url = (
-    'https://raw.githubusercontent.com/dandi/schema/master/'
-    f'releases/{settings.DANDI_SCHEMA_VERSION}/dandiset.json'
-)
+
+def get_schema_url():
+    """Get the URL for the schema based on current server deployment."""
+    scheme, netloc = urlparse(settings.DANDI_API_URL)[:2]
+    return urlunparse(
+        ParseResult(
+            scheme=scheme,
+            netloc=netloc,
+            path=reverse('schema-view'),
+            query=urlencode({'model': 'Dandiset'}),
+            params='',
+            fragment='',
+        )
+    )
 
 
 class ApiServiceSerializer(serializers.Serializer):
@@ -55,12 +68,12 @@ def __init__(self, *args, **kwargs):
     method='GET',
 )
 @api_view()
-def info_view(self):
+def info_view(request):
     api_url = f'{settings.DANDI_API_URL}/api'
     serializer = ApiInfoSerializer(
         data={
             'schema_version': settings.DANDI_SCHEMA_VERSION,
-            'schema_url': schema_url,
+            'schema_url': get_schema_url(),
             'version': __version__,
             'cli-minimal-version': '0.60.0',
             'cli-bad-versions': [],
diff --git a/dandiapi/api/views/schema.py b/dandiapi/api/views/schema.py
@@ -0,0 +1,60 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+from dandischema.models import Asset, Dandiset, PublishedAsset, PublishedDandiset
+from dandischema.utils import TransitionalGenerateJsonSchema
+from drf_yasg.utils import swagger_auto_schema
+from rest_framework import serializers
+from rest_framework.decorators import api_view
+from rest_framework.response import Response
+
+if TYPE_CHECKING:
+    from rest_framework.request import Request
+
+
+_model_name_mapping = {
+    m.__name__: m
+    for m in [
+        Dandiset,
+        Asset,
+        PublishedDandiset,
+        PublishedAsset,
+    ]
+}
+
+
+class SchemaQuerySerializer(serializers.Serializer):
+    model = serializers.ChoiceField(choices=list(_model_name_mapping))
+
+
+@swagger_auto_schema(method='GET', operation_summary='List schema models')
+@api_view(['GET'])
+def schema_list_view(request: Request) -> Response:
+    """Return the list of models which can be requested via the schema endpoint."""
+    return Response(_model_name_mapping.keys())
+
+
+@swagger_auto_schema(
+    method='GET',
+    operation_summary='Get model schema',
+    operation_description='Returns the JSON Schema of the requested metadata model',
+    query_serializer=SchemaQuerySerializer,
+)
+@api_view(['GET'])
+def schema_view(request: Request) -> Response:
+    """
+    Return the JSON Schema of the requested metadata model.
+
+    This endpoint returns the JSON Schema of the requested metadata model
+    as it is defined in this DANDI archive instance, with instance specific
+    parameters such as instance name and DOI prefix.
+    """
+    serializer = SchemaQuerySerializer(data=request.query_params)
+    serializer.is_valid(raise_exception=True)
+
+    # Generate the JSON schema using the same approach as dandischema
+    model_class = _model_name_mapping[serializer.validated_data['model']]
+    schema = model_class.model_json_schema(schema_generator=TransitionalGenerateJsonSchema)
+
+    return Response(schema)
diff --git a/dandiapi/urls.py b/dandiapi/urls.py
@@ -23,6 +23,8 @@
     mailchimp_csv_view,
     robots_txt_view,
     root_content_view,
+    schema_list_view,
+    schema_view,
     stats_view,
     upload_complete_view,
     upload_initialize_view,
@@ -66,6 +68,8 @@
     path('api/stats/', stats_view),
     path('api/info/', info_view),
     path('api/blobs/digest/', blob_read_view, name='blob-read'),
+    path('api/schemas/available/', schema_list_view, name='schema-list-view'),
+    path('api/schemas/', schema_view, name='schema-view'),
     path('api/uploads/initialize/', upload_initialize_view, name='upload-initialize'),
     re_path(
         r'api/uploads/(?P<upload_id>[0-9a-f\-]{36})/complete/',
diff --git a/doc/design/vendor-configurable-metadata-models-1.md b/doc/design/vendor-configurable-metadata-models-1.md
@@ -0,0 +1,154 @@
+# Vendor-Configurable Metadata Models
+
+This document illustrates the limitations originate from the metadata models in the DANDI ecosystem with
+[dandi-schema](https://github.com/dandi/dandi-schema) before version 0.12.0 and outlines the solution
+of vendor-configurable metadata models implemented across multiple components of the ecosystem,
+with the implementation starting and coordinated with dandi-schema at v0.12.0, to overcome these
+limitations.
+
+## Limitations
+
+The current metadata models and their uses in the DANDI ecosystem have the following limitations:
+
+1. We do not actually support or use multiple versions of the metadata models in dandi-archive.
+2. We use two representations of metadata models – Pydantic models and their respective JSON Schema derivatives – and
+   rely on an external process to generate the JSON Schema representation from the Pydantic models.
+3. We manually trigger updates of web frontend files according to a specific version of the JSON Schema representation of the models.
+4. We hardcode vendor-specific information inside the dandi-archive codebase (backend and frontend).
+5. Any vendor-specific configuration done at runtime in Pydantic models is not reflected in the JSON Schema representation used
+   by the web frontend since the web frontend uses a static versioned JSON Schema representation stored at
+   [dandi/schema](https://github.com/dandi/schema) that has been generated by the external process mentioned in point 2.
+
+```mermaid
+flowchart TD
+  %% repositories as grouped nodes
+  subgraph dandi_schema_repo["<a href='https://github.com/dandi/dandi-schema/'>dandi/dandi-schema</a>"]
+    Pydantic["Pydantic Models"]
+  end
+
+  subgraph schema_repo["<a href='https://github.com/dandi/schema/'>dandi/schema</a>"]
+    JSONSchema["JSON Schema<br>serializations"]
+
+  end
+
+  subgraph dandi_cli_repo["<a href='https://github.com/dandi/dandi-cli'>dandi-cli</a>"]
+    CLI["CLI & Library<br>validation logic<br/>(Python)"]
+  end
+
+  subgraph dandi_archive_repo["<a href='https://github.com/dandi/dandi-archive/'>dandi-archive</a>"]
+    Meditor["Web UI<br/>Metadata Editor<br/>(meditor; Vue)"]
+    API["Archive API<br/>(Python; DJANGO)"]
+    Storage[("DB (Postgresql)")]
+  end
+
+  %% main flow
+  Pydantic -->|"serialize into<br/>(CI)"| JSONSchema
+  Pydantic -->|used to validate| CLI
+  Pydantic -->|used to validate| API
+
+  JSONSchema -->|used to produce| Meditor
+  JSONSchema -->|used to validate| Meditor
+  Meditor    -->|submits metadata| API
+
+  CLI        -->|used to upload & submit metadata| API
+
+  API <-->|metadata JSON| Storage
+
+  %% styling
+  classDef repo   fill:#f9f9f9,stroke:#333,stroke-width:1px;
+  classDef code   fill:#e1f5fe,stroke:#0277bd,stroke-width:1px;
+  classDef ui     fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px;
+  classDef data   fill:#fff3e0,stroke:#e65100,stroke-width:1px;
+  JSONSchema@{ shape: docs }
+
+  class dandi_schema_repo,schema_repo,dandi_cli_repo,dandi_archive_repo repo;
+  class Pydantic,CLI,API code;
+  class JSONSchema,Storage data;
+  class Meditor ui;
+```
+> The diagram above depicts how metadata models are defined, represented, and used in the DANDI ecosystem before version 0.12.0 of dandi-schema.
+
+## Solution to Overcome Limitations
+
+The solution of vendor-configurable metadata model addresses the limitations outlined above by implementing the following changes:
+
+1. Make the metadata models vendor-configurable in dandi-schema.
+2. Create an API endpoint at `/api/schemas/` in dandi-archive to distribute dynamically generated JSON Schema representations of the metadata models
+   from the Pydantic models that define the metadata models at runtime.
+3. Update the `/api/info/` endpoint at dandi-archive
+   1. to point to the endpoint in point 2 for retrieving a JSON Schema representation of the metadata models
+      instead of a static versioned JSON Schema representation stored at [dandi/schema](https://github.com/dandi/schema).
+   2. to include vendor-specific information.
+4. Remove any hardcoded vendor-specific information in dandi-archive, and use the vendor-specific configuration
+   provided by dandi-schema.
+5. Have clients, such as the web UI (meditor), use the endpoints in point 2 and 3 to retrieve vendor-specific configurations
+   and JSON Schema representations of the metadata models that match the metadata models used by the particular instance of dandi-archive.
+
+
+```mermaid
+flowchart TD
+  %% repositories as grouped nodes
+  subgraph dandi_schema_repo["<a href='https://github.com/dandi/dandi-schema/'>dandi/dandi-schema</a>"]
+    Pydantic["Pydantic Models"]
+  end
+
+  subgraph schema_repo["<a href='https://github.com/dandi/schema/'>dandi/schema</a>"]
+    JSONSchema["JSON Schema<br>serializations"]
+
+  end
+
+  subgraph dandi_archive_repo["<a href='https://github.com/dandi/dandi-archive/'>dandi-archive</a>"]
+    Meditor["Web UI<br/>Metadata Editor<br/>(meditor; Vue)"]
+    API["Archive API<br/>(Python; DJANGO)"]
+    SchemaEndpoint["Schema API Endpoint<br/>(JSONSchema)"]
+    Storage[("DB (Postgresql)")]
+  end
+
+  subgraph dandi_cli_repo["<a href='https://github.com/dandi/dandi-cli'>dandi-cli</a>"]
+    CLI["CLI & Library<br>validation logic<br/>(Python)"]
+  end
+
+  %% main flow
+  Pydantic -->|"serialize into<br/>(CI)"| JSONSchema
+  Pydantic -->|used to validate| CLI
+  Pydantic -->|used to validate| API
+
+  API -->|serialize at runtime| SchemaEndpoint
+  SchemaEndpoint -->|used to produce| Meditor
+  SchemaEndpoint -->|used to validate| Meditor
+
+  Meditor    -->|submits metadata| API
+  CLI        -->|used to upload & submit metadata| API
+  API <-->|metadata JSON| Storage
+
+  %% styling
+  classDef repo   fill:#f9f9f9,stroke:#333,stroke-width:1px;
+  classDef code   fill:#e1f5fe,stroke:#0277bd,stroke-width:1px;
+  classDef ui     fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px;
+  classDef data   fill:#fff3e0,stroke:#e65100,stroke-width:1px;
+  JSONSchema@{ shape: docs }
+
+  class dandi_schema_repo,dandi_cli_repo,dandi_archive_repo repo;
+  class Pydantic,CLI,API,SchemaEndpoint code;
+  class JSONSchema,Storage data;
+  class Meditor ui;
+```
+> The diagram above depicts how metadata models are defined, represented, and used in the DANDI ecosystem starting with version 0.12.0 of dandi-schema.
+
+
+### Benefits
+
+This implementation provides several benefits:
+
+1. **Runtime Consistency**: The schema used by the frontend will always match the one used by the backend, including any vendor-specific configurations.
+2. **Simplified Deployment**: No need to manually update JSON Schema files or manage the [dandi-schema](https://github.com/dandi/dandi-schema) repository for storing
+   JSON Schema representations of the metadata models.
+3. **Future-Proofing**: The implementation allows for future support of multiple schema versions if needed.
+4. **Reduced Dependencies**: Removes the dependency on external GitHub URLs for schema definitions.
+
+### Immediate Steps and Longer-Term Opportunities
+
+The immediate implementation supports distributing, through an API endpoint, the JSON Schema representations of only the vendor-configurable metadata models currently being used
+by the particular instance of dandi-archive. However, the API endpoint has been structured to allow support for multiple versions of JSON Schema representations in the future if needed.
+
+Additionally, the JSON-LD context.json could also be similarly generated and served by the backend if needed in the future.
