WiP: Branch with auto converted linkml model by yarikoptic · Pull Request #381 · dandi/dandi-schema

yarikoptic · 2026-03-10T19:46:27Z

to not be merged

The plan here

Establish branch linkml-auto-converted (TODO Here for @candleindark ) -- make link
git mv dandischema/{,orig_}models.py
Made ‘hatch’ script (you could add pydantic2linkml as dependency there) to convert orig_models.py into dandischema/models.yaml : hatch ... TODO
Translated the original models.py into dandischema/models.yaml and overlaid with an [dandischema/models_overlay.yaml] overlay file.
define model_instances.yaml (or alike) which would define pre-populated records such as standards (bids, nwb, ...). aim for potentially multiple classes there.
In branch ‘linkml-master’ you merge ‘linkml-auto-converted’ and introduce changes to linkml model to cover what is missing
There you produce pydantic model out of this patched model
Merge ‘linkml-master’ with its autoconverted model into ‘master’ where you have also tuneups to pydantic if needed.
Where possible keep additional checks in some extra file which would tune auto-converted model

yarikoptic

initial review pointers

dandischema/models.yaml

yarikoptic · 2026-03-10T20:10:58Z

dandischema/models.yaml

+  contributor:
+    name: contributor
+    notes:
+    - 'pydantic2linkml: Warning: The translation is incomplete. Tagged union types


for all 3 of those (present in 4 spots in pydantic model) translate to use linkml's designates_type

See https://claude.ai/share/66fe0f31-8cf1-40e6-b9fd-47c6719f2006

(Self)Note: This is an issue regarding translation of discriminated unions in Pydantic models.

dandischema/models.yaml

yarikoptic · 2026-03-10T20:31:38Z

dandischema/models.yaml

+        name: identifier
+        range: string
+        required: true
+        pattern: ^(?:urn:uuid:)?[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?4[0-9a-fA-F]{3}-?[89abAB][0-9a-fA-F]{3}-?[0-9a-fA-F]{12}$


per discussion with @candleindark and claude -- it seems we should be able to use types here. Here is an example excerpt from https://claude.ai/share/66fe0f31-8cf1-40e6-b9fd-47c6719f2006

types: ORCID: uri: xsd:string base: str pattern: "^https://orcid\\.org/\\d{4}-\\d{4}-\\d{4}-\\d{3}[0-9X]$" description: An ORCID identifier ISSN: uri: xsd:string base: str pattern: "^\\d{4}-\\d{3}[0-9X]$" description: An ISSN identifier UUID: uri: xsd:string base: str pattern: "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$" description: A UUID slots: contributor_id: range: ORCID publication_id: range: ISSN asset_id: range: UUID

The definitions in dandischema.models doesn't define those types with patterns information attached. Those types are actually defines as alias of str

dandi-schema/dandischema/models.py

Lines 714 to 718 in 1deb319

Identifier = str

ORCID = str

RORID = str

DANDI = str

RRID = str

.

Because of that, I don't think these type definitions should be part of the result of auto translation but added manually something to add to the result of the auto translation.

yarikoptic · 2026-03-10T20:55:10Z

dandischema/models.yaml

+  genotype:
+    name: genotype
+    notes:
+    - 'pydantic2linkml: Warning: The translation is incomplete. The union core schema


seems "obvious" -- union of a list[object] vs str . Should be doable now.

This kind of union can now be expressed in LinkML using any_of as demo in #257 (comment).

yarikoptic · 2026-03-10T20:58:07Z

dandischema/models.yaml

+    notes:
+    - 'pydantic2linkml: Unable to translate the logic contained in the wrap validation
+      function, <function _BaseUrl.__get_pydantic_core_schema__.<locals>.wrap_val
+      at 0x7f8c96c36f80>.'


this is for List[AnyHttpUrl] and @candleindark says that there is custom validator attached. May be we could just provide custom type for validation here.

codecov · 2026-03-10T22:07:26Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.92%. Comparing base (4b89e4f) to head (bd38142).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #381   +/-   ##
=======================================
  Coverage   97.92%   97.92%           
=======================================
  Files          18       18           
  Lines        2405     2405           
=======================================
  Hits         2355     2355           
  Misses         50       50

Flag	Coverage Δ
unittests	`97.92% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

dandischema/models.yaml

pyproject.toml

Specify Hatch-managed env for auto converting `dandischema.models` to LinkML schema and back to Pydantic models

Provide script to translate `dandischema.models` in to a LinkML schema and overly it with definition provided by an overlay file.

Provide script to translate `dandischema/models.yaml` back to Pydantic models and store them in `dandischema/models.py`

=== Do not change lines below === { "chain": [], "cmd": "hatch run linkml-auto-converted:2linkml", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [], "pwd": "." } ^^^ Do not change lines above ^^^

These prefixes are copied from https://github.com/dandi/schema/blob/master/releases/0.7.0/context.json

…_orig for now

…tions Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

There is no prefix defined as `dandi_default`. The intended default prefix is `dandi`

… models.yaml The previous BRE pattern used `\+` (GNU sed extension) which silently fails on macOS BSD sed. Switch to `-E` (extended regex) with POSIX character class `[^[:space:]]` instead of `\S` (also unsupported by BSD sed), making the normalization work on both macOS and Linux. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

yarikoptic commented Mar 10, 2026

View reviewed changes

candleindark force-pushed the linkml-auto-converted branch from fa28737 to c78f9d3 Compare March 10, 2026 23:08

yarikoptic commented Mar 12, 2026

View reviewed changes

dandischema/models.yaml Show resolved Hide resolved

This was referenced Mar 12, 2026

Duplicate Publishable attributes in CommonModel + DOI in Dandiset as Optional #285

Closed

Migration to LinkML from Pydantic #382

Open

yarikoptic commented Mar 13, 2026

View reviewed changes

pyproject.toml Show resolved Hide resolved

candleindark and others added 5 commits March 13, 2026 17:01

build(hatch-env): specify env for conversion to LinkML and back

cffee39

Specify Hatch-managed env for auto converting `dandischema.models` to LinkML schema and back to Pydantic models

build(hatch-env): provide script to translate dandischema.models

b5644c1

Provide script to translate `dandischema.models` in to a LinkML schema and overly it with definition provided by an overlay file.

build(hatch-env): provide script to translate dandischema/models.yaml

cfe3bea

Provide script to translate `dandischema/models.yaml` back to Pydantic models and store them in `dandischema/models.py`

[DATALAD RUNCMD] Auto-convert the model

21e7e9c

=== Do not change lines below === { "chain": [], "cmd": "hatch run linkml-auto-converted:2linkml", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [], "pwd": "." } ^^^ Do not change lines above ^^^

Now we will sanitize the converted model addresses

f109ff4

yarikoptic force-pushed the linkml-auto-converted branch from 28c5b44 to bd38142 Compare March 13, 2026 21:17

yarikoptic and others added 9 commits March 13, 2026 18:32

more sanitization

7580f80

More Sanitization applied

f130f4f

Reextracted model (now less clutter and with spaces)

0a8ef6b

feat: provide prefixes definition in the overlay file

63dd301

These prefixes are copied from https://github.com/dandi/schema/blob/master/releases/0.7.0/context.json

Added linkml: prefix and sorted all the entries

f8125d0

Provide default prefix to be custom dandi_default

7c5077a

Reextracted model (now with prefixes and deterministic order of notes)

3485245

rename original models into models_orig

bff037c

Adding models.py back to import from converted and some symbols from …

6bb4b52

…_orig for now

yarikoptic force-pushed the linkml-auto-converted branch from bd38142 to 6bb4b52 Compare March 13, 2026 22:33

candleindark and others added 3 commits March 15, 2026 22:40

Expand comment for linkml-auto-converted hatch env with usage instruc…

725e957

…tions Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix: correct default_prefix to dandi

a2324fa

There is no prefix defined as `dandi_default`. The intended default prefix is `dandi`

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WiP: Branch with auto converted linkml model#381

WiP: Branch with auto converted linkml model#381
yarikoptic wants to merge 17 commits intomasterfrom
linkml-auto-converted

yarikoptic commented Mar 10, 2026 •

edited by candleindark

Loading

Uh oh!

yarikoptic left a comment

Uh oh!

Uh oh!

yarikoptic Mar 10, 2026

Uh oh!

candleindark Mar 16, 2026

Uh oh!

Uh oh!

yarikoptic Mar 10, 2026

Uh oh!

candleindark Mar 12, 2026

Uh oh!

yarikoptic Mar 10, 2026

Uh oh!

candleindark Mar 16, 2026

Uh oh!

yarikoptic Mar 10, 2026

Uh oh!

codecov bot commented Mar 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

yarikoptic commented Mar 10, 2026 • edited by candleindark Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yarikoptic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yarikoptic Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

candleindark Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yarikoptic Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

candleindark Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

yarikoptic Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

candleindark Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

yarikoptic Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yarikoptic commented Mar 10, 2026 •

edited by candleindark

Loading

codecov bot commented Mar 10, 2026 •

edited

Loading