Add support for ASDF serialisation and deserialisation #776

Cadair · 2024-11-06T09:57:03Z

This PR adds support for saving and loading the vast majority of ndcube objects to/from ASDF files.

Thanks to @ViciousEagle03 who did the majority of this work.

This PR is now ready for review, to save every possible ndcube object there's still two PRs to asdf-astropy which need to be merged, some tests are skipped here for that. However, this is now ready for review.

ndcube/asdf/resources/schemas/global_coords-1.0.0.yaml

ndcube/asdf/resources/schemas/extra_coords-1.0.0.yaml

ndcube/asdf/resources/schemas/multipletablecoordinate-1.0.0.yaml

ndcube/asdf/converters/ndcube_converter.py

ndcube/asdf/resources/schemas/compoundwcs-1.0.0.yaml

ndcube/asdf/resources/schemas/ndcollection-1.0.0.yaml

ndcube/asdf/resources/schemas/ndcube-1.0.0.yaml

ndcube/asdf/resources/schemas/quantitytablecoordinate-1.0.0.yaml

ndcube/asdf/resources/schemas/reorderedwcs-1.0.0.yaml

ndcube/asdf/resources/schemas/resampledwcs-1.0.0.yaml

ndcube/asdf/resources/schemas/skycoordtablecoordinate-1.0.0.yaml

ndcube/asdf/resources/schemas/timetablecoordinate-1.0.0.yaml

pyproject.toml

ndcube/asdf/resources/schemas/ndcollection-1.0.0.yaml

braingram

Thanks for pushing this forward. Overall I think it looks great. I left a few comments:

1 possible bug in the collection converter (which might instead be a schema bug where aligned_axes should be required)
a few comments about schema required items vs values filled in by converter

I mostly did not review the test changes.

braingram · 2025-10-07T14:31:00Z

docs/explaining_ndcube/asdf_serialization.rst

+  >>> with asdf.AsdfFile(tree=my_tree) as f:  # doctest:  +SKIP
+  ...     f.write_to("somefile.asdf")  # doctest:  +SKIP


Suggested change

>>> with asdf.AsdfFile(tree=my_tree) as f: # doctest: +SKIP

... f.write_to("somefile.asdf") # doctest: +SKIP

>>> asdf.AsdfFile(tree=my_tree).write_to("somefile.asdf"): # doctest: +SKIP

No need for the with here since the AsdfFile instance isn't holding onto a file.

braingram · 2025-10-07T14:38:51Z

ndcube/asdf/resources/manifests/ndcube-1.0.0.yaml

Minor nitpick but maybe order the tags alphabetically? That way it's easier to:

find a tag definition

match tag definitions to other things (like listed schema files)

braingram · 2025-10-07T14:49:45Z

ndcube/asdf/resources/schemas/extra_coords-1.0.0.yaml

+  ndcube:
+    tag: "tag:sunpy.org:ndcube/ndcube-1.*"
+
+required: [ndcube]


It looks like the converter always writes dropped_tables should this be required?

👍 the converter is wrong

braingram · 2025-10-07T14:59:28Z

ndcube/asdf/converters/ndcollection_converter.py

+    def from_yaml_tree(self, node, tag, ctx):
+        from ndcube.ndcollection import NDCollection
+
+        aligned_axes = list(node.get("aligned_axes").values())


Suggested change

aligned_axes = list(node.get("aligned_axes").values())

aligned_axes = list(node.get("aligned_axes").values())

Won't this raise an exception if "aligned_axes" is missing?

braingram · 2025-10-07T15:40:43Z

ndcube/asdf/resources/schemas/quantitytablecoordinate-1.0.0.yaml

+  physical_types:
+    type: array
+
+required: ["table", "unit"]


It looks like the converter always writes "mesh". Should this be required or perhaps "mesh" conditionally written (since the from_yaml_tree uses get to access it).

braingram · 2025-10-07T15:42:47Z

ndcube/asdf/resources/schemas/reorderedwcs-1.0.0.yaml

+  world_order:
+    type: array
+
+required: [wcs]


Similar to the other comments, the converter appears to always write "pixel_order" and "world_order". Should these be required?

braingram · 2025-10-07T15:43:40Z

ndcube/asdf/resources/schemas/resampledwcs-1.0.0.yaml

+  offset:
+    tag: "tag:stsci.edu:asdf/core/ndarray-1.*"
+
+required: [wcs]


Similar to other comments. Should factor and offset be required?

braingram · 2025-10-07T15:44:32Z

ndcube/asdf/resources/schemas/skycoordtablecoordinate-1.0.0.yaml

+  physical_types:
+    type: array
+
+required: ["table"]


Similar to other comments, should mesh be required?

braingram · 2025-10-07T15:45:28Z

ndcube/asdf/resources/schemas/timetablecoordinate-1.0.0.yaml

+  reference_time:
+    tag: "tag:stsci.edu:asdf/time/time-*"
+
+required: ["table"]


Similar to the other comments, should reference_time be required?

DanRyanIrish

More review to follow

DanRyanIrish · 2025-10-07T17:54:43Z

docs/explaining_ndcube/asdf_serialization.rst

+*************************
+
+:ref:`asdf` is an extensible format for validating and saving complex scientific data along with its metadata.
+`ndcube` provides schemas and converters for all the ND objects (`~ndcube.NDCube`, `~ndcube.NDCubeSequence` and `~ndcube.NDCollection`) as well as for various WCS and table objects required by them.


Suggested change

`ndcube` provides schemas and converters for all the ND objects (`~ndcube.NDCube`, `~ndcube.NDCubeSequence` and `~ndcube.NDCollection`) as well as for various WCS and table objects required by them.

`ndcube` provides schemas and converters for all the ND objects (`~ndcube.NDCube`, `~ndcube.NDCubeSequence`, `~ndcube.NDCollection` and `~ndcube,NDMeta`) as well as for various WCS and table objects required by them.

DanRyanIrish

I didn't see any problems that jumped out. So as long as the tests pass, and those more qualified to review this also approve, I'm happy to see this merged

DanRyanIrish · 2025-10-08T13:26:50Z

ndcube/asdf/converters/compoundwcs_converter.py

+    types = ["ndcube.wcs.wrappers.compound_wcs.CompoundLowLevelWCS"]
+
+    def from_yaml_tree(self, node, tag, ctx):
+        from ndcube.wcs.wrappers import CompoundLowLevelWCS


Why are imports inside from_yaml... methods through PR?

Because these files are imported at import asdf time so by moving them into these methods these imports are only triggered when actually reading a file, i.e. if you read a non-ndcube asdf these imports are never run.

See https://www.asdf-format.org/projects/asdf/en/latest/asdf/extending/extensions.html#entry-point-performance-considerations

ViciousEagle03 and others added 30 commits November 6, 2024 09:25

initial commit

ac36722

version alignment

6be61e7

Remove method to remove any delay and

6e619fc

Update tag and

f505703

enable schema test

af2f9c7

revert small change

adfc6b3

Add the converters for ExtraCoords and TimeTableCoordinate class

78eeb53

Add ExtraCoords and TimeTableCoord Schema

0804f1d

Update manifest and entry_point.py

0f4a86f

Add the QuantityTableCoordinate and SkyCoordTableCoordinate converters

86bfac9

Add the schema for QuantityTableCoordinate and SkyCoordTableCoordinate

da1c14e

Update manifest and entry_point.py

dc24d40

Add validation for schema and manifests

ef208ed

pre-commit

c3d2606

Update the tox.ini and CI workflow

86c2b6c

Update schemas

3a875ab

Add the converter and schema for GlobalCoords

ce5a14b

Update converters

9247b46

Update entry_points,schemas and manifest

c8a1bc4

minor change

c681cb2

lowercase schema URIs and use wildcards

42c590e

Add tests and GWCS objects

7f3a81b

apply suggestions from code review

7622950

Remove mesh as a property validator

f89e195

Update the dependencies version

8b519ae

Style changes and add warnings to NDCube converter

37bf563

env name update

693e63e

Add asdf as an optional dep

de6944e

Remove asdf as an optional dependency

22a839e

Add support for meta in converter and schema

ec863ad

Cadair commented May 27, 2025

View reviewed changes

ndcube/asdf/resources/schemas/global_coords-1.0.0.yaml Outdated Show resolved Hide resolved

Cadair commented May 27, 2025

View reviewed changes

ndcube/asdf/resources/schemas/extra_coords-1.0.0.yaml Outdated Show resolved Hide resolved

ndcube/asdf/resources/schemas/extra_coords-1.0.0.yaml Outdated Show resolved Hide resolved

Cadair commented Jun 2, 2025

View reviewed changes

pyproject.toml Show resolved Hide resolved

Cadair commented Jun 2, 2025

View reviewed changes

pyproject.toml Show resolved Hide resolved

Cadair added 2 commits June 2, 2025 10:35

Apply suggestions from code review

b56c702

Only add meta, global and extra to ndcube if populated

0762b8d

Cadair commented Jun 3, 2025

View reviewed changes

ndcube/asdf/resources/schemas/ndcollection-1.0.0.yaml Show resolved Hide resolved

Cadair added 2 commits June 3, 2025 17:01

More lax schemas

eb0c6f3

no property order

2559057

Cadair mentioned this pull request Jul 2, 2025

Support for level 2 data DKISTDC/dkist#567

Draft

4 tasks

Cadair added 10 commits August 20, 2025 15:35

Merge branch 'main' of github.com:sunpy/ndcube into asdf-support

c9b4611

Fix tag URIs

ff4b79f

Merge branch 'main' into asdf-support

fa41ede

Exclude all asdf files from import-at-top ruff

3ead8ad

Attempt to fix ndcube sequence schema

b467dab

Finesse some deps

771c1a2

Fix tests with dev asdf-astropy

a59c901

Merge branch 'main' of github.com:sunpy/ndcube into asdf-support

a66756e

Fix some tests requiring dev asdf-astropy

cc8d915

More tests

504f230

Cadair requested review from DanRyanIrish and braingram October 7, 2025 14:06

Try this

ba8e495

braingram reviewed Oct 7, 2025

View reviewed changes

DanRyanIrish reviewed Oct 7, 2025

View reviewed changes

DanRyanIrish reviewed Oct 8, 2025

View reviewed changes

DanRyanIrish approved these changes Oct 9, 2025

View reviewed changes

Cadair added 2 commits October 9, 2025 15:59

Schema updates

b720b8a

Merge branch 'main' of github.com:sunpy/ndcube into asdf-support

b9988e6

		>>> with asdf.AsdfFile(tree=my_tree) as f: # doctest: +SKIP
		... f.write_to("somefile.asdf") # doctest: +SKIP

	>>> with asdf.AsdfFile(tree=my_tree) as f: # doctest: +SKIP
	... f.write_to("somefile.asdf") # doctest: +SKIP
	>>> asdf.AsdfFile(tree=my_tree).write_to("somefile.asdf"): # doctest: +SKIP

	aligned_axes = list(node.get("aligned_axes").values())
	aligned_axes = list(node.get("aligned_axes").values())

	`ndcube` provides schemas and converters for all the ND objects (`~ndcube.NDCube`, `~ndcube.NDCubeSequence` and `~ndcube.NDCollection`) as well as for various WCS and table objects required by them.
	`ndcube` provides schemas and converters for all the ND objects (`~ndcube.NDCube`, `~ndcube.NDCubeSequence`, `~ndcube.NDCollection` and `~ndcube,NDMeta`) as well as for various WCS and table objects required by them.

Uh oh!

Add support for ASDF serialisation and deserialisation #776

Are you sure you want to change the base?

Add support for ASDF serialisation and deserialisation #776

Uh oh!

Conversation

Cadair commented Nov 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

braingram left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DanRyanIrish left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DanRyanIrish left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Cadair commented Nov 6, 2024 •

edited

Loading

braingram left a comment •

edited

Loading