Tests for device output expected by downstream processing

[DiamondJoseph]

There previously were tests for the DCM to ensure that its output was compatible with downstream processing (e.g. the NeXus service). Due to ophyd-async changes these tests were disabled, then changes to consolidate the DCM across multiple beamlines made changes to the output of the device.

It's not uncommon for "breaking changes" of modifying the output signal names of a device to be "fixed" by modifying the tests, as sometimes the tests are just to increase coverage rather than denoting a specific desired behaviour.

How can we mark/monitor/maintain tests for devices that are required not to change?
When consolidating multiple similar devices, how much should we push for downstream processing to change vs. allowing changes within dodal?
Are breaking changes to device outputs sufficient to require a breaking change in dodal version, which should be passed as part of the metadata of a scan and checked by downstream services?

When output of bluesky is captured by a document store, processing like writing a NeXus file can be re-run once the processing has been updated, but currently data is converted to a processed form and the original is lost.

[callumforrester]

A related question, how do we capture all of the requirements across all beamlines? A DCM should provide the required metadata for everyone's nexus implementation, for example.

[stan-dot]

CODEOWNERs per devices/beamlin that exhausts all the beamlines?

codeowners with an email there

[callumforrester]

Would prefer an automatic check if possible

[stan-dot]

we'd need a nexus definition for all beamlines. that is the key node then. how do we get from nexus definitions into the tests and how do we get an up-to-date nexus definition ?

those are the two next questions

[DiamondJoseph]

Annoyingly it's not quite a NeXus definition per beamline, but potentially as many as per-plan.
Probably closer to on the order of known ApplicationDefinitions https://manual.nexusformat.org/classes/applications/index.html#application-definitions See also their xml schemas https://github.com/nexusformat/definitions/tree/main/applications.

And then a mapping per beamline to which fields should be in their chosen ApplicationDefinition(s)

[stan-dot]

clarification query: those per-plan would be constant across time per-techanique? that is can we reuse across beamlines?

who's doing the custody of application definitions and how do they get into the nexus file writer? we could have a simple REST api with an S3 bucket centrally to compare those definitions

[DominicOram]

required not to change?
how much should we push for downstream processing to change

I think we always push for downstream to change. Particularly right now while there is a small downstream ecosystem. My concern is if we're not careful we end up with a GDA like system where everyone is scared to change anything because it might break downstream and so they make a copy, with slightly different behavior...

Specifically in this case there are two things that would solved it:

• There should be an end-to-end system test that wrote a nexus file with the DCM and confirmed it had the right data in it
• It should be clear in the test name/comment why the test is important. e.g. test_dcm_has_sufficient_metadata_for_nexus_writing_on_i22. test_reading tells me nothing as to what is actually important about the reading