What reference to use for dataset organization

[bruno-f-cruz]

The previous version of aind-data-schema was opinionated on how schemas should be logged to the dataset. This information seems to no longer exist in the most recent docs (or maybe I missed it?)

https://github.com/AllenNeuralDynamics/aind-data-schema/blob/v1.4.0/docs/source/data_organization.rst#primary-data-conventions

Is there a new reference we can use or should we assume that everything is up to the user and resolved at the aind-data-transfer step?

[dougollerenshaw]

@dbirman, tagging you for visibility.

[dbirman]

We removed documentation that wasn't about aind-data-schema itself.

@dyf was supposed to move the conventions here: https://docs.allenneuraldynamics.org/en/latest/metadata/data_conventions.html -- although as you can see that page is currently out of date.

In the absence of updated documentation you should not assume everything is up to the user, but rather assume everything remains the same as before :)

That said, we are actually allowing users to point to different data and metadata paths in the next release of the transfer service. Still best to keep them together for simplicity.

[bruno-f-cruz]

Right now that happens on the data-transfer-service which is bad
Why is this worse than occurring at the rig? The rig is far more resource-constrained than VAST. With our current data volumes, if metadata fails to be created for more than a day, chances are that no further data can be collected in that machine.

hides issues with the metadata service and allows people to upload all sorts of junk.
I am not sure I understand why this would be any different if the merging were to be done with data in VAST. It's mostly an issue where the "junk" gets accumulated. Can you give me an example of a current issue that fits in this category?

As far as installation, the mapper will be a pip installable package with minimal dependencies (only aind-data-schema, ideally). All the custom logic needed for specific types of data assets will be separated out of the mapper and moved into packages specific to those assets. All of this can then be run in an isolated environment or docker image to keep from polluting the rig itself.

Once again, if it works like this, I don't see the issue with running it on VAST. I am also not sure how this solution will generalize to data produced from distinct machines.

At this point, I think we should keep operating under the assumption that each modality will create its own metadata and "something" will take care of it automatically. In the meantime, I will wait for documentation on what names those files should be named so your tool can ingest them.

[dbirman]

x_<anything>.json will be fine, I will set the mapper up to handle that. The documentation for that will go in the metadata-mapper repo readme once it's ready to be used.

To the other points: an average simple metadata error takes me a half day to fix, complex errors can take an entire day or more. Doug has spent entire weeks tracking down and fixing very complex errors. This is a huge waste of engineering time. Many of these errors were almost trivial. If someone had caught them at the time the asset was generated we could probably have fixed them in 30 minutes or less and they would have been solved permanently. For example, there are many assets where the "rig_id" is not the same in the rig and session, something that often happens if a session file gets re-used on different rigs, for example (or vice versa, rig.json gets copied to a second rig). This is super quick fix at the time of asset generation, but after the asset is created it becomes incredibly difficult to figure out which rig_id is the real one. Just one example, there are many more of varying complexity.

As long as metadata is validated after the data gets handed off there's no mechanism for users to see these issues. It would also be fine if the validation process were run independently and errors were sent to users and that's something we're discussing with SIPE. So in the long run I think that will be where this goes, in the near future we need this to happen on the rigs until people are generating metadata of sufficient quality to support downstream tooling (qc portal, data explorer, analysis pipeline, etc)

So all that to say, we've been listening to everything people have brought up about this plan and this is the compromise we're going with for now. If I didn't say it already it's probably important to add that the mapper will continue to not be a blocking process, same as how users can currently send garbage metadata to the transfer-service and it will happily accept it. Everybody will still be able to upload data with invalid metadata, we're not going to slow down data collection for this. Now though you'll be specifically choosing to ignore error messages and continue when you do that and we want that to prompt people to reach out for help (whereas previously it was all hidden and impossible to even know there were issues).

[dbirman]

Small update here: Patrick Latimer and I will pitch a plan to sci comp / SIPE for shifting the mapper off the rigs in the long term. The key capability we're missing to accomplish this is the ability to pre-validate metadata prior to launching an experimental session. There's a lot of steps between where we are now and that capability but I'm optimistic that we can move toward it over the next few months, especially with the migration away from SLIMS.

[bruno-f-cruz]

That's great news!
I am not sure what you have in mind. It sounds like you want to provide data-mappers from something that would exist in dataverse and maintained elsewhere. I would try to approach this in a much more holistic way like I did 2 years ago. By using aind-data-schema as not only descriptive metadata, but instead the language we use to specify experiments...
I will leave here my 2 cents and experience trying to do something similar in the past :P

Issues

• I need a way to define the input to a rig configuration
• SciComp needs a way to keep track of metadata
• Metadata should be as "ground-truth" as possible and self-validating
• We interact with different tools, so definitions should be standard across them
• We need a way to store (e.g. database) settings for the full instrument, but also for single devices

Attempted solution

• aind-data-schema already provides a vocabulary for devices. The idea is to use that vocabulary but, instead of assembling it into a monolithic rig.json, compose new rig configuration schemas from individual devices.
• By adopting the same vocabulary data-mapping is not only solved, but it is essentially not a problem to begin with. If everyone refers to a camera the same way, little, if anything, has to be mapped
• Given a json-schema compiled by pydantic, I can easily automate the generation of C# code to derive experimental schemas. This means that metadata is no longer a retrospective description of the experiment and instead documents what was used as input to run the experiment itself.
• The schema is effectively a DSL that users interact with from the python side via pydantic. Since json-schema defines an interoperable contract inputs to the task become self-documenting, self-validating and much easier to maintain.

Why it failed

• At the time aind-data-schema was changing too often and made it impossible to stabilize a target for vocabulary
• aind-data-schema was not meant to be used as vocabulary (i.e. in parts) and instead only as a "whole". Some patterns (e.g. lossy deserialization from children classes) made it really difficult to use individual device classes in models outside the ones defined in the main library
• Interoperability was never a priority. Despite several attempts to raise awareness/fixes to some of the patterns used in this library that prevent easy interop with other json-schema tools, it was never really a use case that was entertained. This can be seen in issues like Lists vs Dictionaries when using List[Model] #609, Modeling Decimal types as json-schema results in ambiguous typing #710, Refactor discriminated unions to ensure references to original discriminator are preserved #731 and Current Enum-like Union pattern does not provide type hinting #843 to name a few.

Current status

• A small library maintained by me (https://github.com/AllenNeuralDynamics/Aind.Behavior.Services/tree/main/src/aind_behavior_services/rig) provides vocabulary to define experiments. This vocabulary can be used by Vrforaging (https://github.com/AllenNeuralDynamics/Aind.Behavior.VrForaging/blob/4f194d22fae91f8108ab26cecba19527b9f19485/src/aind_behavior_vr_foraging/rig.py#L37) or any other task/pipeline(https://github.com/AllenNeuralDynamics/Aind.Behavior.IsoForce/blob/1f550e72acd9577f66bab22cc2d0ae9a582605cb/src/aind_behavior_iso_force/rig.py#L35). This has huge benefits at the level of generating data acquisition code, logging and metadata.
• Bonsai.sgen takes each one of the json-schema generated by the pydantic model and maps it to c#/bonsai. E.g. python -> json-schema -> CSharp
• The whole thing is completely automated and runs as part of my CICD for every single behavior repo (https://github.com/AllenNeuralDynamics/Aind.Behavior.GitHubActions/blob/7b1f9bac50aee9088017ab7e1a3e4463337c5703/.github/workflows/test-aind-behavior-framework.yml#L71-L78)
• We are in the process of storing these models in a database (dataverse, zookeeper, postgres, or tbd ?) with the support of SIPE. The idea is that users will be able to push their "calibrated devices" or "rig configuration" and at the rig we will simply fetch them as we need.
• We use our models and create mappers to aind-data-schema (this is a huge cost that I would love to get rid of since it feels like a "ORM-esque" exercise).

[patricklatimer]

I also support Bruno's proposal as complementary to my own, as it's certainly clean and seems beneficial from a repeatability point of view.

I think that no matter what, we're going to have prescriptive and descriptive metadata, the former used to set up and carry out experiments, and the latter used to explore and analyze data after it's acquired. Expanding aind-data-schema to cover everything (which is I think what Bruno is proposing) is a good idea IMO, but it's not my call as it's not my problem to maintain it.

Here's some things that I don't think are included in data-schema, but are nonetheless necessary to run acquisitions.

• Internal service urls/ports
• secrets and credentials
• hardware details like which COM port a device is on

Regardless of whether we put work into expanding data-schema, we're currently having to set up data tables and APIs in our new homebrewed LIMS hosted in Dataverse. Those are going to have to map into data-schema. We should shoot for as simple a mapping as can be accomplished, either 1-1 fields or a superset of data-schema fields such that mapping is just dropping the extra keys.

Additionally (and this is the focus of my opinions) this mapping should happen in infrastructure, not at the rigs. Acquisitions should be treated like black box functions where prevalidated instructions get sent, and data is received.

All the rig should be responsible for is initiating a session (which creates a mostly-filled-out Session object) via a simple handshake including the Experimenter name and the current time. Then it has to put its data somewhere according to the instructions it received.