How and when do we create rig.json metadata?

[alexpiet]

Following up on the discussion in the 05/21/2024 behavior meeting on the creation of rig.json metadata.

Requirements:

• We supply a rig.json for every behavior session across multiple different rigs (446, 447, 428, Ephys1/3)
• Must be flexible enough to handle diverse modalities, equipment, and changing hardware
• Must be easy to update when parts are replaced
• Must be generated automatically without manual intervention on all training computers (446/447)

Option 1 (implemented in #447)

• The task GUI will use information already provided in ForagingSettings.json, and Settings_box#.csv to create the rig.json metadata. It will check the saved version on the computer. If anything is out of date, the new version will be saved.
• If parts on the training rigs are updated, a limited amount of information can be provided in Settings_box#.csv, and the rig.json will be automatically updated on the next session
• If we change the way the rig.json is formatted, or add more details, then the repo code will automatically be updated, so unless there is a rig specific field that needs to be provided we will not need to manually update each computer
• This approach can include rig information for optogenetics and fiber photometry because those hardware pieces are consistent across rigs
• This implementation does not support the ephys rigs (although it could in theory). This decision was made because the ephys rigs are significantly more complex, and change much more frequently. Ephys users can set a flag in ForagingSettings.json that skips the rig.json generation, and they can supply their own (both ephys1 and ephys3 already have implementations)
• Advantages: easy maintenance for training rigs, ready to use now
• Disadvantages: does not support the ephys rigs

Option 2(proposed by Xinxin)

• We maintain a separate rig.json generation repository
• Each rig has a specification file (similar to Settings_box#.csv), which lists all the components
• The task GUI uses this separate repository to generate the rig.json and check if the current version is up to date
• Advantages: can support ephys rigs, and brings each ephys user's ad-hoc implementation into alignment.
• Disadvantages: Creating a repo that supports all rig types will be difficult, requires more maintenance, and coordination between many different users/projects. Updating the separate repo on each training computer requires more overhead or manual activity.

I prefer option 1 because I think the ephys rigs are significantly more complex, and are going to update much more frequently. However if option 2 is implemented well, it could work, and I don't oppose it as long as manual interventions/code updates to the training rigs are not required except at implementation. At the very least, I think we should merge in option 1 so we have rig metadata while we think about/develop option 2.

Thoughts?
@JeremiahYCohen @jsiegle @XX-Yin @hanhou @hagikent @galenlynch @KanghoonJ

[XX-Yin]

My main reason to support Option 2 is flexibility and independence-generating rig.json should be independent from the current behavior GUI. The advantage is that 1) it will make the code much easier to maintain and expand. 2) It's more flexible to generate rig metadata both for behavior and ephys rigs. 3) People can generate rig metadata without using the behavior GUI.

My suggestion is to combine Alex's current PR (#447) with my code (https://github.com/AllenNeuralDynamics/CTLUT-metadata-gen/blob/generate_rig_metadata/src/CTLUT_metadata_gen/generate_rig_metadata.py). Keep the check for changes section from Alex's code and use the build rig.json from my code.

If we agree on this combination, I can make some changes based on Alex's current PR.

[jsiegle]

I also support having a separate repo for generating rig metadata. I think it makes sense to keep this independent of the behavior GUI so it can be reused by multiple projects.

[hanhou]

If we choose Option 2, I think it makes sense to ask Sci Comp to develop and maintain such a separate repo with the metadata information we provide. Otherwise, it's hard to coordinate across different rigs. However, I don't know if this is acceptable in terms of time, since rig.json is now blocking our next steps using the AIND data upload pipeline.

[alexpiet]

I strongly advocate that we merge #447 in, which will also us to generate rig.json files immediately. If we want to pursue a stand alone repo, we can do that on a longer timescale. My code is future-proofed - its easy to simply swap the function call to a stand alone repo.

I'll point out that currently Xinxin's CTLUT_metadata package depends on the dynamic-foraging-task to parse water calibration files. So if the GUI is dependent on the CTLUT_metadata package it will create a circular dependency. Easy to remove, just pointing out the potential issue.

[XX-Yin]

"I'll point out that currently Xinxin's CTLUT_metadata package depends on the dynamic-foraging-task to parse water calibration files. So if the GUI is dependent on the CTLUT_metadata package it will create a circular dependency. Easy to remove, just pointing out the potential issue."

I don't see this is an issue. The water calibration path can an input variable.

[alexpiet]

I'm not talking about the path. Your code imports dynamic-foraging-task in order to use some functions that parse the water calibration file. If dynamic-foraging-task were to in turn import your code, that creates a circular dependency which can create unexpected behavior.

A imports B
B imports A

[XX-Yin]

I didn't expect these two packages to be used together in the previous code design, but this should be a simple fix to make "parse water calibration" independent.

[jsiegle]

There is already a repo maintained by Sci Comp for this, but it's not widely used: https://github.com/allenneuraldynamics/aind-metadata-mapper/

It sounds like there are three stages needed:

1. Merge Alex's PR so behavior sessions can have the metadata needed to start moving to the cloud
2. Extend the CTLUT_metadata package (and also rename that package) to make it easier to generate metadata for ephys + behavior sessions
3. Work with Sci Comp to figure out how to centralize metadata code into aind-metadata-mapper

[hagikent]

I support Option#1 for now, primarily because of the expected time-line (yes, we should proceed to transfer data to AWS before long). At least more than 24rigs in 446/447 (+428) are standardized/identical rigs. Integrated rig.json generation function might be less flexible, but I would say that having a dedicated function for more than 24rigs is already general enough.

[hanhou]

I'll go ahead and approve PR #447, and then let's work on Josh's steps 2 and 3.

[XX-Yin]

Let's first work on merging #447 given the urgency. And optimize it later.

[JeremiahYCohen]

I'm late to the party, but I think this warrants some in-person discussion. It's not clear to me how long it will take to implement Alex's option 2 (preferable long term, but we may need a short-term solution for now). I'll follow up.

[galenlynch]

In case we want to use these features, I extended the CTLUT metadata project to allow automatic scraping of open ephys settings files here: gcamp8-ephys-metadata-gen) (src/gcamp8_ephys_metadata_gen/passive_rigs.py is the relevant entry point).

I think it would be great to have a library that builds as much of the metadata as possible from files or other ways software can interrogate the rig. For the rest of the unconnected devices, @dyf suggested we all maintain our own scripts or something similar since the path to making a formalism to produce metadata can lead to madness.

I think hijacking a repo that works as is for Anna's CTLUT project is not the best approach. We should just make a new repo if we're going to make a new effort.

Strongly agree that metadata generation should be outside of the GUI, though for now the perfect shouldn't be the enemy of the good, and we should use the solution we have.

[KanghoonJ]

Considering the diverse aspects of ephys projects across different rigs, it seems more practical to have a separate repository dedicated to generating rig metadata for ephys rigs outside of the GUI. During our discussion about ephys metadata last month, it became clear that developing a common repository to handle rig metadata for different rigs presents significant challenges.

However, for training rigs in rooms 446 and 447, generating standardized rig metadata through the Foraging GUI would simplify maintenance and ensure consistency across various experiments, including optogenetics and photometry.