Re-organize QualityControl.tags as a dict

[dbirman]

The current implementation of tags: list[str] doesn't work the way we intended it to. We wanted people to tag metrics with things like ["probeA", "motion-correction"], ["probeB", "spike-sorting"], etc. In practice in the QC portal this ends up being a problem, because we can't "see" that "probeA" and "probeB" are the same "thing" and can be used to hierarchically group metrics.

What we really want is to let users group the metrics in a hierarchy, where for example if they choose the grouping elements to be:

• probe
  • metric-type

Then what they see in the portal is:

• ProbeA
  • Motion Correction
  • Spike Sorting
  • Raw LFP
  • etc...
• ProbeB
  • Motion Correction
  • Spike Sorting
  • Raw LFP
  • etc...

The hierarchy would be user-configurable to let them look at motion correction across all probes, etc.

David and I discussed re-organizing tags as key:value pairs in a dictionary to make this possible. So users would need to give us a dictionary like

{
  "probe": "ProbeA",
  "metric-type": "Motion Correction"
}

To get the structure I'm showing above. On the plus side we're catching this before a lot of QC has been run in v2, unfortunately this is a breaking change. I don't know how to make this non-breaking unfortunately. For metrics that only have a single tag you can convert them to look like:

{
  "tag": "<old-tag",
  "name": "name"
}

Which then allows you to group by tag and then name hierarchically to get the same view that we currently have.

For the upgrader this is a tiny bit easier, we can generate the keys from the evaluations and the type from the metric names so that at least you can still get the hierarchy that used to show up in the original portal:

• evaluation
  • metric

[dbirman]

Flagging this for @alejoe91 @seanmcculloch @arjunsridhar12345 @arielleleon all of you have been making QC recently. Does this change make sense to you and does this clarify what we intended the tags to be for? If we made this change would it be expensive to re-run QC on all the assets that you've generated with V2 metadata so far?

[alejoe91]

Flagging this for @alejoe91 @seanmcculloch @arjunsridhar12345 @arielleleon all of you have been making QC recently. Does this change make sense to you and does this clarify what we intended the tags to be for? If we made this change would it be expensive to re-run QC on all the assets that you've generated with V2 metadata so far?

Thanks @dbirman
That makes sense to me and it would be straightforward to implement on the ecephys side. Also, we haven't pushed the 2.0 version of the pipeline to prod yet, so we won't need to regenerate metadata on our end.

[arjunsridhar12345]

@dbirman this makes sense. Not sure how complicated it would be for VR-foraging but I think its mainly grouped by harp device right now. I think the VR-foraging only V2 pipeline has not been hit yet, but its pretty cheap and costs a few cents so rerunning should be good to go

[seanmcculloch]

@dbirman
Ophys is still using V1. No need to reprocess at this point.

To clarify: If we submitted dicts like your example

{
  "probe": "ProbeA",
  "metric-type": "Motion Correction"
}

How would one choose the hierarchy between

• probe
• • metric-type

and

• metric-type
• • probe

Is this per QC Portal user, or defined in the QC.json? Is there a default heirarchy?

[dbirman]

How would one choose the hierarchy between

* probe

* * metric-type

and

* metric-type

* * probe

Is this per QC Portal user, or defined in the QC.json? Is there a default heirarchy?

I think we would still use the default_groups field to order these, by having a list that defines the tag that gets used to split metrics up at each level of the hierarchy. So the above would be ["probe", "metric-type"]. Then of course users in the UI can change the ordering at their whim. I'll make a mock-up and send it around so people can see what this would look like in practice.