Requirements for configuration tracking

[bruno-f-cruz]

A module to track animal-related configuration would be useful to agilize experiments. A short term solution would be to extend the current default picker with a more opinionated logic on how to fetch and validate a full session of a given animal. Alternatively, this could ideally be handled by a centralized data base structure.

Requirements: (Mostly derived from VrForaging but in theory applicable to other pipelines)

• We need to be able to store JSON documents (bonus for other asset types?)
• From a rig_id (aibs_rig_id and a task (task_name), grab rig configuration (or potentially multiple if available)
• List multiple task configuration files (these would be used as defaults if needed)
• From an animal id, return a task_configuration (task_logic.json or a CurriculumState, both of which are sufficient to grab a task_name).
• From an animal id, return a experimental schedule: e.g. Experimenter, aibs_rig_id, TimeOfDay, etc...
• From a given rig configuration instance, grab a rig.json (aind-data-schema)
• Need an API to read and write to the collection. Ideally Python, but bonus if also C#.
• Ideally would store all the documents ever written/updates

[tiffanyona]

• Store calibration files (i.e. water) with log of changes

[bruno-f-cruz]

Follow-up from meeting with @patricklatimer @jessy-liao

Database api requirements for curriculum integration

• Mouse training suggestions are in the form of an instance of the following model https://github.com/AllenNeuralDynamics/aind-behavior-curriculum/blob/0af84e75131e8f015f28fbe10009d1b2c499043b/src/aind_behavior_curriculum/trainer.py#L20. Specifically, the stage.task has all the parameters necessary to run the animal session when it comes to task logic settings (this does NOT include rig settings for instance)

• These instances are stored in the form of json (powered by pydantic serializer)

Table schema as provided by SciComp

Key	Column name	Notes / Formatting
PK	behavior_session_id	Primary key
FK	mouse_id	Foreign key
	date: date	Date field
	task_name
	stage_name
	stage_configuration
FK	instrument	Foreign key
FK	operator	Foreign key

Expected behavior

Fetching data at the start of the session

(For all these API calls I am going to assume that stage_configuration contains the serialized TrainerState. Alternative, most of these API calls must be able to support attached document fetching too.)

The following queries should be supported:

• get_latest_suggestion(mouse_id) -> returns a single row with the freshest suggestion for a given animal, if it exists
• get_suggestions(mouse_id, date) -> returns N suggestions with date < date
• get_date_suggestions(date) -> returns N suggestions (for all animals) with date = date

All the queries should additionally support simple filtering options (i.e. WHERE-like behavior) for the following fields:
- task_name
- stage_name
- operator
- instrument

Pushing Suggestions

At the end of the session, users will take the current trainer state and the behavior of the animal. They will then create a new suggestion based on this information and push it to the database.

The following abstract API call should be supported:

• push_suggestion(mouse_id, suggestion: TrainerState) -> pushes a new suggestion for a given animal

  • All other required fields can be automatically populated:
    • behavior_session_id (uid ?)
    • date (current date)
    • task_name (trainerstate.stage.task.name)
    • stage_name (trainerstate.stage.name)
    • stage_configuration (trainerstate)

• push_suggestion(mouse_id, suggestion: TrainerState, operator: str, instrument: str) (probably worth supporting too if people want to push it on their own)

In practice, you will probably implement push_suggestion like this:

`def push_suggestion(orm_for_table)` and I can assemble the orm on my side if needed

[bruno-f-cruz]

For future reference, here is how we read/write frm dataverse File types

def _download_file_from_guid(client: DataverseRestClient, session: dict):
    uuid = session["aibs_fact_mouse_proposed_behavior_sessionsid"]
    url = client.config.api_url + _SUGGESTIONS_TABLE
    url = f"{url}({uuid})/aibs_trainer_state/$value"
    response = requests.get(url, headers=client.headers)

    response.raise_for_status()
    return response.content


def _upload_file_from_guid(client: DataverseRestClient, session: dict, file_path: str):
    uuid = session["aibs_fact_mouse_proposed_behavior_sessionsid"]

    with open(file_path, "rb") as f:
        file_bytes = f.read()

    url = f"{client.config.api_url}{_SUGGESTIONS_TABLE}({uuid})/aibs_trainer_state"
    headers = client.headers.copy()
    headers["Content-Type"] = "application/octet-stream"

    headers["x-ms-file-name"] = os.path.basename(file_path)

    response = requests.patch(url, headers=headers, data=file_bytes, timeout=30)
    response.raise_for_status()
    return response.content