QC Portal

The QC Portal is a browser application to view and interact with the AIND QC metadata and to annotate PENDING metrics with qualitative evaluations. The portal is currently maintained by Dan Birman in scientific computing, reach out with any questions or concerns.

The portal works by pulling the metadata from the Document Database (DocDB) and pulling reference figures from Code Ocean (CO) data assets, or from storage in Kachery Cloud.

The portal allows users to annotate PENDING metrics. Logged in users can modify the value, state, and notes on metrics. When you make changes the submit button will be enabled. Submitting pushes your updates to DocDB along with a timestamp and your name.

General documentation about the QC metadata.

IMPORTANT: The QC Portal relies on certain fields in the metadata being set correctly. These include all files in the data_description file. You must generate valid metadata or the QC portal will mangle displaying your data assets.

Metric or Curation?

At AIND we separate our quality control process into two steps:

1. Quality control of each modality in a data asset, i.e. we answer the question: "Can the data in this asset be used for analysis?"
2. Curation of the individual parts of an asset, i.e. we answer the question: "Can each neuron in this data asset be used for analysis?"

QCMetric is designed to store information about the former, while the CurationMetric should be used for the latter.

Defining metrics for the QC portal

Metrics should have actionable value fields. Either the value should be a number that a rule can be applied to (e.g. a threshold) or it should refer to the state of the reference (e.g. "high drift" when linked to a drift map, or "acceptable contrast" when linked to a video).

Almost all metrics should have a reference image, figure, or video attached. Often the reference should be shared across multiple metrics. Even if you are just calculating numbers, your reference figures can put those numbers in context for viewers, keep in mind that the portal is a public-facing resource! References can also embed linked pages in an iframe. Embedded links can point to Neuroglancer, FigURL, Rerun, and SortingView.

Q: QCMetric.value has type Any, what types are acceptable?

We expect the value to refer to a quantitative or qualitative assessment of some property of the data. When compared to a rule or threshold, the value establishes where that metric passes or fails quality control. So in general, the value field should be a number, string, or list of numbers/strings. Below is a table describing how different types are displayed in the portal:

Type	Display format	Panel type	Notes
Number	Editable number field	IntInput or FloatInput
String	Editable text field	TextInput
Boolean	Checkbox	Checkbox
Dictionary	Table	Dataframe	Values must have the same length
DropdownMetric	Dropdown	Dropdown	See aind-qcportal-schema
CheckboxMetric	Checkboxes	Checkbox	See aind-qcportal-schema
CurationMetric	Custom view		See aind-qcportal-schema

Q: CurationMetric.value has type Any, what types are acceptable?

In general you should put a dictionary mapping from an identifier (neuron ID, for example) to the properties of that object. Work with Dan to develop a custom Panel for your curation. The properties you provide can be used to display anything you want in your Panel app, but often you'll want a reference for every object so that you can flip through the images.

Note that in most situations a CurationMetric should always be set to pass.

Q: How does the reference get pulled into the QC Portal?

There are two aspects to references: (1) the type of the reference data, and (2) where the data is stored. These are independent.

Reference data types

• Vector files (svg, pdf)
• Images (png, jpg, etc)
• Interactive figures (e.g. altair)
• Videos (mp4)
• Embedded Neuroglancer and Figurl views (url, will be embedded in an iframe)
• Rerun files (rrd - version number must be embedded in the filename in the format filename_vX.Y.Z.rrd)

Data storage

You have a few options for where to store files. In general we prefer that you store files in the same data asset where the quality_control.json is located. The options below are in order of preference:

1. Provide the path to a file relative to the quality_control.json file, i.e. "figures/my_figure.png". Do not include the mount, asset name, or s3:// prefix.
2. Provide a kachery-cloud hash, i.e. "sha1://uuid.ext", note that you must append the filetype. The easiest way to do this is to set the label field to the filename, see below.
3. Provide a url to a publicly accessible file, i.e. "https://mywebsite.com/myfile.png"
4. Provide a path to any public S3 bucket, i.e. "s3://bucket/myfile.png"

Neuroglancer, Figurl, and SortingView links should point to the exact URL that opens the view you want.

Q: Can I put links into the description field to other resources?

The description field gets parsed as markdown, use the format [text](url).

Q: I saw fancy things like dropdowns in the QC Portal, how do I do that?

Custom value fields

The portal supports a few special cases to allow a bit more flexibility or to constrain the actions that manual annotators can take. Install the aind-qcportal-schema package and set the value field to the corresponding pydantic object to use these. Current options include:

• Dropdowns (optionally the options can auto-set the value)
• Checkboxes (again options can auto-set the value)
• Rule-based metrics (the rule is automatically run to set the value)
• Multi-asset metrics where each asset is assigned it's own value
• A dictionary where every value is a list of equal length, it will be displayed as a table where the keys are column headers and the values are rows. If a key "index" is included the values will be used to name the rows.

Special reference conditions

• If you put two reference strings separated by a semicolon ; they will be displayed in a Swipe pane that lets you swipe back and forth between the two things. Mostly useful for overlay images.
• If you re-use the same reference in multiple metrics, all of the metrics will be stacked in a single "Metric group".

Testing Metadata

You can upload test metadata to the /upload_metadata endpoint and then view them at /view?name=<metadata.name>. Uploaded metadata should be a dictionary with the fields the fields _id, name, location, data_description.project_name and quality_control. This metadata JSON does need to be fully valid, but the quality_control object does need to be valid against the current release of aind-data-schema.

import json
import requests

with open('metadata.json', 'r') as f:
    metadata = json.load(f)

response = requests.post('https://qc.allenneuraldynamics.org/upload_metadata', json=metadata)
print(f"Status: {response.status_code}")

Test metadata is transient, there is no guarantee for how long your metadata will be accessible. In addition, records in DocDB with identical names are prioritized over test records.

How to upload data from CO Capsules

Preferred workflow

Use the preferred workflow if you are generating a data asset, e.g. when uploading raw data or generating a new derived data asset. Your quality_control.json will go in the top level folder alongside other metadata and your figures will go in a subfolder. Follow the steps below:

1. Develop your QC pipeline, generating metrics and reference figures as needed. Place reference files in the results/ folder.
2. Populate QCEvaluation objects with metrics. The reference field should contain the path relative to the results folder. I.e. the file results/figures/my_figure.png should be included as QCMetric.reference = "figures/my_figure.png".
3. If your input data asset already has a quality_control.json file, then load the previous QC file by using qc = QualityControl(**json.loads(your_file)) and append your evaluations to qc.evaluations. If your input data asset has no QC, or this will be a new raw data asset, generate the QC object now qc = QualityControl(evaluations)
4. Write the standard QC file to the results folder: qc.write_standard_file()

Make sure to follow the standard instructions for creating derived assets.

Done! In the preferred workflow no additional permissions are required. Your QC data will appear in the portal once they are picked up by the indexer.

Alternate workflow

Use the alternate workflow only if you are not generating a data asset and therefore need to push your QC metadata back to an already existing data asset. You will push your QCEvaluation objects directly to DocDB and you will need to push your figures to kachery-cloud, an external repository that generates permanent links to uploaded files. Before using the alternate workflow, please consult with the Scientific Computing team.

Two things need to be setup in your capsule:

1. You'll need to run pip install kachery-cloud and pip install aind-data-access-api[docdb] as part of your environment setup.
2. In your capsule settings attach the aind-codeocean-power-user role. If you don't have access to this role, ask someone in Scientific Computing to attach it for you.

(1) Acquire your DocDB _id using your data asset's name

To upload directly to DocDB you'll need to know your asset's _id. You can get it by adding this code to your capsule and calling query_docdb_id(asset_name). Note that this is not the data asset id in Code Ocean!

from aind_data_access_api.document_db import MetadataDbClient

def query_docdb_id(asset_name: str):
    """
    Returns docdb_id for asset_name.
    Returns empty string if asset is not found.
    """

    # Resolve DocDB id of data asset
    API_GATEWAY_HOST = "api.allenneuraldynamics.org"
    DATABASE = "metadata_index"
    COLLECTION = "data_assets"

    docdb_api_client = MetadataDbClient(
    host=API_GATEWAY_HOST,
    database=DATABASE,
    collection=COLLECTION,
    )

    response = docdb_api_client.retrieve_docdb_records(
    filter_query={"name": asset_name},
    projection={"_id": 1},
    )

    if len(response) == 0:
        return ""
    docdb_id = response[0]["_id"]
    return docdb_id

(2) Generate your QC data

Generate your metrics and reference figures. Put your figures in folders in the results, e.g. results/figures/ and store the filepaths.

(3) Push figures to kachery-cloud

Your figures should already exist in folders in your results/. Then, in your capsule code, pull the Kachery Cloud credentials using this function:

import boto3

def get_kachery_secrets():
    """Obtains the three kachery-cloud environment keys/secrets```
    secret_name = "/aind/prod/kachery/credentials"
    region_name = "us-west-2"

    # Create a Secrets Manager client
    session = boto3.session.Session()
    client = session.client(
        service_name='secretsmanager',
        region_name=region_name
    )

    try:
        get_secret_value_response = client.get_secret_value(
            SecretId=secret_name
        )
    except ClientError as e:
        # For a list of exceptions thrown, see
        # https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html
        raise e

    secret = get_secret_value_response['SecretString']
    
    kachery_secrets = json.loads(secret)

    os.environ['KACHERY_ZONE'] = kachery_secrets['KACHERY_ZONE']
    os.environ['KACHERY_CLOUD_CLIENT_ID'] = kachery_secrets['KACHERY_CLOUD_CLIENT_ID']
    os.environ['KACHERY_CLOUD_PRIVATE_KEY'] = kachery_secrets['KACHERY_CLOUD_PRIVATE_KEY']

get_kachery_secrets()

The credentials are now stored as enviroment keys.

Each of your figures should then be uploaded using the store_file function:

import kachery_cloud as kcl

file_path = "your_file_path.ext"
uri = kcl.store_file(file_path, label=file_path)

(4) Generate your QCEvaluation objects

Generate your QCEvaluation objects now. Make sure to set the QCMetric.reference field of each metric to the returned uri QCMetric.reference = uri for that figure. Each URI is a unique hashed string that will allow the portal to recover your file. Make sure to include the label parameter or we won't be able to identify your filetype in the portal.

Store all your QCEvaluation objects in a list.

(5) Push metadata to DocDB

Run the following code snippet. You can pass all your evaluations as a list or pass them one at a time:

session = boto3.Session()
credentials = session.get_credentials()
host = "api.allenneuraldynamics.org"

auth = AWSRequestsAuth(
aws_access_key=credentials.access_key,
aws_secret_access_key=credentials.secret_key,
aws_token=credentials.token,
aws_host="api.allenneuraldynamics.org",
aws_region='us-west-2',
aws_service='execute-api'
)
url = f"https://{host}/v1/add_qc_evaluation"
post_request_content = {"data_asset_id": docdb_id,
                        "qc_evaluation": qc_eval.model_dump(mode='json')}
response = requests.post(url=url, auth=auth, 
                        json=post_request_content)

if response.status_code != 200:
    print(response.status_code)
    print(response.text)

If you get errors, contact Dan for help debugging.

Development

Launch

panel serve src/aind_qc_portal/qc_portal_app.py src/aind_qc_portal/qc_asset_app.py src/aind_qc_portal/qc_app.py --static-dirs images=src/aind_qc_portal/images --autoreload --show --port 5007 --allow-websocket-origin=10.128.141.92:5007 --keep-alive 10000

(port is set to differentiate from aind-metadata-viz app)

CI/CD

There is a Dockerfile which includes the entrypoint to launch the app.

Local dev

1. Build the Docker image locally and run a Docker container:

docker build -t aind-qc-portal .
docker run -e ALLOW_WEBSOCKET_ORIGIN=localhost:8000 -p 8000:8000 aind-qc-portal

2. Navigate to 'localhost:8000` to view the app.

AWS

1. On pushes to the dev or main branch, a GitHub Action will run to publish a Docker image to ghcr.io/allenneuraldynamics/aind-qc-portal:dev or ghcr.io/allenneuraldynamics/aind-qc-portal:latest.
2. The image can be used by a ECS Service in AWS to run a task container. Application Load Balancer can be used to serve the container from ECS. Please note that the task must be configured with the correct env variables (e.g. API_GATEWAY_HOST, ALLOW_WEBSOCKET_ORIGIN).