Contribute community partner specific test summaries

A partner may generate test summaries.

In the partner section of collection_rdf_template.yaml we need to specify your partner_id (e.g. 'ilastik') and your workflow which creates and deploys your test summaries.

These dedicated partner test workflows are triggered by the collection CI trigger_partner_wf.yaml for any newly added resource version. For this to work the partner repository needs to invite @bioimageiobot as collaborator and the partner repository needs to implement a GitHub Actions workflow which is callable from another workflow and accepts a pending_matrix input:

on:
  workflow_dispatch:
    inputs:
      pending_matrix:
        description: 'json encoded gh matrix of pending validations for new or updated resources (default: test on all resources)'
        required: true
        default: '{"include": [{"resource_id": "**", "version_id": "**"}]}'
        type: string

The workflow should generate test summaries for the resource versions given by pending_matrix. The RDFs of any resource version can be found in the gh-pages branch of the collection-bioimage-io repository as https://github.com/bioimage-io/collection-bioimage-io/tree/gh-pages/rdfs/<resource_id>/<version_id>/rdf.yaml

Test summaries are mappings with the following keys saved as .yaml files:

bioimageio_spec_version: # str
bioimageio_core_version: # str
error: # Union[None, str, Dict[str, Any]]
name: # str
nested_errors: # Optional[Dict[str, dict]]
source_name: # str
status: # Union[Literal["passed", "failed"], str]
traceback: # Optional[List[str]]
warnings: # dict

The current keys are defined here.

The GitHub Actions workflow should generated these test summaries and deploy them as configured in https://github.com/bioimage-io/collection-bioimage-io/blob/main/collection_rdf_template.yaml :config:partner:<partner entry>:test_summaries https://github.com/ilastik/bioimage-io-resources/tree/gh-pages/test_summaries/10.5281/zenodo.5749843/5877226 The collection-bioimage-io CI combines all partner and the bioimageio core test summaries and deploys this test summary to https://github.com/bioimage-io/collection-bioimage-io/blob/gh-pages/rdfs/<resource_id>/<version_id>.test_summary.yaml, e.g. here from which the bioimage.io website renders them.

• example ilastik:

  • repository with ilastik's test ci: https://github.com/ilastik/bioimage-io-resources
  • this github actions workflow is called by the collection CI: test_bioimageio_resources.yaml
  • deployed test summaries: https://github.com/ilastik/bioimage-io-resources/tree/gh-pages/test_summaries
  • config of this process in collection repo: collection_rdf_template.yaml#L63-L68

• (unfinished) example deepimagej:

  • gh workflow called by collection CI: test_bioimageio_resources.yaml
  • config in collection repo: collection_rdf_template.yaml#L41-L50
  • Specific material from deepImageJ's model repository to run the CI:
    • A drafted GitHub actions workflow to download the installation of Fiji/ImageJ and call the deepImageJ macro that processes the test image and generates an output for each of the models in the repo. The workflow will test that all the models produce the corresponding output tiff file.
    • The workflow calls deepImageJ in headless mode