Informatics Matters Data Manager Job Decoder

A package that simplifies the decoding of encoded text strings and validation of Job Definitions and Manifests.

Given an encoded string the decode() method returns the decoded value or an error.

For example, given the following jinja2 encoded string '{{ foo }}, bar={{ bar }}, baz={{ baz }}' and variable map {'foo': 1, 'bar': 2, 'baz': 3} the decoder returns the string 'foo=1, bar=2, baz=3'.

The following encoding/decoding formats are supported: -

• jinja2 (3.0)

The package also provides validate_job_schema() and validate_manifest_schema() functions, which can (should) be used to validate the Manifests and Job definitions against the built-in schemas.

If you have a Job Definition or Manifest file you can add a reference to the corresponding schema in this package to your file to help you identify schema violations prior to deployment. To do this you just need to add a line starting # yaml-language-server: at the top of your file. For example, you can link to the v2.3.0 Job Definition schema by adding the following line:

# yaml-language-server: $schema=https://raw.githubusercontent.com/InformaticsMatters/squonk2-data-manager-job-decoder/refs/tags/2.3.0/decoder/job-definition-schema.yaml

Installation (Python)

The Job decoder is published on PyPI and can be installed from there:

pip install im-data-manager-job-decoder

Once installed you can validate the definition (expected to be a dictionary formed from the definition YAML file) with:

>>> from decoder import decoder
>>> error: Optional[str] = decoder.validate_manifest_schema(manifest)
>>> error: Optional[str] = decoder.validate_job_schema(job_definition)

And you can decode encoded fields within the job definition. Here we're decoding the job's 'command' using a map of variables and their values:

>>> decoded, success = decoder.decode(definition['command'],
                                      variables,
                                      'command',
                                      decoder.TextEncoding.JINJA2_3_0)

A method also exists to generate a Data Manger Job documentation URL for the supported repository types. For example, to get the auto-generated documentation URL for a job definition hosted in a GitHub repository you can do this:

>>> doc_url: str = decoder.get_job_doc_url("github",
                                           collection
                                           job_id,
                                           job_definition,
                                           manifest_url)

We support github and gitlab as repository types.

Get in touch

• Report bugs, suggest features or view the source code on GitHub.