Assign static IDs to DTS E2E tests to be able to add them to OSFV Dashboard

[DaniilKl]

The problem you're addressing (if any)

Currently DTS E2E tests are being generated dynamically according to the template https://github.com/Dasharo/open-source-firmware-validation/blob/d326bfb29d5a3fb374d4c728efb786b61aa8eb2a/dts/dts-e2e.robot#L23

The tests IDs are generated dynamically too. While it is still possible to identify the test case by its name, the OSFV Dashboard needs IDs.

Describe the solution you'd like

The DTS E2E test is being defined by the three fields:

${platform}    ${workflow}    ${release}

Every of the three fields are being defined in the platform-configs/*.robot files. Can we define the IDs in the same place, so every combination possible of the three fields above will have a unique ID?

Where is the value to a user, and who might that user be?

We could:

• Unify the tests results storage and presentation with Dasharo via OSFV Dashboard.
• Present the tests to clients via Dasharo/osfv-results or OSFV Dashboard.

Describe alternatives you've considered

Using DTS E2E tests names insted of the IDs in the OSFV Dashboard.

Additional context

No response

[DaniilKl]

@m-iwanicki, what do you think?

[m-iwanicki]

Does ID has to have some specific format? We could use hash of ${platform}, ${workflow}, ${release}. While there exists chance of collision it's very unlikely unless we use very short hash or trim sha256

There is also possibility of hardcoding those values e.g.

E2E<${platform_id}><${workflow_id}><${release_id}>:

[DaniilKl]

The ID format is defined by OSFV Dashboard, check https://github.com/Dasharo/open-source-firmware-validation/blob/develop/docs/adding-and-naming-test-cases.md .

[m-iwanicki]

It's a three digit number with leading zeros. Identifies test cases in a test suite. Typically test cases are numbered incrementally starting from 001.

The biggest problem here might be three digit number. We won't be able to use E2E<${platform_id}><${workflow_id}><${release_id}> format in that case as I'm pretty sure we have more than 10 platforms and workflows (or soon we will have). I need to think on it a little longer as I'm unsure how to resolve it.

[DaniilKl]

It's a three digit number with leading zeros. Identifies test cases in a test suite. Typically test cases are numbered incrementally starting from 001.

The biggest problem here might be three digit number. We won't be able to use E2E<${platform_id}><${workflow_id}><${release_id}> format in that case as I'm pretty sure we have more than 10 platforms and workflows (or soon we will have). I need to think on it a little longer as I'm unsure how to resolve it.

Why not use the ID incrementally without dividing it into <${platform_id}><${workflow_id}><${release_id}>, is there other advantages than better IDs interpretation by human? We would have 1000 combinations then.

[m-iwanicki]

Why not use the ID incrementally

Because we would need to keep list or json with mappings between test name and id. We could do it but maybe another way would be better.

Use 2 digits for platform, 2 digits for workflow and 1 digit for release e.g.:

E2E<${platform_id}><${release_id}>.2<${workflow_id}>.

Not sure if even this will be allowed as:

All the environment_ids of all the tested environments are defined as Robot Framework variables in the os-config/environment-test-ids.py file.

and in this file:

# 2xx - Linux
ENV_ID_UBUNTU = "201"
ENV_ID_FEDORA = "202"
ENV_ID_QUBES = "203"
ENV_ID_TRENCHBOOT = "204"
ENV_ID_XCP_NG = "205"

So I guess those xx are also reserved

[DaniilKl]

@BeataZdunczyk, can we use E2E<${platform_id}><${release_id}>.2<${workflow_id}> or E2E<${platform_id}><${release_id}>.0<${workflow_id}> in the way @m-iwanicki, described it above?

[m-iwanicki]

Currently template test names are made up of 3 parts:

${platform} ${workflow} - ${release}

• platform - platform we are emulating during test, it's the robot filename in platform-configs e.g. novacusom-nv41pz (without extension)
• workflow - easy to understand name of what we are testing e.g. SeaBIOS Update, UEFI->Heads Transition, etc.
• release - either DCR or DPP. The only difference is whether we are inputting DPP keys at the beginning of workflow

Example implementation of IDs:
Lets say IDs are made up of 6 digits: xyz.abc

• In OSFV there are platform-config files, in each platform file we will set DTS_PLATFORM_ID to unique value. This value will be used as first 2 digits (xy)
• In default.robot we define all possible releases. Each release would have it's own unique ID e.g. DTS_RELEASE_ID. This will be used as 3rd digit (z). Currently we only support 2 releases (DCR and DPP)
• In default.robot we define all possible workflows. Each workflow would have it's own unique ID e.g. DTS_WORKFLOW_ID. This will be used as last 2 digits in the second ID part (bc). Currently we support 8 workflows.

To make sure there are no duplicate IDs template tests should have different suite_id e.g. E2ET from normal tests were IDs are hard-coded.

---

Another way:
Create file with list of IDs matching test name e.g.:

001.201: <platform> <workflow> - <release1>
002.201: <platform> <workflow> - <release2>

When generating template test, find a line that contains test name (after <ID>:) and prepend found ID.

[DaniilKl]

@macpijan, do you approve the idea?

[3mkusiak]

@macpijan ping