Poor performance of tmt/testing-farm with thousands of tests

[fpokryvk]

There seems to be some performance issue when too many testcases are defined in tests.fmf. Reading the logs in testing farm, it takes 4-6s to jump to the next test while it generates some yamls. This is a blocker for a testsuite with (not that) too many testcases defined, as few seconds multiplied by thousands makes few hours "of doing nothing" in the testsuite.

15:32:48         environment
15:32:48         Command returned '0' (success).
15:32:48         Copy '/var/ARTIFACTS/work-alln2wid3ea/plan/all/data' from the guest to '/'.
15:32:48         Run command: rsync -z --links -s -r -R --safe-links -e 'ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oConnectionAttempts=5 -oConnectTimeout=60 -oServerAliveInterval=5 -oServerAliveCountMax=60 -oIdentitiesOnly=yes -p22 -i /etc/citool.d/id_rsa_artemis -oPasswordAuthentication=no -S/var/ARTIFACTS/work-alln2wid3ea/ssh-sockets/10.0.178.196-22-root.socket' root@10.0.178.196:/var/ARTIFACTS/work-alln2wid3ea/plan/all/data /
15:32:48         environment
15:32:49         Command returned '0' (success).
15:32:49     Write file '/var/ARTIFACTS/work-alln2wid3ea/plan/all/execute/step.yaml'.
15:32:54     Write file '/var/ARTIFACTS/work-alln2wid3ea/plan/all/execute/results.yaml'.
15:32:54             00:00:01 pass /tests/skip_step_skip (on default-0) [2/1577]
15:32:54             test: /tests/skip_step_pass
15:32:54         Execute '/tests/skip_step_pass' as a 'shell' test.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and agent prompts for this issue.

• Create Plan

Examples

• Example 1
• Example 2

🔗 Similar & Duplicate Issues

Related Issues

• teemtee/tmt#3691
• teemtee/tmt#3647
• teemtee/tmt#3359
• teemtee/tmt#2332
• teemtee/tmt#4278
• teemtee/tmt#3453
• teemtee/tmt#4066
• teemtee/tmt#3955

👤 Suggested Assignees

• thrix
• bgoncalv
• tuliom
• The-Mule
• psss

---

🧪 Issue enrichment is currently in early access.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

[happz]

There seems to be some performance issue when too many testcases are defined in tests.fmf. Reading the logs in testing farm, it takes 4-6s to jump to the next test while it generates some yamls. This is a blocker for a testsuite with (not that) too many testcases defined, as few seconds multiplied by thousands makes few hours "of doing nothing" in the testsuite.

15:32:48         environment
15:32:48         Command returned '0' (success).
15:32:48         Copy '/var/ARTIFACTS/work-alln2wid3ea/plan/all/data' from the guest to '/'.
15:32:48         Run command: rsync -z --links -s -r -R --safe-links -e 'ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oConnectionAttempts=5 -oConnectTimeout=60 -oServerAliveInterval=5 -oServerAliveCountMax=60 -oIdentitiesOnly=yes -p22 -i /etc/citool.d/id_rsa_artemis -oPasswordAuthentication=no -S/var/ARTIFACTS/work-alln2wid3ea/ssh-sockets/10.0.178.196-22-root.socket' root@10.0.178.196:/var/ARTIFACTS/work-alln2wid3ea/plan/all/data /
15:32:48         environment
15:32:49         Command returned '0' (success).
15:32:49     Write file '/var/ARTIFACTS/work-alln2wid3ea/plan/all/execute/step.yaml'.
15:32:54     Write file '/var/ARTIFACTS/work-alln2wid3ea/plan/all/execute/results.yaml'.
15:32:54             00:00:01 pass /tests/skip_step_skip (on default-0) [2/1577]
15:32:54             test: /tests/skip_step_pass
15:32:54         Execute '/tests/skip_step_pass' as a 'shell' test.

Can you share any links, e.g. to the TF job, git repo with such tests, anything else related to your issue?

[fpokryvk]

The repository is https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci/ - I have only internal testing-farm links for now.

[happz]

Some notes:

• YAML isn't the cheapest format. We can add support for other formats for internal files, e.g. use minified JSON, some binary format, etc. YAML has advantage of being human-readable, but with 1500+ tests, I wouldn't be reading such a file anyway :)
• results.yaml being part of tmt API, with TF expecting results.yaml, we would need help from TF to integrate with new format.

[LecrisUT]

What are the candidates to this slow down? If I read the logs correctly, it is slow to write the initial results.yaml file and not at updating/rewriting the file, or is that slow down for each test? If it's the latter maybe the file could be written only once and then have it point to external file reference instead. The content of exceute/results.yaml is replicated in each test's results.yaml isn't it?

[happz]

What are the candidates to this slow down? If I read the logs correctly, it is slow to write the initial results.yaml file and not at updating/rewriting the file, or is that slow down for each test?

Each test.

If it's the latter maybe the file could be written only once and then have it point to external file reference instead. The content of exceute/results.yaml is replicated in each test's results.yaml isn't it?

Not replicated, the internal collection of results is updated with results from the test that just finished, and the collection is saved into results.yaml. There is no per-test results.yaml, just one results.yaml in the execute step directory. Every time a couple of list entries change, or new one appear, with the custom results, and the file is overwritten.

Splitting to per-test results.yaml would be impractical, especially from the PoV of consumers of results.yaml. We can use a binary format instead, or something as packed as JSON, with less formatting - YAML is really expensive because of this.

[LecrisUT]

We can use a binary format instead, or something as packed as JSON, with less formatting - YAML is really expensive because of this.

Wouldn't that be basically just as disruptive? We could offer an interface first so that we can more easily edit the interface underneath.

Alternatively, have we checked ruamel.yaml's performance for in-place updates?

[happz]

We can use a binary format instead, or something as packed as JSON, with less formatting - YAML is really expensive because of this.

Wouldn't that be basically just as disruptive?

I don't think so. Serializing 1500+ test results and formatting them as a YAML list is expensive; formatting them as a JSON or BSON would not get anywhere close to the time needed for YAML.

[tyll]

From the description of the algorithm, it seems sufficient to change it so that each test result is only written once instead of re-writing the full file all the time. Writing a full results.yaml from NMCI only takes about 1.2 seconds on my laptop, so serializing each test results only once would already fix this and scale a lot better. In the end, 1500 results are not that much. The results.yaml seems to contain a list and just appending another list element to an existing yaml file should not be a problem. JSON/BSON do not allow to simply append another item without changing existing file contents. A better choice might take this into account and use a format that supports streaming, for example https://github.com/pyrustic/paradict#paradict (just something I found quickly).