feat(tests): add disk performance benchmark tests by Millefeuille42 · Pull Request #324 · xcp-ng/xcp-ng-tests

Millefeuille42 · 2025-06-13T09:46:52Z

No description provided.

Signed-off-by: Mathieu Labourier <mathieu.labourier@vates.tech>

Millefeuille42 · 2025-06-13T15:13:54Z

@Nambrok @AnthoineB I added usage of the SR, VM, VBD, etc. fixtures. For now it's only using a local SR but I could make it use a specific SR (shared or local).

I still need to add some stuff like making all parameters editable through the config file / flags. Feel free to add feedback on the current state of the PR if needed

tests/storage/benchmarks/conftest.py

last-genius · 2025-06-17T12:15:28Z

Please squash some of the fix and style commits. Otherwise it's difficult to review commits when they are changed just later in the series

- fix(disk_perf): add configurable numjob - fix(disk_perf): styling and readability - misc(disk_perf): simplify fixtures - misc(disk_perf): styling - fix(disk_perf): usage of double quote in nested string Signed-off-by: Mathieu Labourier <mathieu.labourier@vates.tech>

- style(disk_perf): sort imports and minor style fixes Signed-off-by: Mathieu Labourier <mathieu.labourier@vates.tech>

- fix(disk_perf): csv paths and add logging Signed-off-by: Mathieu Labourier <mathieu.labourier@vates.tech>

Signed-off-by: Mathieu Labourier <mathieu.labourier@vates.tech>

Millefeuille42 · 2025-06-17T12:25:27Z

@last-genius

Please squash some of the fix and style commits. Otherwise it's difficult to review commits when they are changed just later in the series

Done

Nambrok · 2025-06-18T08:05:45Z

jobs.py

+        "params": {
+            "--vm": "single/small_vm",
+        },
+        "paths": ["tests/storage"],


path should point to benchmark subdirectory directly in the job

Suggested change

"paths": ["tests/storage"],

"paths": ["tests/storage/benchmarks"],

Fixed in 377d577

Nambrok · 2025-06-19T07:31:47Z

tests/storage/benchmarks/conftest.py

+
+
+# use vhd, qcow2, raw... when image_format support will be available
+@pytest.fixture(scope="module", params=["vdi"])


The temporary one could have been vhd ^^

Fixed in 377d577

Nambrok · 2025-06-19T07:46:48Z

tests/storage/benchmarks/conftest.py

+        ("command -v apk", "apk add fio", "apk del fio"),
+    )
+
+    for check_cmd, install_cmd, remove in install_cmds:


There is already code allowing to install things on a VM.
You have for example lib/vm.py:VM.detect_package_manager() that exist.
You can see example of its usage in tests/guest_tools/unix/test_guest_tools_unix.py.
To uninstall, you can make a snapshot before installing your package and revert at the end to have the VM reset to its initial state (There might be a fixture already doing this).

fixed in df0b40f

Nambrok · 2025-06-19T07:47:57Z

tests/storage/benchmarks/conftest.py

+@pytest.fixture(scope="module")
+def vdi_on_local_sr(host, local_sr_on_hostA1, image_format):
+    sr = local_sr_on_hostA1
+    vdi = sr.create_vdi("testVDI", MAX_LENGTH)


The image_format has to be given to create_vdi

Suggested change

vdi = sr.create_vdi("testVDI", MAX_LENGTH)

vdi = sr.create_vdi("testVDI", MAX_LENGTH, image_format=image_format)

Image format is not supported yet, resulting in an error. I will provide it as a comment though.

Fixed in 377d577

Nambrok · 2025-06-19T07:48:11Z

tests/storage/benchmarks/conftest.py

+def vdi_on_local_sr(host, local_sr_on_hostA1, image_format):
+    sr = local_sr_on_hostA1
+    vdi = sr.create_vdi("testVDI", MAX_LENGTH)
+    vdi.image_format = image_format


Suggested change

vdi.image_format = image_format

Fixed in 377d577

- change job path - set "vhd" as the placeholder image_format - remove leftover code Signed-off-by: Mathieu Labourier <mathieu.labourier@vates.tech>

…rt install Signed-off-by: Mathieu Labourier <mathieu.labourier@vates.tech>

AnthoineB · 2025-06-20T08:21:40Z

tests/storage/benchmarks/conftest.py

+        "--file-sizes",
+        action="store",
+        type=lambda value: str_to_tuple(value, sep=","),
+        default=("1G", "4G", f"{int((system_memory // (1024.**3)) * 2)}G"),


Small comment

Suggested change

default=("1G", "4G", f"{int((system_memory // (1024.**3)) * 2)}G"),

default=("1G", "4G", f"{int((system_memory // (1024.**3)) * 2)}G"), # (2*Memory) GiB

fixed in 65157ae

AnthoineB · 2025-06-20T13:01:55Z

tests/storage/benchmarks/test_disk_perf.py

+        vm = running_unix_vm_with_fio
+        vbd = plugged_vbd
+        device = f"/dev/{vbd.param_get(param_name='device')}"
+        test_type = "{}-{}-{}-{}".format(block_size, file_size, rw_mode, image_format)


Do we want to add a prefix here, like "bench-fio-", to have a common part of the name. If there is other tests we can have something like "bench-newtool-".

I know that adding more in the test_type could look heavy but shouldn't we add iodepth and numjob in the test_type string?

We should add the bench- prefix indeed as well as the numjob and iodepth. I forgot to add the latter part when I added those options. I will fix that.

done in 63e2cf5

AnthoineB · 2025-06-20T13:43:41Z

tests/storage/benchmarks/test_disk_perf.py

+    logging.info("Performance difference summary:")
+    for k, v in diffs.items():
+        sign = "+" if v < 0 else "-"
+        logging.info(f"- {k}: {sign}{abs(v):.2f}%")


It may be interesting to have a condition that log a special message if the performance are above a threshold to signal a perf improvement.

That could be interesting. Both threshold (for failure and future positive message) could be changed through flags too

done in 63e2cf5

Co-authored-by: Anthoine <anthoine.bourgeois@vates.tech> Signed-off-by: Millefeuille <millefeuille42@proton.me>

- add improvement threshold - adjust wording in messages - add additional information in test_type - fix percentage calculation - styling Signed-off-by: Mathieu Labourier <mathieu.labourier@vates.tech>

Wescoeur · 2025-08-04T14:58:33Z

tests/storage/benchmarks/conftest.py

+MAX_LENGTH = 64 * (1024**3)  # 64GiB
+
+
+# use vhd, qcow2, raw... when image_format support will be available


Should have a TODO: label in this case, otherwise "will" is incorrect. Otherwise we can block this PR as long as the desired code is not merged I suppose.

added the TODO label in 718b41f

As for blocking the PR, idk since the support of image formats is not required to make it work for now. If we don't block this PR we should be cautious to not loose track of this TODO (maybe make a card about this?)

Afaik, qcow2 is now available for testing. Is it hard to also include it here ? Otherwise, I'm fine with a todo.

I will add it

Signed-off-by: Mathieu Labourier <mathieu.labourier@vates.tech>

glehmann

That's a very good start for performance monitoring.
Later, we'll probably need something more accessible than a local file to store the results, to be able to run from multiple hosts and raise alerts based on more data than just the previous one.

glehmann · 2026-02-17T07:22:07Z

tests/storage/benchmarks/conftest.py

+def pytest_addoption(parser):
+    system_memory = os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES")
+
+    parser.addoption(


IMO, instead of adding options to pytest, we should have multiple tests with predefined values that we can run in our CI.

Millefeuille42 self-assigned this Jun 13, 2025

feat(tests): add disk performance benchmark tests

500f788

Signed-off-by: Mathieu Labourier <mathieu.labourier@vates.tech>

Millefeuille42 force-pushed the test-disk-performance branch 2 times, most recently from c89c4e6 to 385f290 Compare June 13, 2025 15:06

Millefeuille42 force-pushed the test-disk-performance branch 2 times, most recently from 539c38d to 68ecbbc Compare June 16, 2025 10:15

Millefeuille42 commented Jun 17, 2025

View reviewed changes

tests/storage/benchmarks/conftest.py Outdated Show resolved Hide resolved

Millefeuille42 marked this pull request as ready for review June 17, 2025 09:14

Millefeuille42 requested review from AnthoineB and Nambrok June 17, 2025 09:15

Millefeuille42 added 4 commits June 17, 2025 14:24

feat(disk_perf): add support of uri for previous csv file

577eff2

- style(disk_perf): sort imports and minor style fixes Signed-off-by: Mathieu Labourier <mathieu.labourier@vates.tech>

feat(disk_perf): parameterize most test variables

e58c459

- fix(disk_perf): csv paths and add logging Signed-off-by: Mathieu Labourier <mathieu.labourier@vates.tech>

misc(disk_perf): add marks and job definition

efa4b26

Signed-off-by: Mathieu Labourier <mathieu.labourier@vates.tech>

Millefeuille42 force-pushed the test-disk-performance branch from 68ecbbc to efa4b26 Compare June 17, 2025 12:25

Nambrok requested changes Jun 19, 2025

View reviewed changes

Millefeuille42 added 2 commits June 19, 2025 10:54

fix(benchmarks): implement short changes from review

377d577

- change job path - set "vhd" as the placeholder image_format - remove leftover code Signed-off-by: Mathieu Labourier <mathieu.labourier@vates.tech>

fix(benchmarks): use lib to find package manager and snapshot to reve…

df0b40f

…rt install Signed-off-by: Mathieu Labourier <mathieu.labourier@vates.tech>

Millefeuille42 requested a review from Nambrok June 19, 2025 08:56

AnthoineB reviewed Jun 20, 2025

View reviewed changes

Millefeuille42 and others added 2 commits June 20, 2025 16:01

misc: add comment explaining the default file size calculation

65157ae

Co-authored-by: Anthoine <anthoine.bourgeois@vates.tech> Signed-off-by: Millefeuille <millefeuille42@proton.me>

misc(benchmarks): implement suggestions from reviews

63e2cf5

- add improvement threshold - adjust wording in messages - add additional information in test_type - fix percentage calculation - styling Signed-off-by: Mathieu Labourier <mathieu.labourier@vates.tech>

Millefeuille42 requested a review from AnthoineB July 3, 2025 09:48

Millefeuille42 mentioned this pull request Jul 31, 2025

lib/host: Avoid using identical source and destination paths in execute_script scp #334

Merged

Wescoeur requested changes Aug 4, 2025

View reviewed changes

fix: add TODO label for image_format fixture

718b41f

Signed-off-by: Mathieu Labourier <mathieu.labourier@vates.tech>

klmp200 requested a review from Wescoeur January 23, 2026 14:17

Wescoeur removed their request for review January 27, 2026 09:57

glehmann reviewed Feb 17, 2026

View reviewed changes

Lankou66 self-assigned this Feb 23, 2026

	"paths": ["tests/storage"],
	"paths": ["tests/storage/benchmarks"],



		# use vhd, qcow2, raw... when image_format support will be available
		@pytest.fixture(scope="module", params=["vdi"])

	vdi = sr.create_vdi("testVDI", MAX_LENGTH)
	vdi = sr.create_vdi("testVDI", MAX_LENGTH, image_format=image_format)

	default=("1G", "4G", f"{int((system_memory // (1024.*3)) 2)}G"),
	default=("1G", "4G", f"{int((system_memory // (1024.*3)) 2)}G"), # (2*Memory) GiB

		MAX_LENGTH = 64 * (1024**3) # 64GiB


		# use vhd, qcow2, raw... when image_format support will be available

Conversation

Millefeuille42 commented Jun 13, 2025

Uh oh!

Millefeuille42 commented Jun 13, 2025

Uh oh!

Uh oh!

last-genius commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Millefeuille42 commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Nambrok Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Millefeuille42 Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Wescoeur Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

glehmann left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

last-genius commented Jun 17, 2025 •

edited

Loading

Millefeuille42 commented Jun 17, 2025 •

edited

Loading

Nambrok Jun 19, 2025 •

edited

Loading

Millefeuille42 Jun 20, 2025 •

edited

Loading

Wescoeur Aug 4, 2025 •

edited

Loading