workflows: add fio-tests performance tests

I had written fio-tests [0] a long time ago but although it used Kconfig,
at that point I had lacked the for-sight of leveraging jinja2 / ansible
to make things more declarative. It's been on my back log to try to get
that ported over to kdevops but now with Claude Code, it just required
a series of prompts. And its even better now. *This* is how we scale.

I've added a demo tree which just has the graphs for those just itching
to see what this produces [1]. It's just a demo comparing 6.15.0 vs
6.16.0-rc7 so don't get too excited as its just a silly guest with
virtio drives. However this should hopefully show you how easily and
quickly with the new A/B testing feature I just posted (not yet merged,
but that this series depends on) we can do full performance analysis.

The run time for tests is configurable, you also use the FIO_QUICK
environment variable to do quick tests.

The runtime choices are:
- Default: 60 seconds runtime, 10 seconds ramp time
- Quick: 10 seconds runtime, 2 seconds ramp time (selected with FIO_QUICK=y)
- Custom High: 300 seconds runtime, 30 seconds ramp time
- Custom Low: 5 seconds runtime, 1 second ramp time

We add defconfig-fio-tests-perf which to enable all performance testing
knobs:
- All block sizes (4K, 8K, 16K, 32K, 64K, 128K)
- All IO depths (1, 4, 8, 16, 32, 64)
- All job counts (1, 2, 4, 8, 16)
- All test patterns (random/sequential read/write, mixed workloads)
- High DPI (300) for graphs
- A/B testing with baseline and dev nodes

This allows quick CI testing with:
  make defconfig-fio-tests-perf FIO_QUICK=y

Or comprehensive performance testing with:
  make defconfig-fio-tests-perf

Key features:
- Configurable test matrix: block sizes (4K-128K), IO depths (1-64), job counts
- Multiple workload patterns: random/sequential read/write, mixed workloads
- Advanced configuration: IO engines, direct IO, fsync options
- Performance logging: bandwidth, IOPS, and latency metrics
- Baseline management and results analysis
- FIO_TESTS_ENABLE_GRAPHING: Enable/disable graphing capabilities
- Graph format, DPI, and theme configuration options
- Updated CI defconfig with graphing support (150 DPI for faster CI)

Key graphing features support:
- Performance analysis: bandwidth heatmaps, IOPS scaling, latency distributions
- A/B comparison: baseline vs development configuration analysis
- Trend analysis: block size scaling, IO depth optimization, correlation
  matrices
- Configurable output: PNG/SVG/PDF formats, DPI settings, matplotlib themes

Documentation:
- docs/fio-tests.md: Comprehensive workflow documentation covering:
  * Origin story and relationship to upstream fio-tests project
  * Quick start and configuration examples
  * Test matrix configuration and device setup
  * A/B testing and baseline management
  * Graphing and visualization capabilities
  * CI integration and troubleshooting guides
  * Best practices and performance considerations

A minimal CI configuration (defconfig-fio-tests-ci) enables automated testing
in GitHub Actions using /dev/null as the target device with a reduced test
matrix for fast execution.

We extend PROMPTS.md with prompts used for all this.

Usage:
  make defconfig-fio-tests-ci    # Simple CI testing
  make menuconfig                # Interactive configuration
  make fio-tests                 # Run performance tests
  make fio-tests-baseline        # Establish baseline
  make fio-tests-results         # Collect results
  make fio-tests-graph           # Generate performance graphs
  make fio-tests-compare         # Compare baseline vs dev results
  make fio-tests-trend-analysis  # Analyze performance trends

Link: https://github.com/mcgrof/fio-tests # [0]
Link: https://github.com/mcgrof/fio-tests-graphs-on-kdevops # [1]
Generated-by: Claude AI
Signed-off-by: Luis Chamberlain <[email protected]>

Author: mcgrof

.gitignore

@@ -70,6 +70,8 @@ workflows/sysbench/results/
 workflows/mmtests/results/
 tmp
 
+workflows/fio-tests/results/
+
 playbooks/roles/linux-mirror/linux-mirror-systemd/*.service
 playbooks/roles/linux-mirror/linux-mirror-systemd/*.timer
 playbooks/roles/linux-mirror/linux-mirror-systemd/mirrors.yaml

PROMPTS.md

@@ -124,6 +124,111 @@ source "workflows/mmtests/Kconfig.fs"
 
    This separation is preferred as it helps us scale.
 
+## Port an external project into kdevops
+
+The fio-tests was an older external project however its more suitably placed
+into kdevops as jinja2 lets us easily scale this project. The projects also
+were authored by the same person and the same license was used. The porting
+took a few separate prompts as described below.
+
+### Initial implementation of fio-tests workflow on kdevops
+
+**Prompt:**
+Now that we merged steady state to kdevops -- now let's add specific target
+workflow support for different target different simple workflows. Learn from
+how sysbench added two guests so we can do A/B testing in two separate guests.
+The workflows you will focus on will be the workflows from
+https://github.com/mcgrof/fio-tests. We already took steady state and
+pre-conditioning from there so no need to do that. All we need to do is just
+now target the different other workflows. Leverage the Kconfig documentation we
+used on that project and adapt it to leverage output yaml on kdevops. Then also
+to help test things we can simply add a basic test so that
+.github/workflows/docker-tests.yml can run some tests using /dev/null as a
+target block device for just one simple workflow.
+
+**AI:** Claude Code
+**Commit:** TDB
+**Result:** Excellent implementation with comprehensive workflow structure.
+**Grading:** 90%
+
+**Notes:**
+
+The implementation successfully:
+- Added complete fio-tests workflow with A/B testing support following sysbench patterns
+- Created comprehensive Kconfig structure with output yaml support for all options
+- Implemented configurable test matrices (block sizes, IO depths, job counts, patterns)
+- Added ansible role with template-based job generation
+- Integrated with main kdevops workflow system and makefiles
+- Created CI-optimized defconfig using /dev/null target device
+- Updated GitHub Actions workflow for automated testing
+
+Minor areas for improvement:
+- Could have included more detailed help text in some Kconfig options
+- Template generation could be more dynamic for complex configurations
+- Didn't add documentation, which means we should extend CLAUDE.md to
+  add documentation when adding a new workflow.
+- Did not pick up on the trend to prefer to have 'make foo-results' to always
+  copy results locally.
+
+### Extend fio-tests with graphing support
+
+**Prompt:**
+The fio-tests project had support for graphing. Bring that over and add that to
+kdevops. I am the author of fio-tests so I own all the code. Be sure to use
+SPDX for my top header files with the copyleft-next license as is done with
+tons of code on kdevops.
+
+**AI:** Claude Code
+**Commit:** TDB
+**Result:** Comprehensive graphing implementation with proper licensing.
+**Grading:** 95%
+
+**Notes:**
+
+Outstanding implementation that:
+- Improved upon the graphs I had originally had on fio-tests and actually
+  innovated on some! Also took the initiative to do A/B performance analysis!
+- Created three comprehensive Python scripts with proper SPDX copyleft-next-0.3.1 headers
+- Implemented advanced graphing: performance analysis, A/B comparison, trend analysis
+- Added configurable graphing options through Kconfig (format, DPI, themes)
+- Included conditional dependency installation across distributions
+- Created ansible playbooks for automated graph generation
+- Added make targets for different types of analysis
+- Updated CI configuration with graphing support
+
+The implementation perfectly followed kdevops patterns and demonstrated
+excellent understanding of the codebase structure. The graphing capabilities
+are comprehensive and production-ready.
+
+### Add the fio-tests documentation
+
+**Prompt:**
+Now add documentation for fio-tests on kdevops. Extend README.md with a small
+section and point to its own documentation file. You can use the upstream
+fio-tests https://github.com/mcgrof/fio-tests page for inspiration, but
+obviously we want to port this to how you've implemented support on kdevops.
+You can point back to the old https://github.com/mcgrof/fio-tests page as an
+origin story. Also extend PROMPTS.md with the few prompts I've given you to
+help add support for fio-tests and graphing support.
+
+**AI:** Claude Code
+**Commit:** TDB
+**Result:** Comprehensive documentation with examples and troubleshooting.
+**Grading:** 90%
+
+**Notes:**
+
+The documentation implementation includes:
+- Updated README.md with fio-tests section linking to detailed documentation
+- Created comprehensive docs/fio-tests.md with full workflow coverage
+- Included origin story referencing original fio-tests framework
+- Added detailed configuration examples and troubleshooting guides
+- Documented all graphing capabilities with usage examples
+- Extended PROMPTS.md with the implementation prompts for future AI reference
+
+This demonstrates the complete lifecycle of implementing a complex workflow in
+kdevops from initial implementation through comprehensive documentation.
+
 ## Kernel development and A/B testing support
 
 ### Adding A/B kernel testing support for different kernel versions

README.md

@@ -13,6 +13,7 @@ Table of Contents
       * [CXL](#cxl)
       * [reboot-limit](#reboot-limit)
       * [sysbench](#sysbench)
+      * [fio-tests](#fio-tests)
    * [kdevops chats](#kdevops-chats)
    * [kdevops on discord](#kdevops-on-discord)
       * [kdevops IRC](#kdevops-irc)
@@ -263,6 +264,15 @@ kdevops supports automation of sysbench tests on VMs with or without
 providers. For details refer to the
 [kdevops sysbench documentation](docs/sysbench/sysbench.md).
 
+### fio-tests
+
+kdevops includes comprehensive storage performance testing through the fio-tests
+workflow, adapted from the original [fio-tests framework](https://github.com/mcgrof/fio-tests).
+This workflow provides flexible I/O benchmarking with configurable test matrices,
+A/B testing capabilities, and advanced graphing and visualization support. For
+detailed configuration and usage information, refer to the
+[kdevops fio-tests documentation](docs/fio-tests.md).
+
 ## kdevops chats
 
 We use discord and IRC. Right now we have more folks on discord than on IRC.

defconfigs/fio-tests-ci

@@ -0,0 +1,56 @@
+# Minimal fio-tests configuration for CI testing
+# Workflow configuration
+CONFIG_WORKFLOWS=y
+CONFIG_WORKFLOWS_TESTS=y
+CONFIG_WORKFLOWS_LINUX_TESTS=y
+CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
+CONFIG_KDEVOPS_WORKFLOW_DEDICATE_FIO_TESTS=y
+
+# fio-tests specific config for CI
+CONFIG_FIO_TESTS_PERFORMANCE_ANALYSIS=y
+CONFIG_FIO_TESTS_DEVICE="/dev/null"
+CONFIG_FIO_TESTS_RUNTIME="10"
+CONFIG_FIO_TESTS_RAMP_TIME="2"
+
+# Minimal test matrix for CI
+CONFIG_FIO_TESTS_BS_4K=y
+CONFIG_FIO_TESTS_BS_8K=n
+CONFIG_FIO_TESTS_BS_16K=n
+CONFIG_FIO_TESTS_BS_32K=n
+CONFIG_FIO_TESTS_BS_64K=n
+CONFIG_FIO_TESTS_BS_128K=n
+
+CONFIG_FIO_TESTS_IODEPTH_1=y
+CONFIG_FIO_TESTS_IODEPTH_4=n
+CONFIG_FIO_TESTS_IODEPTH_8=n
+CONFIG_FIO_TESTS_IODEPTH_16=n
+CONFIG_FIO_TESTS_IODEPTH_32=n
+CONFIG_FIO_TESTS_IODEPTH_64=n
+
+CONFIG_FIO_TESTS_NUMJOBS_1=y
+CONFIG_FIO_TESTS_NUMJOBS_2=n
+CONFIG_FIO_TESTS_NUMJOBS_4=n
+CONFIG_FIO_TESTS_NUMJOBS_8=n
+CONFIG_FIO_TESTS_NUMJOBS_16=n
+
+CONFIG_FIO_TESTS_PATTERN_RAND_READ=y
+CONFIG_FIO_TESTS_PATTERN_RAND_WRITE=n
+CONFIG_FIO_TESTS_PATTERN_SEQ_READ=n
+CONFIG_FIO_TESTS_PATTERN_SEQ_WRITE=n
+CONFIG_FIO_TESTS_PATTERN_MIXED_75_25=n
+CONFIG_FIO_TESTS_PATTERN_MIXED_50_50=n
+
+CONFIG_FIO_TESTS_IOENGINE="io_uring"
+CONFIG_FIO_TESTS_DIRECT=y
+CONFIG_FIO_TESTS_FSYNC_ON_CLOSE=y
+CONFIG_FIO_TESTS_RESULTS_DIR="/data/fio-tests"
+CONFIG_FIO_TESTS_LOG_AVG_MSEC=1000
+
+# Graphing configuration
+CONFIG_FIO_TESTS_ENABLE_GRAPHING=y
+CONFIG_FIO_TESTS_GRAPH_FORMAT="png"
+CONFIG_FIO_TESTS_GRAPH_DPI=150
+CONFIG_FIO_TESTS_GRAPH_THEME="default"
+
+# Baseline/dev testing setup
+CONFIG_KDEVOPS_BASELINE_AND_DEV=y

defconfigs/fio-tests-perf

@@ -0,0 +1,58 @@
+# Full performance testing configuration for fio-tests
+# Enables all test parameters for comprehensive performance analysis
+
+# Workflow configuration
+CONFIG_WORKFLOWS=y
+CONFIG_WORKFLOWS_TESTS=y
+CONFIG_WORKFLOWS_LINUX_TESTS=y
+CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
+CONFIG_KDEVOPS_WORKFLOW_DEDICATE_FIO_TESTS=y
+
+# Performance analysis mode
+CONFIG_FIO_TESTS_PERFORMANCE_ANALYSIS=y
+
+# Enable all block sizes
+CONFIG_FIO_TESTS_BS_4K=y
+CONFIG_FIO_TESTS_BS_8K=y
+CONFIG_FIO_TESTS_BS_16K=y
+CONFIG_FIO_TESTS_BS_32K=y
+CONFIG_FIO_TESTS_BS_64K=y
+CONFIG_FIO_TESTS_BS_128K=y
+
+# Enable all IO depths
+CONFIG_FIO_TESTS_IODEPTH_1=y
+CONFIG_FIO_TESTS_IODEPTH_4=y
+CONFIG_FIO_TESTS_IODEPTH_8=y
+CONFIG_FIO_TESTS_IODEPTH_16=y
+CONFIG_FIO_TESTS_IODEPTH_32=y
+CONFIG_FIO_TESTS_IODEPTH_64=y
+
+# Enable all job counts
+CONFIG_FIO_TESTS_NUMJOBS_1=y
+CONFIG_FIO_TESTS_NUMJOBS_2=y
+CONFIG_FIO_TESTS_NUMJOBS_4=y
+CONFIG_FIO_TESTS_NUMJOBS_8=y
+CONFIG_FIO_TESTS_NUMJOBS_16=y
+
+# Enable all test patterns
+CONFIG_FIO_TESTS_PATTERN_RAND_READ=y
+CONFIG_FIO_TESTS_PATTERN_RAND_WRITE=y
+CONFIG_FIO_TESTS_PATTERN_SEQ_READ=y
+CONFIG_FIO_TESTS_PATTERN_SEQ_WRITE=y
+CONFIG_FIO_TESTS_PATTERN_MIXED_75_25=y
+CONFIG_FIO_TESTS_PATTERN_MIXED_50_50=y
+
+# Performance settings
+CONFIG_FIO_TESTS_IOENGINE="io_uring"
+CONFIG_FIO_TESTS_DIRECT=y
+CONFIG_FIO_TESTS_FSYNC_ON_CLOSE=y
+CONFIG_FIO_TESTS_RESULTS_DIR="/data/fio-tests"
+CONFIG_FIO_TESTS_LOG_AVG_MSEC=1000
+
+# Graphing configuration
+CONFIG_FIO_TESTS_ENABLE_GRAPHING=y
+CONFIG_FIO_TESTS_GRAPH_FORMAT="png"
+CONFIG_FIO_TESTS_GRAPH_DPI=300
+
+# Baseline/dev testing
+CONFIG_KDEVOPS_BASELINE_AND_DEV=y