[Topic] Rewrite benchmark-operator and limit scope

[jtaleric]

Benchmark Operator has been the Swiss Army Knife of k8s-benchmarking. However, overtime it has become extremely difficult to debug, maintain and contribute to. Similarly, we have built some other tooling to help mitigate the need to continue enhancing benchmark-operator.

After some internal evaluation, we still see the need to keep aspects of the benchmark-operator alive -- mainly for its ability to run storage based workloads. (fio, ycsb, and hammerdb).

I am suggesting we :

1. move away from the ansible-sdk mainly for the lack of easy debug. The output from the playbook execution is very busy, and it is hard to triage where issues manifest from.
2. Much of the benchmark knowledge is in https://github.com/cloud-bulldozer/benchmark-wrapper, so we could continue to use that (or maybe specific aspects of it).
3. since we move away from ansible-sdk, rewrite the benchmark-operator using the golang framework to manage the deployment of the pods running the workloads.
4. minimize the scope of benchmark-operator to be more storage focused.

[ddomingu]

A pluggable operator for performance testing with multiple tools is exactly what's needed in the field. The cloud-benchmark operator isn’t widely used because it seems abandoned, but it’s still a solid and functional tool. The experience varies depending on the plugin: FIO works great, while others may not be as smooth. Without a centralized operator, people in the field have to juggle multiple tools instead of relying on a single solution for metric aggregation and execution control across different plugins.

Rewriting the operator in Go and simplifying it with a stronger focus on storage would be a huge help. An operator capable of running various storage benchmarking tools would be ideal. +1 to this idea!

[stale]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[ddomingu]

This issue is still relevant, but it seems there hasn't been enough traction or follow-up activity to move it forward

[jtaleric]

@ddomingu yeah, it is on the back-burner. Are you interested in working together on this?

[ddomingu]

@jtaleric I'm definitely interested, but I don't have the time or the technical background to lead this. However, I’d be happy to help with testing and provide feedback from an end-user perspective.

[jtaleric]

@ddomingu since FIO is of interest, checkout a new project I just kicked off. https://github.com/jtaleric/k8s-io

[ddomingu]

Hi @jtaleric, thanks for kicking off the new project, it looks really promising! I’ll try to get my hands dirty with it as soon as I can. If I want to suggest new functionality, should I open issues directly in the new project, or is there a preferred process for that? Thanks again!

[jtaleric]

yup!