[Umbrella] Performance Optimization planned for v1.17

[RainbowMango]

Goal

Track and deliver the performance optimizations planned for the 1.17 release.

Action Items

1. ControllerPriorityQueue feature

• Wait for bugfixes in controller-runtime, and bump to a new version
  • 🐛 Priorityqueue: Do FIFO ordering within priorities and not across kubernetes-sigs/controller-runtime#3408
  • ✨ Use a buffer to optimize priority queue AddWithOpts performance kubernetes-sigs/controller-runtime#3415
• Move feature ControllerPriorityQueue to beta(enabled by default). (@zach593, Feat: Promote ControllerPriorityQueue to beta #7200)
  • This feature was introduced in release 1.15 and has been verified on master since then.
  • References: feat: introduce ControllerPriorityQueue feature gate #6662

2. Performance benchmark infrastructure

• Setup performance testing workflow. (@XiShanYongYe-Chang, Add performance test and comparison pipeline for Karmada #7180)

3. ResourceBinding to Work synchronization(TBD)
Based on the discussion at #7062

4. Lua interpreter VM pool size

• Reference: feat: expose lua vm pool size as flag #7049

5. Update object status

• AggregatedStatus in rb status may be deleted #6858
• feat: improve deployment creation performance by using informer cache… #7128

6. Summary

• Publish a short perf improvement summary for 1.17.

[sbueringer]

@RainbowMango if "One more PR hasn't been sent out yet" is about a PR for controller-runtime PQ can you give us a heads up what it's about? Thx!

[RainbowMango]

Hey @sbueringer, that's another issue found by @zach593 (author of kubernetes-sigs/controller-runtime#3408).

@zach593 can you briefly describe the issue here?

[sbueringer]

Thx, yup a short description would be helpful. We're close to the next CR minor release so it would be good to know about it already :)

P.S. Thx for the really helpful feedback so far!

cc @alvaroaleman (just for awareness)

[zach593]

Simply put, when there are many items and many workers, spin() can take a relatively long time to execute. This negatively impacts the write throughput of AddWithOpts(), causing items to get stuck in the processorListener ring buffer. The reason is that they share the same lock: spin() processes many items in each invocation, whereas the event handler that calls AddWithOpts() only processes one item at a time.

I am working on the corresponding optimization. The code itself is relatively straightforward, but making it compatible with the existing test cases takes a bit of extra time. During this process, kubernetes-sigs/controller-runtime#3408 also came up.

I will submit the corresponding code ASAP.

[alvaroaleman]

@zach593 I think kubernetes-sigs/controller-runtime#3416 should address the perf issues. Do you have any way to test for them and/or a flamegraph of the priorityqueue when you see these issues?

[mohamedawnallah]

2. Performance benchmark infrastructure(TBD)

Hi @RainbowMango, I have worked recently on designing and developing a performance benchmarking framework from scratch for a previous project in Golang. The benchmarks there ended up being runnable on free GitHub hosted runners. I would love to bring this experience to Karmada. Anything is up for community contributions in that area? I would like to help

Concrete Examples:
https://redirect.github.com/btcsuite/btcwallet/issues/1066
https://redirect.github.com/btcsuite/btcwallet/issues/1086
https://redirect.github.com/btcsuite/btcwallet/issues/1087
https://redirect.github.com/btcsuite/btcwallet/issues/1088
https://redirect.github.com/btcsuite/btcwallet/issues/1117

It was also helpful for detecting performance regressions:
https://redirect.github.com/btcsuite/btcwallet/pull/1050#discussion_r2352601998
https://redirect.github.com/btcsuite/btcwallet/pull/1050#discussion_r2366836443

[RainbowMango]

Thank you @mohamedawnallah for the information ~ cc @XiShanYongYe-Chang, who is working on the benchmark thing recently.

[zach593]

Wait for bugfixes in controller-runtime,

FYI, this work has already been addressed by kubernetes-sigs/controller-runtime#3408, kubernetes-sigs/controller-runtime#3415, and kubernetes-sigs/controller-runtime#3416.

[RainbowMango]

Great Job! I see they are included in v0.23.0.

[XiShanYongYe-Chang]

Hi @mohamedawnallah, I look the performance testing you provided for another project, and it appears their focus is on function-level benchmarking. Am i right?

The task described in this issue should be about a performance testing benchmark for the Karmada system, such as collecting metrics from various components under large-scale application distribution scenarios.

Perhaps we can communicate more on Slack.

[mohamedawnallah]

Hi @mohamedawnallah, I look the performance testing you provided for another project, and it appears their focus is on function-level benchmarking. Am i right?

The focus was on performance testing, but a side-benefit of this was functional testing e.g fuzzing inputs generated during benchmarking etc. Copying example I worked on:

Allocations per Operation

Execution Time (ns/op)

Memory Usage (B/op)

Those three performance metrics (Allocations per Operation/Execution Time (ns/op)/Memory Usage (B/op)) are provided natively by Golang https://pkg.go.dev/testing#hdr-Benchmarks

One thing also in that benchmarking framework, we can look for statistical significance, so perhaps we don't need to stress test it to evaluate performance we can evaluate performance even on small dataset.

But yeah, I think a relatively straightforward start is just doing stress testing first, and then those stress tests can be shaped/transformed to fast running benchmarks to be running in CI later if it is a desirable goal

Perhaps we can communicate more on Slack.

Sounds good. Sent a message there~

PS: Those go benchmark result visualizations generated by https://github.com/goptics/vizb

EDIT: Also I started to think that doing benchmarking here in Karmada gonna be a bit more involved since it involves network traffic etc not just on a single process level