Skip to content

Conversation

@ballard26
Copy link
Contributor

@ballard26 ballard26 commented Dec 1, 2022

This PR extends the many_partitions test to use transactions in background traffic during a restart stress test. Not sure if this is the best place to insert transactions into our scale tests. However, I thought it'd be a good starting place for a discussion about where adding in transactions would be appropriate.

This PR requires redpanda-data/kgo-verifier#16 to be merged for the changed test to pass.

Backports Required

  • none - not a bug fix
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v22.3.x
  • v22.2.x
  • v22.1.x

UX Changes

N/a

Release Notes

N/a

@ballard26 ballard26 requested review from bharathv and jcsp December 1, 2022 04:58
raise
# non-txn background traffic
repeater_msg_size = 16384
with repeater_traffic(context=self._ctx,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit odd to pick out the restart stress portion of the test and only run that bit with transactions.

The approach for compaction is to have a top level set of repeater_kwargs, and two different test cases (compacted vs. non) -- can we do the same for transactions? Maybe rather than each feature independently, we should have just two variants of the test: vanilla (no compaction or transactions) and "fully loaded" (compaction, transactions enabled).

I think that'll be useful in identifying which failures result from issues in the primary data path, vs. which ones relate to particular optional features.

@jcsp
Copy link
Contributor

jcsp commented Dec 1, 2022

Not sure where you're running this in development, but would be good to try it on i3en.6xlarge instances. The nightly runs are on i3en.xlarge for cost, but that's only ~10k partitions, the i3en.6xlarge instances will get you up to the HARD_PARTITION_LIMIT in the test.

When you try the version with+without transactions, it may be that we get an interesting result in terms of what the hard partition limit can be with transactions on vs. off.

@bharathv bharathv removed their request for review July 14, 2025 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants