[IcebergIO] load tests #33728
[IcebergIO] load tests #33728
Conversation
ahmedabu98
force-pushed
the
iceberg_perf
branch
from
8693e5b to
e894a86
Compare
|
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
|
Assigning reviewers: R: @robertwb for label java. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
|
||
| int multiplier = this.startMultiplier; | ||
| long elapsedTimeMillis = timestamp.getMillis() - startTimesMillis; | ||
| long elapsedTimeMillis = Instant.now().getMillis() - startTimesMillis; |
There was a problem hiding this comment.
What are you trying to accomplish here? One side effect of this change is that now 2 different load tests will send different volumes of messages (e.g. if I'm slow processing messages, I'll get extra messages because more of them will have late timestamps).
I'm not super familiar with this code, so possible I'm misunderstanding the previous behavior, but growing load like this seems like it could be suboptimal
| readPipeline | ||
| .getOptions() | ||
| .as(DataflowPipelineOptions.class) | ||
| .setNumWorkers(configuration.numWorkers / 5); |
There was a problem hiding this comment.
Why is read so much lower than write?
|
|
||
| /** | ||
| * Determines whether to use Dataflow runner v2. If set to true, it uses SDF mode for reading | ||
| * from Kafka. Otherwise, Unbounded mode will be used. |
There was a problem hiding this comment.
I think this comment needs updated
|
waiting on author |
github-actions
bot
added
Next Action: Author
and removed
Next Action: Reviewers
slow-review
labels
|
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions. |
|
Hi @ahmedabu98, is this PR still valid? Thanks! |
|
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions. |
|
not stale |
3 participants
No description provided.