Skip to content

Conversation

@nodece
Copy link
Member

@nodece nodece commented Aug 24, 2024

Fix #4465

Motivaction

In SingleThreadExecutor, the runner drains all tasks from the queue into localTasks. Although the queue is empty in memory at this point, the tasks are still pending execution in localTasks—so logically, the queue is still "full."

Calling execute() during this phase should not enqueue a new runnable into the queue, as doing so would exceed the intended capacity. This can lead to increased memory usage and potential OutOfMemory (OOM) issues.

Changes

To address this, we introduce a variable to track the total number of pending runnables. This counter is used to control whether a new task should be added:

  • The counter is incremented when a runnable is added to the queue.

  • The counter is decremented when a runnable is actually executed.

This ensures accurate tracking of pending tasks and prevents overfilling the logical task queue.

@nodece
Copy link
Member Author

nodece commented Apr 9, 2025

@lhotari Could you have a chance to review?

Copy link
Member

@StevenLuMT StevenLuMT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Boundary value confirmation is required

@nodece nodece force-pushed the fix-SingleThreadExecutor-reject branch 2 times, most recently from f0c9bb2 to 7785861 Compare April 16, 2025 04:18
@nodece nodece requested review from StevenLuMT and dlg99 April 16, 2025 04:20
Copy link
Member

@StevenLuMT StevenLuMT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link

@xiezhx9 xiezhx9 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

squash into one commit might be better

@nodece
Copy link
Member Author

nodece commented Apr 16, 2025

squash into one commit might be better

The code owner can use squash merge to resolve this.

@nodece nodece requested a review from StevenLuMT April 17, 2025 03:59
@nodece nodece force-pushed the fix-SingleThreadExecutor-reject branch from b35429c to aaafec6 Compare April 22, 2025 02:45
@nodece
Copy link
Member Author

nodece commented Apr 23, 2025

@lhotari @merlimat Could you have a chance to review?

Copy link
Member

@jiazhai jiazhai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. LTGM. Good Catch.

@jiazhai jiazhai merged commit 39e7a91 into apache:master Apr 28, 2025
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The maxPendingAddRequestsPerThread configuration is inconsistent with the actual behavior, resulting in netty direct memory OOM

5 participants