Skip to content

Update Jetty thread pool metrics#18883

Merged
abhishekrb19 merged 5 commits intoapache:masterfrom
abhishekrb19:update-jetty-metrics
Jan 5, 2026
Merged

Update Jetty thread pool metrics#18883
abhishekrb19 merged 5 commits intoapache:masterfrom
abhishekrb19:update-jetty-metrics

Conversation

@abhishekrb19
Copy link
Contributor

@abhishekrb19 abhishekrb19 commented Jan 2, 2026

Jetty recommends getReadyThreads() and getUtilizedThreads() instead of getIdleThreads() and getBusyThreads(). The former (and the new metrics jetty/threadPool/ready and jetty/threadPool/utilized) are intended to capture the portion of the thread pool that is actually available for serving requests (short, transient jobs).
Otoh the latter also capture the state of internal system threads that are reserved and/or leased for internal Jetty mechanics.

I noticed on busy servers with sustained query load where the “busy” thread count appeared to be constantly saturated, likely because threads leased for thread pool maintenance were being deprioritized even though "isLowOnThreads" was not 1.

It's also useful to get the utilization rate of the thread pool that's serving requests to understand thread pool usage patterns for different servers: jetty/threadPool/utilizationRate

Also update the documentation for the new metrics and also fixes up descriptions for some existing Jetty metrics.

Release note

Add new Jetty thread pool metrics to capture request-serving thread statistics: jetty/threadPool/utilized, jetty/threadPool/ready and jetty/threadPool/utilizationRate.

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • been tested in a test Druid cluster.

Copy link
Contributor

@jtuglu1 jtuglu1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comments – thanks.

|`jetty/threadPool/busy`|Number of busy threads that has work to do from the worker queue.|Less than or equal to `threadPoolNumTotalThreads`.|
|`jetty/threadPool/isLowOnThreads`|A rough indicator of whether number of total workable threads allocated is enough to handle the works in the work queue.|0|
|`jetty/threadPool/total`|Number of total workable threads allocated. This includes internal threads and threads that are ready to serve requests.|Equals the total number of usable threads in the Jetty thread pool (idle + busy).|
|`jetty/threadPool/ready`| Number of threads that are ready to serve requests. | Equals the total number of usable threads in the Jetty thread pool to serve requests (idle + busy).|
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: spacing on the markdown |.

|`jetty/threadPool/isLowOnThreads`|A rough indicator of whether number of total workable threads allocated is enough to handle the works in the work queue.|0|
|`jetty/threadPool/total`|Number of total workable threads allocated. This includes internal threads and threads that are ready to serve requests.|Equals the total number of usable threads in the Jetty thread pool (idle + busy).|
|`jetty/threadPool/ready`| Number of threads that are ready to serve requests. | Equals the total number of usable threads in the Jetty thread pool to serve requests (idle + busy).|
|`jetty/threadPool/utilized`| Number of threads currently in use by the thread pool to serve requests. | Typically well below ready.|
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we frame this as an inequality?

@abhishekrb19 abhishekrb19 merged commit 37f88fa into apache:master Jan 5, 2026
43 checks passed
@abhishekrb19 abhishekrb19 deleted the update-jetty-metrics branch January 5, 2026 22:48
@kgyrtkirk kgyrtkirk added this to the 36.0.0 milestone Jan 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants