Revise content to match new troubleshooting guidelines (#118033)

* Revise to match new guidelines

* Address review suggestions and comments

* Apply suggestions from review

Co-authored-by: shainaraskas <[email protected]>

* Apply suggestions from review

Co-authored-by: shainaraskas <[email protected]>

* Apply suggestions from review

Co-authored-by: shainaraskas <[email protected]>

* Apply suggestions from review

---------

Co-authored-by: shainaraskas <[email protected]>

Author: marciw

docs/reference/troubleshooting/common-issues/task-queue-backlog.asciidoc

@@ -1,103 +1,149 @@
 [[task-queue-backlog]]
-=== Task queue backlog
+=== Backlogged task queue
 
-A backlogged task queue can prevent tasks from completing and put the cluster
-into an unhealthy state. Resource constraints, a large number of tasks being
-triggered at once, and long running tasks can all contribute to a backlogged
-task queue.
+*******************************
+*Product:* Elasticsearch +
+*Deployment type:* Elastic Cloud Enterprise, Elastic Cloud Hosted, Elastic Cloud on Kubernetes, Elastic Self-Managed  +
+*Versions:* All
+*******************************
+
+A backlogged task queue can prevent tasks from completing and lead to an 
+unhealthy cluster state. Contributing factors include resource constraints, 
+a large number of tasks triggered at once, and long-running tasks.
 
 [discrete]
 [[diagnose-task-queue-backlog]]
-==== Diagnose a task queue backlog
+==== Diagnose a backlogged task queue
+
+To identify the cause of the backlog, try these diagnostic actions.
 
-**Check the thread pool status**
+* <<diagnose-task-queue-thread-pool>>
+* <<diagnose-task-queue-hot-thread>>
+* <<diagnose-task-queue-long-running-node-tasks>>
+* <<diagnose-task-queue-long-running-cluster-tasks>>
+
+[discrete]
+[[diagnose-task-queue-thread-pool]]
+===== Check the thread pool status
 
 A <<high-cpu-usage,depleted thread pool>> can result in
 <<rejected-requests,rejected requests>>. 
 
-Thread pool depletion might be restricted to a specific <<data-tiers,data tier>>. If <<hotspotting,hot spotting>> is occuring, one node might experience depletion faster than other nodes, leading to performance issues and a growing task backlog.
-
-You can use the <<cat-thread-pool,cat thread pool API>> to see the number of
-active threads in each thread pool and how many tasks are queued, how many
-have been rejected, and how many have completed.
+Use the <<cat-thread-pool,cat thread pool API>> to monitor
+active threads, queued tasks, rejections, and completed tasks:
 
 [source,console]
 ----
 GET /_cat/thread_pool?v&s=t,n&h=type,name,node_name,active,queue,rejected,completed
 ----
 
-The `active` and `queue` statistics are instantaneous while the `rejected` and
-`completed` statistics are cumulative from node startup.
+* Look for high `active` and `queue` metrics, which indicate potential bottlenecks
+and opportunities to <<reduce-cpu-usage,reduce CPU usage>>.
+* Determine whether thread pool issues are specific to a <<data-tiers,data tier>>.
+* Check whether a specific node's thread pool is depleting faster than others. This
+might indicate <<resolve-task-queue-backlog-hotspotting, hot spotting>>.
 
-**Inspect the hot threads on each node**
+[discrete]
+[[diagnose-task-queue-hot-thread]]
+===== Inspect hot threads on each node
 
-If a particular thread pool queue is backed up, you can periodically poll the
-<<cluster-nodes-hot-threads,Nodes hot threads>> API to determine if the thread
-has sufficient resources to progress and gauge how quickly it is progressing.
+If a particular thread pool queue is backed up, periodically poll the
+<<cluster-nodes-hot-threads,nodes hot threads API>> to gauge the thread's
+progression and ensure it has sufficient resources:
 
 [source,console]
 ----
 GET /_nodes/hot_threads
 ----
 
-**Look for long running node tasks**
+Although the hot threads API response does not list the specific tasks running on a thread, 
+it provides a summary of the thread's activities. You can correlate a hot threads response 
+with a <<tasks,task management API response>> to identify any overlap with specific tasks. For 
+example, if the hot threads response indicates the thread is `performing a search query`, you can 
+<<diagnose-task-queue-long-running-node-tasks,check for long-running search tasks>> using the task management API.
+
+[discrete]
+[[diagnose-task-queue-long-running-node-tasks]]
+===== Identify long-running node tasks
 
-Long-running tasks can also cause a backlog. You can use the <<tasks,task
-management>> API to get information about the node tasks that are running.
-Check the `running_time_in_nanos` to identify tasks that are taking an
-excessive amount of time to complete.
+Long-running tasks can also cause a backlog. Use the <<tasks,task
+management API>> to check for excessive `running_time_in_nanos` values:
 
 [source,console]
 ----
 GET /_tasks?pretty=true&human=true&detailed=true
 ----
 
-If a particular `action` is suspected, you can filter the tasks further. The most common long-running tasks are <<docs-bulk,bulk index>>- or search-related.
+You can filter on a specific `action`, such as <<docs-bulk,bulk indexing>> or search-related tasks.
+These tend to be long-running.
 
-* Filter for <<docs-bulk,bulk index>> actions:
+* Filter on <<docs-bulk,bulk index>> actions:
 +
 [source,console]
 ----
 GET /_tasks?human&detailed&actions=indices:data/write/bulk
 ----
 
-* Filter for search actions:
+* Filter on search actions:
 +
 [source,console]
 ----
 GET /_tasks?human&detailed&actions=indices:data/write/search
 ----
 
-The API response may contain additional tasks columns, including `description` and `header`, which provides the task parameters, target, and requestor. You can use this information to perform further diagnosis.
+Long-running tasks might need to be <<resolve-task-queue-backlog-stuck-tasks,canceled>>.
 
-**Look for long running cluster tasks**
+[discrete]
+[[diagnose-task-queue-long-running-cluster-tasks]]
+===== Look for long-running cluster tasks
 
-A task backlog might also appear as a delay in synchronizing the cluster state. You
-can use the <<cluster-pending,cluster pending tasks API>> to get information
-about the pending cluster state sync tasks that are running. 
+Use the <<cluster-pending,cluster pending tasks API>> to identify delays
+in cluster state synchronization: 
 
 [source,console]
 ----
 GET /_cluster/pending_tasks
 ----
 
-Check the `timeInQueue` to identify tasks that are taking an excessive amount 
-of time to complete.
+Tasks with a high `timeInQueue` value are likely contributing to the backlog and might
+need to be <<resolve-task-queue-backlog-stuck-tasks,canceled>>.
 
 [discrete]
 [[resolve-task-queue-backlog]]
-==== Resolve a task queue backlog
+==== Recommendations
+
+After identifying problematic threads and tasks, resolve the issue by increasing resources or canceling tasks.
 
-**Increase available resources** 
+[discrete]
+[[resolve-task-queue-backlog-resources]]
+===== Increase available resources
 
-If tasks are progressing slowly and the queue is backing up, 
-you might need to take steps to <<reduce-cpu-usage>>. 
+If tasks are progressing slowly, try <<reduce-cpu-usage,reducing CPU usage>>.
 
-In some cases, increasing the thread pool size might help.
-For example, the `force_merge` thread pool defaults to a single thread.
+In some cases, you might need to increase the thread pool size. For example, the `force_merge` thread pool defaults to a single thread. 
 Increasing the size to 2 might help reduce a backlog of force merge requests.
 
-**Cancel stuck tasks**
+[discrete]
+[[resolve-task-queue-backlog-stuck-tasks]]
+===== Cancel stuck tasks
+
+If an active task's <<diagnose-task-queue-hot-thread,hot thread>> shows no progress, consider <<task-cancellation,canceling the task>>.
+
+[discrete]
+[[resolve-task-queue-backlog-hotspotting]]
+===== Address hot spotting
+
+If a specific node's thread pool is depleting faster than others, try addressing
+uneven node resource utilization, also known as hot spotting.
+For details on actions you can take, such as rebalancing shards, see <<hotspotting>>.
+
+[discrete]
+==== Resources
+
+Related symptoms:
+
+* <<high-cpu-usage>>
+* <<rejected-requests>>
+* <<hotspotting>>
 
-If you find the active task's hot thread isn't progressing and there's a backlog, 
-consider canceling the task. 
+// TODO add link to standard Additional resources when that topic exists