From b8e461b186d4e0788a96ebf56b31f442716c33d5 Mon Sep 17 00:00:00 2001 From: Lenny Chen Date: Wed, 8 Oct 2025 15:57:16 -0700 Subject: [PATCH 1/4] docs: add worker deployments best practice --- docs/best-practices/cloud-access-control.mdx | 1 - docs/best-practices/managing-namespace.mdx | 1 - docs/best-practices/security-controls.mdx | 1 - docs/best-practices/worker.mdx | 41 ++++++++++++++++++++ docs/develop/worker-performance.mdx | 15 +++---- 5 files changed, 49 insertions(+), 10 deletions(-) create mode 100644 docs/best-practices/worker.mdx diff --git a/docs/best-practices/cloud-access-control.mdx b/docs/best-practices/cloud-access-control.mdx index c33d140fdd..9d39020443 100644 --- a/docs/best-practices/cloud-access-control.mdx +++ b/docs/best-practices/cloud-access-control.mdx @@ -4,7 +4,6 @@ title: Managing Temporal Cloud Access Control sidebar_label: Managing Cloud Access Control description: Best practices for managing access control, permissions, and user management in Temporal Cloud. toc_max_heading_level: 4 -hide_table_of_contents: true keywords: - temporal cloud - access control diff --git a/docs/best-practices/managing-namespace.mdx b/docs/best-practices/managing-namespace.mdx index ab804ffd86..3b8de22a25 100644 --- a/docs/best-practices/managing-namespace.mdx +++ b/docs/best-practices/managing-namespace.mdx @@ -4,7 +4,6 @@ title: Managing a Namespace sidebar_label: Managing a Namespace description: Best practices for managing Temporal Namespaces including configuration, retention, and optimization strategies. toc_max_heading_level: 4 -hide_table_of_contents: true keywords: - namespace management - temporal namespace diff --git a/docs/best-practices/security-controls.mdx b/docs/best-practices/security-controls.mdx index 01f221de8d..b5cb1405a5 100644 --- a/docs/best-practices/security-controls.mdx +++ b/docs/best-practices/security-controls.mdx @@ -4,7 +4,6 @@ title: Security Controls for Temporal Cloud sidebar_label: Security Controls for Cloud description: Best practices for implementing and managing security controls in Temporal Cloud environments. toc_max_heading_level: 4 -hide_table_of_contents: true keywords: - temporal cloud security - security controls diff --git a/docs/best-practices/worker.mdx b/docs/best-practices/worker.mdx new file mode 100644 index 0000000000..fd8c4cc4cd --- /dev/null +++ b/docs/best-practices/worker.mdx @@ -0,0 +1,41 @@ +--- +title: Worker deployment and performance +sidebar_label: Worker Deployment and Performance +description: Best practices for deploying and optimizing Temporal Workers for performance and reliability. +toc_max_heading_level: 4 +keywords: + - temporal worker + - worker deployment + - performance optimization + - best practices +tags: + - Best Practices + - Workers +--- + + This document outlines best practices for deploying and optimizing Workers to ensure high performance, reliability, and scalability. + +## Core tenets + +[Workers](../encyclopedia/workers/workers.mdx) are the execution layer of Temporal applications — they poll task queues, execute Workflows and Activities, and report results back to the Temporal Server. As such, Worker deployments have the following core tenets: + +- Stateless and ephemeral. Workers are independent pods/processes; all state lives in Temporal; Workers are designed to tolerate restarts and rescheduling. + +- Horizontally scalable. The number of worker replicas must be adjustable based on workload demands. + +- Observable and tunable. Effective Worker tuning requires various metrics, traces, and logs to be collected and acted on. + +These core tenets inform all best practices recommended in the following sections. + +## Deployment model + +This section covers best practices for Worker deployment models. + +### Scope each Worker pool to a single application and environment + +Because workers must be horizontally scalable, it is best to deploy them in pools. A worker pool is a number of workers that run a Temporal application. We recommend each pool be dedicated to a single application and environment. However, one application can have multiple pools, for example, to separate different types of workloads. + + +### Separate Workflow and Activity Worker pools if their resource needs differ significantly + +Even within a single Temporal application, there are often multiple Workflow and Activity types. If your application workloads are small and similar, a single Worker pool can handle all types. However, if your application has distinct workloads with different resource requirements or scaling characteristics, consider separating them into different Worker pools. \ No newline at end of file diff --git a/docs/develop/worker-performance.mdx b/docs/develop/worker-performance.mdx index e54a602a9e..0cfe8901fd 100644 --- a/docs/develop/worker-performance.mdx +++ b/docs/develop/worker-performance.mdx @@ -78,14 +78,14 @@ Available slot suppliers include: ### Worker tuning {#worker-tuning} -**Worker tuning** lets you manage and customize a Worker's runtime performance characteristics. -They use special types called **Worker tuners** that assign slot suppliers to various Task Types, including Worker, Activity, Nexus, and Local Activity Tasks. +Worker tuning is the process of defining customized slot suppliers for the different task slots of a Worker to fine-tune its performance. +You use special types called **Worker tuners** that assign slot suppliers to various Task Types, including Worker, Activity, Nexus, and Local Activity Tasks. -For more on how to configure and use Worker tuners, see [Worker runtime performance tuning](#worker-performance-tuning) below. +For more on how to configure and use Worker tuners, refer to [Worker runtime performance tuning](#worker-performance-tuning). :::caution -- Worker tuners supersede the existing `maxConcurrentXXXTask` style Worker options. +Worker tuners supersede the existing `maxConcurrentXXXTask` style Worker options. Using both styles will cause an error at Worker initialization time. ::: @@ -119,9 +119,10 @@ Temporal Cloud and Temporal Server 1.29.0 and higher have Eager Workflow Start a ::: -Eager Workflow Start is feature that reduces the time it takes to start a Workflow. -The target use case is short-lived Workflows that interact with other services using Local Activities, ideally initiating this interaction in the first Workflow Task, and deployed close to the Temporal Server. -These Workflows have a happy path that needs to initiate interactions within low tens of milliseconds, but they also want to take advantage of server-driven retries, and reliable compensation processes, for those less happy days. +Eager Workflow Start reduces the latency required to initiate a Workflow execution. +It is recommended for short-lived Workflows that use Local Activities to interact with external services, especially when these interactions are initiated in the first Workflow Task and the Workflow is deployed near the Temporal Server to minimize network delay. + +This feature is particularly beneficial for Workflows with a “happy path” that must begin external interactions within tens of milliseconds, while still relying on Temporal’s server-driven retries and compensation mechanisms to ensure reliability in failure scenarios. **Quick Start** From e48931068495033c9fd91c09971c7d2af301c621 Mon Sep 17 00:00:00 2001 From: Lenny Chen Date: Wed, 8 Oct 2025 16:31:22 -0700 Subject: [PATCH 2/4] docs: add worker best practices to sidebar --- docs/best-practices/worker.mdx | 18 ++++++++++++++++-- sidebars.js | 1 + 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/docs/best-practices/worker.mdx b/docs/best-practices/worker.mdx index fd8c4cc4cd..3bc63d7eff 100644 --- a/docs/best-practices/worker.mdx +++ b/docs/best-practices/worker.mdx @@ -19,7 +19,7 @@ tags: [Workers](../encyclopedia/workers/workers.mdx) are the execution layer of Temporal applications — they poll task queues, execute Workflows and Activities, and report results back to the Temporal Server. As such, Worker deployments have the following core tenets: -- Stateless and ephemeral. Workers are independent pods/processes; all state lives in Temporal; Workers are designed to tolerate restarts and rescheduling. +- Stateless and ephemeral. Even though Workers retain a cache to increase performance during execution of the same workloads, at their core they are stateless processes. All state that your applications rely on for durable execution lives in Temporal. Workers are designed to tolerate restarts and rescheduling. - Horizontally scalable. The number of worker replicas must be adjustable based on workload demands. @@ -38,4 +38,18 @@ Because workers must be horizontally scalable, it is best to deploy them in pool ### Separate Workflow and Activity Worker pools if their resource needs differ significantly -Even within a single Temporal application, there are often multiple Workflow and Activity types. If your application workloads are small and similar, a single Worker pool can handle all types. However, if your application has distinct workloads with different resource requirements or scaling characteristics, consider separating them into different Worker pools. \ No newline at end of file +Even within a single Temporal application, there are often multiple Workflow and Activity types. If your application workloads are small and similar, a single Worker pool can handle all types. However, if your application has distinct workloads with different resource requirements or scaling characteristics, consider separating them into different Worker pools. + + +### Use one Kubernetes pod per Worker + +Because workers are stateless and horizontally scalable, Kubernetes is a natural fit for deploying them. We recommend using one pod per Worker instance. This approach simplifies resource allocation, scaling, and monitoring. + +## Resource allocation and monitoring + +This section covers best practices for allocating resources to Workers and monitoring their performance. + +### Monitor both CPU and memory usage + + + diff --git a/sidebars.js b/sidebars.js index c6b0edcd82..d80e13acbc 100644 --- a/sidebars.js +++ b/sidebars.js @@ -613,6 +613,7 @@ module.exports = { "best-practices/managing-namespace", "best-practices/cloud-access-control", "best-practices/security-controls", + "best-practices/worker", ], }, { From 03a97afaddfae694e77cc97048f529b9a91c5d75 Mon Sep 17 00:00:00 2001 From: Lenny Chen Date: Mon, 20 Oct 2025 13:39:30 -0700 Subject: [PATCH 3/4] minor edit --- docs/best-practices/worker.mdx | 14 +++++++++----- .../styles/config/vocabularies/Temporal/accept.txt | 2 -- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/best-practices/worker.mdx b/docs/best-practices/worker.mdx index 3bc63d7eff..dc5526b637 100644 --- a/docs/best-practices/worker.mdx +++ b/docs/best-practices/worker.mdx @@ -17,9 +17,9 @@ tags: ## Core tenets -[Workers](../encyclopedia/workers/workers.mdx) are the execution layer of Temporal applications — they poll task queues, execute Workflows and Activities, and report results back to the Temporal Server. As such, Worker deployments have the following core tenets: +[Workers](../encyclopedia/workers/workers.mdx) are the execution layer of Temporal applications. They poll task queues, execute Workflows and Activities, and report results back to the Temporal Server. As such, Worker deployments have the following core tenets: -- Stateless and ephemeral. Even though Workers retain a cache to increase performance during execution of the same workloads, at their core they are stateless processes. All state that your applications rely on for durable execution lives in Temporal. Workers are designed to tolerate restarts and rescheduling. +- Stateless and ephemeral. Even though Workers retain a cache to increase performance during execution of the same workloads, at their core, workers are stateless processes. All state that your applications rely on for durable execution lives in Temporal. Workers are designed to tolerate restarts and rescheduling. - Horizontally scalable. The number of worker replicas must be adjustable based on workload demands. @@ -33,17 +33,17 @@ This section covers best practices for Worker deployment models. ### Scope each Worker pool to a single application and environment -Because workers must be horizontally scalable, it is best to deploy them in pools. A worker pool is a number of workers that run a Temporal application. We recommend each pool be dedicated to a single application and environment. However, one application can have multiple pools, for example, to separate different types of workloads. +Because workers must be horizontally scalable, it is best to deploy them in pools. A worker pool is a number of workers that run a Temporal Application. We recommend you dedicate each pool to a single application and environment. However, one application can have multiple pools. ### Separate Workflow and Activity Worker pools if their resource needs differ significantly -Even within a single Temporal application, there are often multiple Workflow and Activity types. If your application workloads are small and similar, a single Worker pool can handle all types. However, if your application has distinct workloads with different resource requirements or scaling characteristics, consider separating them into different Worker pools. +Even within a single Temporal Application, there are often multiple Workflow and Activity types. If your application workloads are small and similar, a single Worker pool can handle all types. However, if your application has distinct workloads with different resource requirements or scaling characteristics, consider separating them into different Worker pools. ### Use one Kubernetes pod per Worker -Because workers are stateless and horizontally scalable, Kubernetes is a natural fit for deploying them. We recommend using one pod per Worker instance. This approach simplifies resource allocation, scaling, and monitoring. +Because workers are stateless and horizontally scalable, Kubernetes is a natural fit for deploying them. If you use Kubernetes to deploy Workers, we recommend using one pod per Worker instance. This approach simplifies resource allocation, scaling, and monitoring. ## Resource allocation and monitoring @@ -51,5 +51,9 @@ This section covers best practices for allocating resources to Workers and monit ### Monitor both CPU and memory usage +Worker processes are constrained by both CPU and memory. Monitor both metrics to ensure that Workers have sufficient resources to handle their workloads. +## Starting points for production worker deployments + +While optimal Worker deployment configurations depend on your specific application workloads, the following starting points can help you get started: diff --git a/vale/styles/config/vocabularies/Temporal/accept.txt b/vale/styles/config/vocabularies/Temporal/accept.txt index 4b8333c5c3..c59bcb1cf4 100644 --- a/vale/styles/config/vocabularies/Temporal/accept.txt +++ b/vale/styles/config/vocabularies/Temporal/accept.txt @@ -2,8 +2,6 @@ Temporal's APIs Temporal Application application -ensure -Ensure extract sample samples From abd00c321511eb18fa347f3a61bd06d2b126127f Mon Sep 17 00:00:00 2001 From: Lenny Chen Date: Thu, 23 Oct 2025 13:41:45 -0700 Subject: [PATCH 4/4] docs: add sections to worker deployment --- docs/best-practices/worker.mdx | 60 +++++++++++++++++++++++----------- 1 file changed, 41 insertions(+), 19 deletions(-) diff --git a/docs/best-practices/worker.mdx b/docs/best-practices/worker.mdx index dc5526b637..008146c032 100644 --- a/docs/best-practices/worker.mdx +++ b/docs/best-practices/worker.mdx @@ -4,56 +4,78 @@ sidebar_label: Worker Deployment and Performance description: Best practices for deploying and optimizing Temporal Workers for performance and reliability. toc_max_heading_level: 4 keywords: - - temporal worker - - worker deployment - - performance optimization - - best practices + - temporal worker + - worker deployment + - performance optimization + - best practices tags: - Best Practices - Workers --- - This document outlines best practices for deploying and optimizing Workers to ensure high performance, reliability, and scalability. +This document outlines best practices for deploying and optimizing Workers to ensure high performance, reliability, and +scalability. ## Core tenets -[Workers](../encyclopedia/workers/workers.mdx) are the execution layer of Temporal applications. They poll task queues, execute Workflows and Activities, and report results back to the Temporal Server. As such, Worker deployments have the following core tenets: +[Workers](../encyclopedia/workers/workers.mdx) are the execution layer of Temporal applications. They poll task queues, +execute Workflows and Activities, and report results back to the Temporal Server. As such, Worker deployments have the +following core tenets: -- Stateless and ephemeral. Even though Workers retain a cache to increase performance during execution of the same workloads, at their core, workers are stateless processes. All state that your applications rely on for durable execution lives in Temporal. Workers are designed to tolerate restarts and rescheduling. +- Stateless and ephemeral. Even though Workers retain a cache to increase performance during execution of the same + workloads, at their core, workers are stateless processes. All state that your applications rely on for durable + execution lives in Temporal. Workers are designed to tolerate restarts and rescheduling. -- Horizontally scalable. The number of worker replicas must be adjustable based on workload demands. +- Horizontally scalable. The number of workers must be adjustable based on workload demands. -- Observable and tunable. Effective Worker tuning requires various metrics, traces, and logs to be collected and acted on. +- Observable and tunable. Effective Worker tuning requires various metrics, traces, and logs to be collected and acted + on. These core tenets inform all best practices recommended in the following sections. -## Deployment model +## Deployment model This section covers best practices for Worker deployment models. -### Scope each Worker pool to a single application and environment +### At least two workers per task queue -Because workers must be horizontally scalable, it is best to deploy them in pools. A worker pool is a number of workers that run a Temporal Application. We recommend you dedicate each pool to a single application and environment. However, one application can have multiple pools. +### Scope each Worker pool to a single application and environment +Because workers must be horizontally scalable, it is best to deploy them in pools. A worker pool is a number of workers +that run a Temporal Application. We recommend you dedicate each pool to a single application and environment. However, +one application can have multiple pools. ### Separate Workflow and Activity Worker pools if their resource needs differ significantly -Even within a single Temporal Application, there are often multiple Workflow and Activity types. If your application workloads are small and similar, a single Worker pool can handle all types. However, if your application has distinct workloads with different resource requirements or scaling characteristics, consider separating them into different Worker pools. - +Even within a single Temporal Application, there are often multiple Workflow and Activity types. If your application +workloads are small and similar, a single Worker pool can handle all types. However, if your application has distinct +workloads with different resource requirements or scaling characteristics, consider separating them into different +Worker pools. ### Use one Kubernetes pod per Worker -Because workers are stateless and horizontally scalable, Kubernetes is a natural fit for deploying them. If you use Kubernetes to deploy Workers, we recommend using one pod per Worker instance. This approach simplifies resource allocation, scaling, and monitoring. +Because workers are stateless and horizontally scalable, Kubernetes is a natural fit for deploying them. If you use +Kubernetes to deploy Workers, we recommend using one pod per Worker instance. This approach simplifies resource +allocation, scaling, and monitoring. ## Resource allocation and monitoring This section covers best practices for allocating resources to Workers and monitoring their performance. ### Monitor both CPU and memory usage - -Worker processes are constrained by both CPU and memory. Monitor both metrics to ensure that Workers have sufficient resources to handle their workloads. -## Starting points for production worker deployments +Worker processes are constrained by both CPU and memory. Monitor both metrics to ensure that Workers have sufficient +resources to handle their workloads. + +TODO: Steps to find out why they are high. -While optimal Worker deployment configurations depend on your specific application workloads, the following starting points can help you get started: +1. + +### Latency metrics inform resource allocation + +Monitor Worker latency metrics, such as Workflow and Activity task latencies, to identify bottlenecks. High latencies + +## Starting points for production worker deployments +While optimal Worker deployment configurations depend on your specific application workloads, the following starting +points can help you get started: