Merge pull request #203 from ut-karsh/patch-1

AnnaMHuff · web-flow · commit 5f9f879c3d82 · 2024-10-15T17:30:27.000-06:00
Update container FAQ concurrency
diff --git a/articles/ai-services/speech-service/speech-container-faq.yml b/articles/ai-services/speech-service/speech-container-faq.yml
@@ -55,7 +55,7 @@ sections:
           
           > 2 (clusters) * 5 (VMs per cluster) * $0.3528/hour * 365 (days) * 24 (hours) = $31 K / year
           
-          When mapping to physical machine, a general estimation is 1 vCPU = 1 Physical CPU Core. In reality, 1vCPU is more powerful than a single core.
+          When mapping to a physical machine, a general estimation is 1 vCPU = 1 Physical CPU Core. In reality, 1vCPU is more powerful than a single core.
           
           For on-premises, all of these extra factors come into play:
           
@@ -150,57 +150,66 @@ sections:
           
           
       - question: |
-          What are the recommended resources, CPU and RAM; for 50 concurrent requests?
+          What are the recommended CPU and RAM resources for Speech containers?
         answer: |
-          How many concurrent requests will a 4-core, 8-GB RAM handle? If we have to serve for example, 50 concurrent requests, how many Core and RAM is recommended?
-          
-          At real-time, 8 with our latest `en-US`, so we recommend using more docker containers beyond six concurrent requests. Beyond 16 cores it becomes nonuniform memory access (NUMA) node sensitive. The following table describes the minimum and recommended allocation of resources for each Speech container.
 
-          # [Speech-to-text](#tab/stt)
+          # [Speech to text](#tab/stt)
 
-          The following table describes the minimum and recommended allocation of resources
+          The following table describes the minimum and recommended allocation of resources for Speech to text.
           
-          | Container | Minimum | Recommended | Speech Model |
-          |-----------|---------|-------------|--------------|
-          | Speech to text | 4 core, 4-GB memory | 8 core, 8-GB memory | +4 to 8 GB memory |
+          | Container | Minimum | Recommended | Speech Model | Concurrency Limit |
+          |-----------|---------|-------------|--------------|-------------------|
+          | Speech to text | 4 core, 4-GB memory | 8 core, 8-GB memory | +4 to 8 GB memory | 2 sessions per core |
           
           > [!NOTE]
           > The minimum and recommended allocations are based on Docker limits, *not* the host machine resources.
           > For example, speech to text containers memory map portions of a large language model. We recommend that the entire file should fit in memory. You need to add an extra 4 to 8 GB to load the speech models (see above table).
           > Also, the first run of either container might take longer because models are being paged into memory.
+
+          - For files, the throttling will be in the Speech SDK, at 2x. The first 5 seconds of audio aren't throttled.
+          - For speech to text, the decoder is capable of doing about 2-3x real-time. That's why we don't recommend keeping more than two active connections per core. The extreme side would be to put about 10 decoders at 2x real-time in an eight-core machine like `DS13_V2`. For the container version 1.3 and later, there's a param you could try setting `-e DECODER_MAX_COUNT=20`.
+          - For microphone, speech to text happens at 1x real-time. The overall usage should be at about one core for a single recognition.
+          - Beyond 16 cores the system becomes nonuniform memory access (NUMA) node sensitive.
           
-          # [Custom speech-to-text](#tab/cstt)
+          # [Custom speech to text](#tab/cstt)
 
-          The following table describes the minimum and recommended allocation of resources
+          The following table describes the minimum and recommended allocation of resources for Custom speech to text.
           
-          | Container | Minimum | Recommended | Speech Model |
-          |-----------|---------|-------------|--------------|
-          | Custom speech to text | 4 core, 4-GB memory | 8 core, 8-GB memory | +4 to 8 GB memory |
+          | Container | Minimum | Recommended | Speech Model | Concurrency Limit |
+          |-----------|---------|-------------|--------------|-------------------|
+          | Custom speech to text | 4 core, 4-GB memory | 8 core, 8-GB memory | +4 to 8 GB memory | 2 sessions per core |
           
           > [!NOTE]
           > The minimum and recommended allocations are based on Docker limits, *not* the host machine resources.
           > For example, speech to text containers memory map portions of a large language model. We recommend that the entire file should fit in memory. You need to add an extra 4 to 8 GB to load the speech models (see above table).
           > Also, the first run of either container might take longer because models are being paged into memory.
+
+          - For files, the throttling will be in the Speech SDK, at 2x. The first 5 seconds of audio aren't throttled.
+          - For speech to text, the decoder is capable of doing about 2-3x real-time. That's why we don't recommend keeping more than two active connections per core. The extreme side would be to put about 10 decoders at 2x real-time in an eight-core machine like `DS13_V2`. For the container version 1.3 and later, there's a param you could try setting `-e DECODER_MAX_COUNT=20`.
+          - For microphone, speech to text happens at 1x real-time. The overall usage should be at about one core for a single recognition.
+          - Beyond 16 cores the system becomes nonuniform memory access (NUMA) node sensitive.
           
           # [Neural Text-to-speech](#tab/tts)
 
-          The following table describes the minimum and recommended allocation of resources
+          The following table describes the minimum and recommended allocation of resources for Neural text to speech.
           
-          | Container | Minimum | Recommended |
-          |-----------|---------|-------------|
-          | Neural text to speech | 6 core, 12-GB memory | 8 core, 16-GB memory |
+          | Container | Minimum | Recommended | Concurrency Limit |
+          |-----------|---------|-------------|-------------------|
+          | Neural text to speech | 6 core, 12-GB memory | 8 core, 16-GB memory \n 8 core, 24-GB memory (multilingual voices) | 5 requests per 8 core CPU |
           
           > [!NOTE]
           > The minimum and recommended allocations are based on Docker limits, *not* the host machine resources.
           > Also, the first run of either container might take longer because models are being paged into memory.
+
+          - Real-time performance varies depending on concurrency. With a concurrency of 1, an NTTS container instance can achieve 10x real-time performance. However, when concurrency increases to 5, real-time performance drops to 3x or lower. We recommended sending less than 5 concurrent requests in one container. Start more containers for increased concurrency.
           
           # [Speech language identification](#tab/lid)
 
-          The following table describes the minimum and recommended allocation of resources
+          The following table describes the minimum and recommended allocation of resources for Language Identification.
           
-          | Container | Minimum | Recommended |
-          |-----------|---------|-------------|
-          | Speech language identification | 1 core, 1-GB memory | 1 core, 1-GB memory |
+          | Container | Minimum | Recommended | Concurrency Limit |
+          |-----------|---------|-------------|-------------------|
+          | Speech language identification | 1 core, 1-GB memory | 1 core, 1-GB memory | 2 sessions per core |
           
           > [!NOTE]
           > The minimum and recommended allocations are based on Docker limits, *not* the host machine resources.
@@ -209,13 +218,10 @@ sections:
           ***
 
           - Each core must be at least 2.6 GHz or faster.
-          - For files, the throttling will be in the Speech SDK, at 2x. The first 5 seconds of audio aren't throttled.
-          - The decoder is capable of doing about 2-3x real-time. For this, the overall CPU usage will be close to two cores for a single recognition. That's why we don't recommend keeping more than two active connections, per container instance. The extreme side would be to put about 10 decoders at 2x real-time in an eight-core machine like `DS13_V2`. For the container version 1.3 and later, there's a param you could try setting `-e DECODER_MAX_COUNT=20`.
-          - For microphone, it is at 1x real-time. The overall usage should be at about one core for a single recognition.
          
-          Consider the total number of hours of audio you have. If the number is large, to improve reliability and availability, we suggest running more instances of containers, either on a single box or on multiple boxes, behind a load balancer. Orchestration could be done using Kubernetes (K8S) and Helm, or with Docker compose.
+          Consider the total number of hours of audio you have, and your expected time to complete the task. If the number is large, to improve reliability and availability, we suggest running more instances of containers, either on a single box or on multiple boxes, behind a load balancer. Orchestration could be done using Kubernetes (K8S) and Helm, or with Docker compose.
          
-          As an example, to handle 1000 hours/24 hours, we have tried setting up 3-4 VMs, with 10 instances/decoders per VM.
+          As an example, we have performed speech to text on 1000 hours of audio within 24 hours with 4-5 VMs and 10 instances/decoders per VM.
 
       - question: |
           Does the Speech container support punctuation?
