Merge pull request #4945 from eric-urban/eur/contextualdata-from-fei

add contextual training set releated doc

Author: v-ccolin

articles/ai-services/speech-service/how-to-custom-voice-training-data.md

@@ -22,20 +22,22 @@ When you're ready to create a custom Text to speech voice for your application,
 
 A voice training dataset includes audio recordings, and a text file with the associated transcriptions. Each audio file should contain a single utterance (a single sentence or a single turn for a dialog system), and be less than 15 seconds long.
 
-In some cases, you might not have the right dataset ready. You can test the custom neural voice training with available audio files, short or long, with or without transcripts. 
+In some cases, you might not have the right dataset ready. You can test the custom neural voice training with available audio files, short or long, with or without transcripts.
 
 This table lists data types and how each is used to create a custom Text to speech voice model.
 
-| Data type | Description | When to use | Extra processing required |
-| --------- | ----------- | ----------- | ------------------------------ |
-| [Individual utterances + matching transcript](#individual-utterances--matching-transcript) | A collection (.zip) of audio files (.wav) as individual utterances. Each audio file should be 15 seconds or less in length, paired with a formatted transcript (.txt). | Professional recordings with matching transcripts | Ready for training. |
-| [Long audio + transcript](#long-audio--transcript-preview) | A collection (.zip) of long, unsegmented audio files (.wav or .mp3, longer than 20 seconds, at most 1,000 audio files), paired with a collection (.zip) of transcripts that contains all spoken words. | You have audio files and matching transcripts, but they aren't segmented into utterances. | Segmentation (using batch transcription).<br>Audio format transformation wherever required. |
-| [Audio only (Preview)](#audio-only-preview) | A collection (.zip) of audio files (.wav or .mp3, at most 1,000 audio files) without a transcript. | You only have audio files available, without transcripts. | Segmentation + transcript generation (using batch transcription).<br>Audio format transformation wherever required.|
+| Data type | Description | When to use | Extra processing required | Processed as |
+| --------- | ----------- | ----------- | ------------------------------ | ---------------- |
+| [Individual utterances + matching transcript](#individual-utterances--matching-transcript) | A collection (.zip) of audio files (.wav) as individual utterances. Each audio file should be 15 seconds or less in length, paired with a formatted transcript (.txt). | Professional recordings with matching transcripts | Ready for training. | Segmented |
+| [Long audio + transcript](#long-audio--transcript-preview) | A collection (.zip) of long, unsegmented audio files (.wav or .mp3, longer than 20 seconds, at most 1,000 audio files), paired with a collection (.zip) of transcripts that contains all spoken words. | You have audio files and matching transcripts, but they aren't segmented into utterances. | Segmentation (using batch transcription).<br>Audio format transformation wherever required. | Segmented, Contextual |
+| [Audio only (Preview)](#audio-only-preview) | A collection (.zip) of audio files (.wav or .mp3, at most 1,000 audio files) without a transcript. | You only have audio files available, without transcripts. | Segmentation + transcript generation (using batch transcription).<br>Audio format transformation wherever required.| Segmented, Contextual |
 
 Files should be grouped by type into a dataset and uploaded as a zip file. Each dataset can only contain a single data type.
 
 > [!NOTE]
 > The maximum number of datasets allowed to be imported per subscription is 500 zip files for standard subscription (S0) users.
+>
+> Processed as Contextual would retain the audio as a whole to keep the contextual information for more natural intonations.
 
 ## Individual utterances + matching transcript
 
@@ -87,10 +89,16 @@ It's important that the transcripts are 100% accurate transcriptions of the corr
 ## Long audio + transcript (Preview)
 
 > [!NOTE]
-> For **Long audio + transcript (Preview)**, only these languages are supported: Chinese (Mandarin, Simplified), Chinese (Cantonese, Traditional), Chinese (Taiwanese Mandarin), English (India), English (United Kingdom), English (United States), French (France), German (Germany), Italian (Italy), Japanese (Japan), Portuguese (Brazil), Spanish (Spain) and Spanish (Mexico).
+> For **Long audio + transcript (Preview)**, only these languages are supported: Chinese (Mandarin, Simplified), Chinese (Cantonese, Traditional), Chinese (Taiwanese Mandarin), English (India), English (United Kingdom), English (United States), French (France), German (Germany), Hindi (India), Italian (Italy), Japanese (Japan), Portuguese (Brazil), Spanish (Spain) and Spanish (Mexico).
+>
+> Processed as Contextual is currently only available for Chinese (Mandarin, Simplified) and English (United States).
 
 In some cases, you might not have segmented audio available. The Speech Studio can help you segment long audio files and create transcriptions. The long-audio segmentation service uses the [Batch Transcription API](batch-transcription.md) feature of speech to text.
 
+The service offers two processing modes:
+- **Segmented**: The default processing mode that works with all supported languages
+- **Contextual**: An enhanced mode that retains the audio as a whole to keep the contextual information for more natural intonations.
+
 During the processing of the segmentation, your audio files and the transcripts are also sent to the custom speech service to refine the recognition model so the accuracy can be improved for your data. No data is retained during this process. After the segmentation is done, only the utterances segmented and their mapping transcripts will be stored for your downloading and training.
 
 ### Audio data for Long audio + transcript
@@ -103,7 +111,7 @@ Follow these guidelines when preparing audio for segmentation.
 | File name	|  File name characters supported by Windows OS, with .wav extension. <br>The characters `\ / : * ? " < > \|` aren't allowed. <br>It can't start or end with a space, and can't start with a dot. <br>No duplicate file names allowed. |
 | Sampling rate	| 24 KHz and higher required when creating a custom neural voice. |
 | Sample format |RIFF(.wav): PCM, at least 16-bit.<br/><br/>mp3: At least 256 KBps bit rate.|
-| Audio length | Longer than 20 seconds |
+| Audio length | Longer than 30 seconds |
 | Archive format | .zip |
 | Maximum archive size | 2048 MB, at most 1,000 audio files included |
 
@@ -133,9 +141,15 @@ After your dataset is successfully uploaded, we'll help you segment the audio fi
 ## Audio only (Preview)
 
 > [!NOTE]
-> For **Audio only (Preview)**, only these languages are supported: Chinese (Mandarin, Simplified), Chinese (Cantonese, Traditional), Chinese (Taiwanese Mandarin), English (India), English (United Kingdom), English (United States), French (France), German (Germany), Italian (Italy), Japanese (Japan), Portuguese (Brazil), Spanish (Spain) and Spanish (Mexico). 
+> For **Audio only (Preview)**, only these languages are supported: Chinese (Mandarin, Simplified), Chinese (Cantonese, Traditional), Chinese (Taiwanese Mandarin), English (India), English (United Kingdom), English (United States), French (France), German (Germany), Hindi (India), Italian (Italy), Japanese (Japan), Portuguese (Brazil), Spanish (Spain) and Spanish (Mexico).
+>
+> Processed as Contextual is currently only available for Chinese (Mandarin, Simplified) and English (United States).
+
+If you don't have transcriptions for your audio recordings, use the **Audio only** option to upload your data. Our system can help you segment and transcribe your audio files.
 
-If you don't have transcriptions for your audio recordings, use the **Audio only** option to upload your data. Our system can help you segment and transcribe your audio files. Keep in mind, this service is charged toward your speech to text subscription usage.
+The service offers two processing modes:
+- **Segmented**: The default processing mode that works with all supported languages
+- **Contextual**: An enhanced mode that retains the audio as a whole to keep the contextual information for more natural intonations.
 
 Follow these guidelines when preparing audio.
 
@@ -154,7 +168,7 @@ Follow these guidelines when preparing audio.
 >
 > Segmented utterances should ideally be between 5 and 15 seconds long. For optimal segmentation results, it is recommended to include natural pauses of 0.5 to 1 second every 5 to 15 seconds of speech, preferably at the end of phrases or sentences.
 
-All audio files should be grouped into a zip file. Once your dataset is successfully uploaded, the Speech service helps you segment the audio file into utterances based on our speech batch transcription service. Unique IDs are assigned to the segmented utterances automatically. Matching transcripts are generated through speech recognition. All .mp3 files will be transformed into the .wav format after processing. You can check the segmented utterances and the matching transcripts by downloading the dataset.
+All audio files should be grouped into a zip file. Once your dataset is successfully uploaded, the Speech service helps you segment the audio file into utterances based on our speech batch transcription service. You can select either the Standard or Contextual processing mode, depending on your language and requirements. Unique IDs are assigned to the segmented utterances automatically. Matching transcripts are generated through speech recognition. All .mp3 files will be transformed into the .wav format after processing. You can check the segmented utterances and the matching transcripts by downloading the dataset.
 
 ## Next steps
 

articles/ai-services/speech-service/includes/how-to/professional-voice/create-training-set/speech-studio.md

@@ -24,9 +24,12 @@ When you're ready to upload your data, go to the **Prepare training data** tab t
 To upload training data, follow these steps:
 
 1. Sign in to the [Speech Studio](https://aka.ms/speechstudio/customvoice).
-1. Select **Custom voice** > Your project name > **Prepare training data** > **Upload data**. 
+1. Select **Custom voice** > Your project name > **Prepare training data** > **Upload data**.
 1. In the **Upload data** wizard, choose a [data type](../../../../how-to-custom-voice-training-data.md) and then select **Next**.
 1. Select local files from your computer or enter the Azure Blob storage URL to upload data.
+1. If you selected **Long audio + transcript** or **Audio only** data types in contextual support projects, you'll see an option to choose the processing mode:
+   - **Processed as Contextual**: Processes audio while preserving contextual information for enhanced conversational abilities and more natural speech patterns.
+   - **Segmented**: Processes audio and transcript into individual utterances using standard segmentation.
 1. Under **Specify the target training set**, select an existing training set or create a new one. If you created a new training set, make sure it's selected in the drop-down list before you continue.
 1. Select **Next**.
 1. Enter a name and description for your data and then select **Next**.
@@ -36,6 +39,8 @@ To upload training data, follow these steps:
 > Duplicate IDs are not accepted. Utterances with the same ID will be removed.
 > 
 > Duplicate audio names are removed from the training. Make sure the data you select don't contain the same audio names within the .zip file or across multiple .zip files. If utterance IDs (either in audio or script files) are duplicates, they're rejected.
+>
+> A training set can only contain data processed in the same mode. For example, if you upload data with **Processed as Contextual** mode to a training set, all subsequent uploads to that same training set must also use the **Processed as Contextual** mode.
 
 Data files are automatically validated when you select **Submit**. Data validation includes series of checks on the audio files to verify their file format, size, and sampling rate. If there are any errors, fix them and submit again. 
 
@@ -45,7 +50,9 @@ After you upload the data, you can check the details in the training set detail
 
 After upload, you can check the data details of the training set. Before continuing to [train your voice model](../../../../professional-voice-train-voice.md), you should try to resolve any data issues.
 
-You can identify and resolve data issues per utterance in [Speech Studio](https://aka.ms/custom-voice-portal). 
+You can identify and resolve data issues per utterance in [Speech Studio](https://aka.ms/custom-voice-portal).
+
+#### Processed as Segmented
 
 1. On the detail page, go to the **Accepted data** or **Rejected data** page. Select individual utterances you want to change, then select **Edit**.
 
@@ -77,6 +84,38 @@ You can identify and resolve data issues per utterance in [Speech Studio](https:
 
    You can also delete utterances with issues by selecting them and clicking **Delete**.
 
+
+#### Processed as Contextual
+
+Unlike processed as segmented, **Processed as Contextual** preserves the original audio files and generates corresponding contextual information.
+
+1. On the detail page, click on individual utterances with "Number Of issues".
+
+   :::image type="content" source="../../../../media/custom-voice/contextual-training/contextual-details.png" alt-text="Screenshot of contextual utterances details page.":::
+
+   Contextual information is presented as segments. Select the segment you want to modify, then click the **Edit** button.
+   
+   :::image type="content" source="../../../../media/custom-voice/contextual-training/contextual-segments.png" alt-text="Screenshot of contextual segments to be displayed.":::
+
+   You can choose which data issues to be displayed based on your criteria.
+   
+   :::image type="content" source="../../../../media/custom-voice/cnv-issues-display-criteria.png" alt-text="Screenshot of choosing which data issues to be displayed.":::
+
+1. Edit the transcript in the text box according to the issue description, then select Done.
+
+   :::image type="content" source="../../../../media/custom-voice/contextual-training/contextual-edit-segment.png" alt-text="Screenshot of selecting Done button after editing transcript.":::
+
+1. After you've made changes to your data, you need to check the data quality by clicking Analyze data before using this dataset for training.
+
+   You can't select this training set for training model before the analysis is complete.
+
+   :::image type="content" source="../../../../media/custom-voice/cnv-edit-trainingset-analyze.png" alt-text="Screenshot of selecting Analyze data on Data details page.":::
+
+> [!NOTE]
+> Deleting a segment of contextual information will not exclude that content from training. Only delete segments when their information is already included or will be included in adjacent segments.
+>
+> Rejected segments will not be used for training.
+
 ### Typical data issues
 
 The issues are divided into three types. Refer to the following tables to check the respective types of errors. 

articles/ai-services/speech-service/includes/how-to/professional-voice/train-voice/bilingual-training.md

@@ -12,14 +12,15 @@ ms.custom: include
 If you select the **Neural** training type, you can train a voice to speak in multiple languages. The `zh-CN`, `zh-HK`, and `zh-TW` locales support bilingual training for the voice to speak both Chinese and English. Depending in part on your training data, the synthesized voice can speak English with an English native accent or English with the same accent as the training data.
 
 > [!NOTE]
-> To enable a voice in the `zh-CN` locale to speak English with the same accent as the sample data, you should choose `Chinese (Mandarin, Simplified), English bilingual` when creating a project or specify the `zh-CN (English bilingual)` locale for the training set data via REST API.
+> To enable a voice in the `zh-CN` locale to speak English with the same accent as the sample data, you should upload English data to a **Contextual** training set, or choose `Chinese (Mandarin, Simplified), English bilingual` when creating a project or specify the `zh-CN (English bilingual)` locale for the training set data via REST API.
+>
+> In your contextual training set, include at least 100 sentences or 10 minutes of English content and do not exceed the amount of Chinese content.
 
 The following table shows the differences among the locales:
 
-| Speech Studio locale | REST API locale | Bilingual support | 
+| Speech Studio locale | REST API locale | Bilingual support |
 |:------------- |:------- |:-------------------------- |
-| `Chinese (Mandarin, Simplified)` | `zh-CN` |If your sample data includes English, the synthesized voice speaks English with an English native accent, instead of the same accent as the sample data, regardless of the amount of English data. | 
+| `Chinese (Mandarin, Simplified)` | `zh-CN` |If your sample data includes English, the synthesized voice speaks English with an English native accent, instead of the same accent as the sample data, regardless of the amount of English data. |
 | `Chinese (Mandarin, Simplified), English bilingual` | `zh-CN (English bilingual)` |If you want the synthesized voice to speak English with the same accent as the sample data, we recommend including over 10% English data in your training set. Otherwise, the English speaking accent might not be ideal. |
-| `Chinese (Cantonese, Simplified)` |`zh-HK` | If you want to train a synthesized voice capable of speaking English with the same accent as your sample data, make sure to provide over 10% English data in your training set. Otherwise, it defaults to an English native accent. The 10% threshold is calculated based on the data accepted after successful uploading, not the data before uploading. If some uploaded English data is rejected due to defects and doesn't meet the 10% threshold, the synthesized voice defaults to an English native accent. | 
-| `Chinese (Taiwanese Mandarin, Traditional)` | `zh-TW` | If you want to train a synthesized voice capable of speaking English with the same accent as your sample data, make sure to provide over 10% English data in your training set. Otherwise, it defaults to an English native accent. The 10% threshold is calculated based on the data accepted after successful uploading, not the data before uploading. If some uploaded English data is rejected due to defects and doesn't meet the 10% threshold, the synthesized voice defaults to an English native accent. | 
-
+| `Chinese (Cantonese, Simplified)` |`zh-HK` | If you want to train a synthesized voice capable of speaking English with the same accent as your sample data, make sure to provide over 10% English data in your training set. Otherwise, it defaults to an English native accent. The 10% threshold is calculated based on the data accepted after successful uploading, not the data before uploading. If some uploaded English data is rejected due to defects and doesn't meet the 10% threshold, the synthesized voice defaults to an English native accent. |
+| `Chinese (Taiwanese Mandarin, Traditional)` | `zh-TW` | If you want to train a synthesized voice capable of speaking English with the same accent as your sample data, make sure to provide over 10% English data in your training set. Otherwise, it defaults to an English native accent. The 10% threshold is calculated based on the data accepted after successful uploading, not the data before uploading. If some uploaded English data is rejected due to defects and doesn't meet the 10% threshold, the synthesized voice defaults to an English native accent. |