Merge pull request #205814 from sally-baolian/FacialMotion

Update blendshapes Cog Svcs （Release on 8/5)

Author: v-ccolin

articles/cognitive-services/Speech-Service/how-to-speech-synthesis-viseme.md

@@ -15,14 +15,14 @@ ms.custom: references_regions
 zone_pivot_groups: programming-languages-speech-services-nomore-variant
 ---
 
-# Get facial pose events for lip-sync
+# Get facial position with viseme
 
 > [!NOTE]
-> At this time, viseme events are available only for [neural voices](language-support.md#text-to-speech).
+> Viseme ID supports neural voices in [all viseme-supported locales](language-support.md#viseme). Scalable Vector Graphics (SVG) only supports neural voices in `en-US` locale, and blend shapes supports neural voices in `en-US` and `zh-CN` locales.
 
-A _viseme_ is the visual description of a phoneme in spoken language. It defines the position of the face and mouth when a person speaks a word. Each viseme depicts the key facial poses for a specific set of phonemes. 
+A *viseme* is the visual description of a phoneme in spoken language. It defines the position of the face and mouth while a person is speaking. Each viseme depicts the key facial poses for a specific set of phonemes.
 
-You can use visemes to control the movement of 2D and 3D avatar models, so that the mouth movements are perfectly matched to synthetic speech. For example, you can:
+You can use visemes to control the movement of 2D and 3D avatar models, so that the facial positions are best aligned with synthetic speech. For example, you can:
 
  * Create an animated virtual voice assistant for intelligent kiosks, building multi-mode integrated services for your customers.
  * Build immersive news broadcasts and improve audience experiences with natural face and mouth movements.
@@ -33,21 +33,39 @@ You can use visemes to control the movement of 2D and 3D avatar models, so that
 For more information about visemes, view this [introductory video](https://youtu.be/ui9XT47uwxs).
 > [!VIDEO https://www.youtube.com/embed/ui9XT47uwxs]
 
-## Azure Neural TTS can produce visemes with speech
+## Overall workflow of producing viseme with speech
 
-Neural Text-to-Speech (Neural TTS) turns input text or SSML (Speech Synthesis Markup Language) into lifelike synthesized speech. Speech audio output can be accompanied by viseme IDs and their offset timestamps. Each viseme ID specifies a specific pose in observed speech, such as the position of the lips, jaw, and tongue when producing a particular phoneme. Using a 2D or 3D rendering engine, you can use these viseme events to animate your avatar.
+Neural Text-to-Speech (Neural TTS) turns input text or SSML (Speech Synthesis Markup Language) into lifelike synthesized speech. Speech audio output can be accompanied by viseme ID, Scalable Vector Graphics (SVG), or blend shapes. Using a 2D or 3D rendering engine, you can use these viseme events to animate your avatar.
 
 The overall workflow of viseme is depicted in the following flowchart:
 
 ![Diagram of the overall workflow of viseme.](media/text-to-speech/viseme-structure.png)
 
-*Viseme ID* and *audio offset output* are described in the following table:
+You can request viseme output in SSML. For details, see [how to use viseme element in SSML](speech-synthesis-markup.md#viseme-element).
 
-| Visme element | Description |
-|-----------|-------------|
-| Viseme ID | An integer number that specifies a viseme.<br>For English (US), we offer 22 different visemes, each depicting the mouth shape for a specific set of phonemes. There is no one-to-one correspondence between visemes and phonemes. Often, several phonemes correspond to a single viseme, because they look the same on the speaker's face when they're produced, such as `s` and `z`. For more specific information, see the table for [mapping phonemes to viseme IDs](#map-phonemes-to-visemes).  |
-| Audio offset | The start time of each viseme, in ticks (100 nanoseconds). |
+## Viseme ID
 
+Viseme ID refers to an integer number that specifies a viseme. We offer 22 different visemes, each depicting the mouth shape for a specific set of phonemes. There's no one-to-one correspondence between visemes and phonemes. Often, several phonemes correspond to a single viseme, because they look the same on the speaker's face when they're produced, such as `s` and `z`. For more specific information, see the table for [mapping phonemes to viseme IDs](#map-phonemes-to-visemes).
+
+Speech audio output can be accompanied by viseme IDs and `Audio offset`. The `Audio offset` indicates the offset timestamp that represents the start time of each viseme, in ticks (100 nanoseconds).
+
+### Map phonemes to visemes
+
+Visemes vary by language and locale. Each locale has a set of visemes that correspond to its specific phonemes. The [SSML phonetic alphabets](speech-ssml-phonetic-sets.md) documentation maps viseme IDs to the corresponding International Phonetic Alphabet (IPA) phonemes.
+
+## 2D SVG animation
+
+For 2D characters, you can design a character that suits your scenario and use Scalable Vector Graphics (SVG) for each viseme ID to get a time-based face position.
+
+With temporal tags that are provided in a viseme event, these well-designed SVGs will be processed with smoothing modifications, and provide robust animation to the users. For example, the following illustration shows a red-lipped character that's designed for language learning.
+
+![Screenshot showing a 2D rendering example of four red-lipped mouths, each representing a different viseme ID that corresponds to a phoneme.](media/text-to-speech/viseme-demo-2D.png)
+
+## 3D blend shapes animation
+
+You can use blend shapes to drive the facial movements of a 3D character that you designed.
+
+The blend shapes JSON string is represented as a 2-dimensional matrix. Each row represents a frame. Each frame (in 60 FPS) contains an array of 55 facial positions.
 
 ## Get viseme events with the Speech SDK
 
@@ -65,9 +83,13 @@ using (var synthesizer = new SpeechSynthesizer(speechConfig, audioConfig))
     {
         Console.WriteLine($"Viseme event received. Audio offset: " +
             $"{e.AudioOffset / 10000}ms, viseme id: {e.VisemeId}.");
+
+        // `Animation` is an xml string for SVG or a json string for blend shapes
+        var animation = e.Animation;
     };
 
-    var result = await synthesizer.SpeakSsmlAsync(ssml));
+    // If VisemeID is the only thing you want, you can also use `SpeakTextAsync()`
+    var result = await synthesizer.SpeakSsmlAsync(ssml);
 }
 
 ```
@@ -86,8 +108,12 @@ synthesizer->VisemeReceived += [](const SpeechSynthesisVisemeEventArgs& e)
         // The unit of e.AudioOffset is tick (1 tick = 100 nanoseconds), divide by 10,000 to convert to milliseconds.
         << "Audio offset: " << e.AudioOffset / 10000 << "ms, "
         << "viseme id: " << e.VisemeId << "." << endl;
+
+    // `Animation` is an xml string for SVG or a json string for blend shapes
+    auto animation = e.Animation;
 };
 
+// If VisemeID is the only thing you want, you can also use `SpeakTextAsync()`
 auto result = synthesizer->SpeakSsmlAsync(ssml).get();
 ```
 
@@ -103,8 +129,12 @@ synthesizer.VisemeReceived.addEventListener((o, e) -> {
     // The unit of e.AudioOffset is tick (1 tick = 100 nanoseconds), divide by 10,000 to convert to milliseconds.
     System.out.print("Viseme event received. Audio offset: " + e.getAudioOffset() / 10000 + "ms, ");
     System.out.println("viseme id: " + e.getVisemeId() + ".");
+
+    // `Animation` is an xml string for SVG or a json string for blend shapes
+    String animation = e.getAnimation();
 });
 
+// If VisemeID is the only thing you want, you can also use `SpeakTextAsync()`
 SpeechSynthesisResult result = synthesizer.SpeakSsmlAsync(ssml).get();
 ```
 
@@ -115,10 +145,17 @@ SpeechSynthesisResult result = synthesizer.SpeakSsmlAsync(ssml).get();
 ```Python
 speech_synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config, audio_config=audio_config)
 
+def viseme_cb(evt):
+    print("Viseme event received: audio offset: {}ms, viseme id: {}.".format(
+        evt.audio_offset / 10000, evt.viseme_id))
+
+    # `Animation` is an xml string for SVG or a json string for blend shapes
+    animation = evt.animation
+
 # Subscribes to viseme received event
-speech_synthesizer.viseme_received.connect(lambda evt: print(
-    "Viseme event received: audio offset: {}ms, viseme id: {}.".format(evt.audio_offset / 10000, evt.viseme_id)))
+speech_synthesizer.viseme_received.connect(viseme_cb)
 
+# If VisemeID is the only thing you want, you can also use `speak_text_async()`
 result = speech_synthesizer.speak_ssml_async(ssml).get()
 ```
 
@@ -132,8 +169,12 @@ var synthesizer = new SpeechSDK.SpeechSynthesizer(speechConfig, audioConfig);
 // Subscribes to viseme received event
 synthesizer.visemeReceived = function (s, e) {
     window.console.log("(Viseme), Audio offset: " + e.audioOffset / 10000 + "ms. Viseme ID: " + e.visemeId);
+
+    // `Animation` is an xml string for SVG or a json string for blend shapes
+    var animation = e.Animation;
 }
 
+// If VisemeID is the only thing you want, you can also use `speakTextAsync()`
 synthesizer.speakSsmlAsync(ssml);
 ```
 
@@ -149,14 +190,20 @@ SPXSpeechSynthesizer *synthesizer =
 // Subscribes to viseme received event
 [synthesizer addVisemeReceivedEventHandler: ^ (SPXSpeechSynthesizer *synthesizer, SPXSpeechSynthesisVisemeEventArgs *eventArgs) {
     NSLog(@"Viseme event received. Audio offset: %fms, viseme id: %lu.", eventArgs.audioOffset/10000., eventArgs.visemeId);
+
+    // `Animation` is an xml string for SVG or a json string for blend shapes
+    NSString *animation = eventArgs.Animation;
 }];
 
+// If VisemeID is the only thing you want, you can also use `SpeakText`
 [synthesizer speakSsml:ssml];
 ```
 
 ::: zone-end
 
-Here is an example of the viseme output.
+Here's an example of the viseme output.
+
+# [Viseme ID](#tab/visemeid)
 
 ```text
 (Viseme), Viseme ID: 1, Audio offset: 200ms.
@@ -168,20 +215,101 @@ Here is an example of the viseme output.
 (Viseme), Viseme ID: 13, Audio offset: 2350ms.
 ```
 
-After you obtain the viseme output, you can use these events to drive character animation. You can build your own characters and automatically animate them.
+# [2D SVG](#tab/2dsvg)
 
-For 2D characters, you can design a character that suits your scenario and use Scalable Vector Graphics (SVG) for each viseme ID to get a time-based face position. With temporal tags that are provided in a viseme event, these well-designed SVGs will be processed with smoothing modifications, and provide robust animation to the users. For example, the following illustration shows a red-lipped character that's designed for language learning.
+The SVG output is a xml string that contains the animation.
+Render the SVG animation along with the synthesized speech to see the mouth movement.
 
-![Screenshot showing a 2D rendering example of four red-lipped mouths, each representing a different viseme ID that corresponds to a phoneme.](media/text-to-speech/viseme-demo-2D.png)
+```xml
+<svg width= "1200px" height= "1200px" ..>
+  <g id= "front_start" stroke= "none" stroke-width= "1" fill= "none" fill-rule= "evenodd">
+    <animate attributeName= "d" begin= "d_dh_front_background_1_0.end" dur= "0.27500
+    ...
+```
 
-For 3D characters, think of the characters as string puppets. The puppet master pulls the strings from one state to another and the laws of physics do the rest and drive the puppet to move fluidly. The viseme output acts as a puppet master to provide an action timeline. The animation engine defines the physical laws of action. By interpolating frames with easing algorithms, the engine can further generate high-quality animations.
+# [3D blend shapes](#tab/3dblendshapes)
 
-## Map phonemes to visemes
 
-Visemes vary by language and locale. Each locale has a set of visemes that correspond to its specific phonemes. The [SSML phonetic alphabets](speech-ssml-phonetic-sets.md) documentation maps viseme IDs to the corresponding International Phonetic Alphabet (IPA) phonemes.
+Each viseme event includes a series of frames in the `Animation` SDK property. These are grouped to best align the facial positions with the audio. Your 3D engine should render each group of `BlendShapes` frames immediately before the corresponding audio chunk. The `FrameIndex` value indicates how many frames preceded the current list of frames.
+
+The output json looks like the following sample. Each frame within `BlendShapes` contains an array of 55 facial positions represented as decimal values between 0 to 1. The decimal values are in the same order as described in the facial positions table below.
 
+```json
+{
+    "FrameIndex":0,
+    "BlendShapes":[
+        [0.021,0.321,...,0.258],
+        [0.045,0.234,...,0.288],
+        ...
+    ]
+}
+```
+
+The order of `BlendShapes` is as follows.
+
+| Order | Facial position in `BlendShapes`|
+| --------- | ----------- | 
+| 1 | eyeBlinkLeft|
+| 2 | eyeLookDownLeft|
+| 3 | eyeLookInLeft|
+| 4 | eyeLookOutLeft|
+| 5 | eyeLookUpLeft|
+| 6 | eyeSquintLeft|
+| 7 | eyeWideLeft|
+| 8 | eyeBlinkRight|
+| 9 | eyeLookDownRight|
+| 10 | eyeLookInRight|
+| 11 | eyeLookOutRight|
+| 12 | eyeLookUpRight|
+| 13 | eyeSquintRight|
+| 14 | eyeWideRight|
+| 15 | jawForward|
+| 16 | jawLeft|
+| 17 | jawRight|
+| 18 | jawOpen|
+| 19 | mouthClose|
+| 20 | mouthFunnel|
+| 21 | mouthPucker|
+| 22 | mouthLeft|
+| 23 | mouthRight|
+| 24 | mouthSmileLeft|
+| 25 | mouthSmileRight|
+| 26 | mouthFrownLeft|
+| 27 | mouthFrownRight|
+| 28 | mouthDimpleLeft|
+| 29 | mouthDimpleRight|
+| 30 | mouthStretchLeft|
+| 31 | mouthStretchRight|
+| 32 | mouthRollLower|
+| 33 | mouthRollUpper|
+| 34 | mouthShrugLower|
+| 35 | mouthShrugUpper|
+| 36 | mouthPressLeft|
+| 37 | mouthPressRight|
+| 38 | mouthLowerDownLeft|
+| 39 | mouthLowerDownRight|
+| 40 | mouthUpperUpLeft|
+| 41 | mouthUpperUpRight|
+| 42 | browDownLeft|
+| 43 | browDownRight|
+| 44 | browInnerUp|
+| 45 | browOuterUpLeft|
+| 46 | browOuterUpRight|
+| 47 | cheekPuff|
+| 48 | cheekSquintLeft|
+| 49 | cheekSquintRight|
+| 50 | noseSneerLeft|
+| 51 | noseSneerRight|
+| 52 | tongueOut|
+| 53 | headRoll|
+| 54 | leftEyeRoll|
+| 55 | rightEyeRoll|
+
+---
+
+After you obtain the viseme output, you can use these events to drive character animation. You can build your own characters and automatically animate them.
 
 ## Next steps
 
-> [!div class="nextstepaction"]
-> [SSML phonetic alphabets](speech-ssml-phonetic-sets.md)
+- [SSML phonetic alphabets](speech-ssml-phonetic-sets.md)
+- [How to improve synthesis with SSML](speech-synthesis-markup.md)

articles/cognitive-services/Speech-Service/language-support.md

@@ -961,6 +961,120 @@ There are two Custom Neural Voice (CNV) project types: CNV Pro and CNV Lite (pre
 | Turkish (Turkey) | `tr-TR` | No |No|
 | Vietnamese (Vietnam) | `vi-VN` | No |No|
 
+### Viseme
+
+A _viseme_ is the visual description of a phoneme in spoken language. It defines the position of the face and mouth while a person is speaking. Each viseme depicts the key facial poses for a specific set of phonemes. Speech audio output can be accompanied by a viseme ID, Scalable Vector Graphics (SVG), or blend shapes. For more information, see [Get facial position with viseme](how-to-speech-synthesis-viseme.md).
+
+> [!NOTE]
+> Viseme ID supports [neural voices](#text-to-speech) in the locales listed below. SVG only supports neural voices in the `en-US` locale, and blend shapes supports neural voices in the `en-US` and `zh-CN` locales. 
+
+The following table lists the languages supported by viseme ID.
+
+| Language | Locale |
+|---|---|
+| Arabic (Algeria) | `ar-DZ` |
+| Arabic (Bahrain) | `ar-BH` |
+| Arabic (Egypt) | `ar-EG` |
+| Arabic (Iraq) | `ar-IQ` |
+| Arabic (Jordan) | `ar-JO` |
+| Arabic (Kuwait) | `ar-KW` |
+| Arabic (Lebanon) | `ar-LB` |
+| Arabic (Libya) | `ar-LY` |
+| Arabic (Morocco) | `ar-MA` |
+| Arabic (Oman) | `ar-OM` |
+| Arabic (Qatar) | `ar-QA` |
+| Arabic (Saudi Arabia) | `ar-SA` |
+| Arabic (Syria) | `ar-SY` |
+| Arabic (Tunisia) | `ar-TN` |
+| Arabic (United Arab Emirates) | `ar-AE` |
+| Arabic (Yemen) | `ar-YE` |
+| Bulgarian (Bulgaria) | `bg-BG` |
+| Catalan (Spain) | `ca-ES` |
+| Chinese (Cantonese, Traditional) | `zh-HK` |
+| Chinese (Mandarin, Simplified) | `zh-CN` |
+| Chinese (Taiwanese Mandarin) | `zh-TW` |
+| Croatian (Croatia) | `hr-HR` |
+| Czech (Czech) | `cs-CZ` |
+| Danish (Denmark) | `da-DK` |
+| Dutch (Belgium) | `nl-BE` |
+| Dutch (Netherlands) | `nl-NL` |
+| English (Australia) | `en-AU` |
+| English (Canada) | `en-CA` |
+| English (Hongkong) | `en-HK` |
+| English (India) | `en-IN` |
+| English (Ireland) | `en-IE` |
+| English (Kenya) | `en-KE` |
+| English (New Zealand) | `en-NZ` |
+| English (Nigeria) | `en-NG` |
+| English (Philippines) | `en-PH` |
+| English (Singapore) | `en-SG` |
+| English (South Africa) | `en-ZA` |
+| English (Tanzania) | `en-TZ` |
+| English (United Kingdom) | `en-GB` |
+| English (United States) | `en-US` |
+| Finnish (Finland) | `fi-FI` |
+| French (Belgium) | `fr-BE` |
+| French (Canada) | `fr-CA` |
+| French (France) | `fr-FR` |
+| French (Switzerland) | `fr-CH` |
+| German (Austria) | `de-AT` |
+| German (Germany) | `de-DE` |
+| German (Switzerland) | `de-CH` |
+| Greek (Greece) | `el-GR` |
+| Gujarati (India) | `gu-IN` |
+| Hebrew (Israel) | `he-IL` |
+| Hindi (India) | `hi-IN` |
+| Hungarian (Hungary) | `hu-HU` |
+| Indonesian (Indonesia) | `id-ID` |
+| Italian (Italy) | `it-IT` |
+| Japanese (Japan) | `ja-JP` |
+| Korean (Korea) | `ko-KR` |
+| Malay (Malaysia) | `ms-MY` |
+| Marathi (India) | `mr-IN` |
+| Norwegian (Bokmål, Norway) | `nb-NO` |
+| Polish (Poland) | `pl-PL` |
+| Portuguese (Brazil) | `pt-BR` |
+| Portuguese (Portugal) | `pt-PT` |
+| Romanian (Romania) | `ro-RO` |
+| Russian (Russia) | `ru-RU` |
+| Slovak (Slovakia) | `sk-SK` |
+| Slovenian (Slovenia) | `sl-SI` |
+| Spanish (Argentina) | `es-AR` |
+| Spanish (Bolivia) | `es-BO` |
+| Spanish (Chile) | `es-CL` |
+| Spanish (Colombia) | `es-CO` |
+| Spanish (Costa Rica) | `es-CR` | 
+| Spanish (Cuba) | `es-CU` |
+| Spanish (Dominican Republic) | `es-DO` |
+| Spanish (Ecuador) | `es-EC` |
+| Spanish (El Salvador) | `es-SV` |
+| Spanish (Equatorial Guinea) | `es-GQ` |
+| Spanish (Guatemala) | `es-GT` |
+| Spanish (Honduras) | `es-HN` |
+| Spanish (Mexico) | `es-MX` |
+| Spanish (Nicaragua) | `es-NI` |
+| Spanish (Panama) | `es-PA` |
+| Spanish (Paraguay) | `es-PY` |
+| Spanish (Peru) | `es-PE` |
+| Spanish (Puerto Rico) | `es-PR` |
+| Spanish (Spain) | `es-ES` |
+| Spanish (Uruguay) | `es-UY` |
+| Spanish (US) | `es-US` |
+| Spanish (Venezuela) | `es-VE` |
+| Swahili (Tanzania) | `sw-TZ` |
+| Swedish (Sweden) | `sv-SE` |
+| Tamil (India) | `ta-IN` |
+| Tamil (Malaysia) | `ta-MY` |
+| Tamil (Singapore) | `ta-SG` |
+| Tamil (Sri Lanka) | `ta-LK` |
+| Telugu (India) | `te-IN` |
+| Thai (Thailand) | `th-TH` |
+| Turkish (Turkey) | `tr-TR` |
+| Ukrainian (Ukraine) | `uk-UA` |
+| Urdu (India) | `ur-IN` |
+| Urdu (Pakistan) | `ur-PK` | 
+| Vietnamese (Vietnam) | `vi-VN` |
+
 ## Language identification
 
 With language identification, you set and get one of the supported locales in the following table. We only compare at the language level, such as English and German. If you include multiple locales of the same language, for example, `en-IN` and `en-US`, we'll only compare English (`en`) with the other candidate languages.