Added SpeechInputSource for integration of the KeywordRecognizer in the InputManager (#354)

* Implemented KeywordManager as a input source. Added all the required event pipeline.

* Added missing meta files

* Using the InputManager to handle the events.

* Changed the KeywordManager scene to test keyword handlers on focused object and also a global listener.

* Changed the KeywordManager scene to test keyword handlers on focused object and also a global listener.

* Fixed method inheritance in KeywordManager

* Made all the changes suggested til this date

* Fixes to the SpeechInputSource test scene

* Fixes to the SpeechInputSource test scene

* Fix not stopping when deactivated #360

* Rolled back enable/disable handling

* Set update() to protected virtual to allow inheritance. Should more methods also change?

* Set Unity methods to private

* Added to input documention

* Renamed all PhraseRecognized to SpeechKeywordRecognized to avoid name clashing.

* Removed LINQ from SpeechInputSource. Refactored SpeechInputSource.ProcessKeyBindings().

* Process key bindings only keyword recognizer is running.

* Duplicate behavior from #360 in SpeechInputSource

* Added a custom editor for SpeechInputSource showing a warning when no keywords are assigned.

* Improved SpeechInputSource keyword list layout in the inspector to show only two columns.

* Removed CanEditMultipleObjects from SpeechInputSourceEditor.

* Added add and remove buttons to the SpeechInputSource custom editor.

* Changed SpeechInputSource initialization to be more readable as in comment at https://github.com/Microsoft/HoloToolkit-Unity/pull/398/files#r92492988

* Added code comments to SpeechInputSourceEditor and KeywordAndKeyCodeDrawer.

* Use TextMesh instead of GUI Text in the SpeechInputSource test scene.

* Added a missing private keyword

* Unload materials in SphereKeywords.cs and SphereGlobalKeywords.cs. Partial fix for #236

* A fix for #236 using a shader. Changing the color requires changing the color of all vertices but rendering requires just one draw call.

* Use [PerRendererData] in the shader to efficiently change color without leaks.

* Unity doesn't like init blocks

* Added SetGlobalListener.cs

* Fixed misspellings

* Reverted changes to HoloToolkit_Default.mat

* Reverted addition of [PerRendererData] to BlinnPhongConfigurable and related code

* Use the SetGlobalListener.cs instead

* Latest comments fixes

Author: aalmada

Assets/HoloToolkit/Input/README.md

@@ -17,6 +17,7 @@ Game objects that want to consume input events can implement one or many **input
 - **IManipulationHandler** for the Windows manipulation gesture.
 - **INavigationnHandler** for the Windows navigation gesture.
 - **ISourceStateHandler** for the source detected and source lost events.
+- **ISpeechHandler** for voice commands.
 
 The **input manager** listens to the various events coming from the input sources, and also takes into account the gaze. Currently, that gaze is always coming from the GazeManager class, but this could be extended to support multiple gaze sources if the need arises.
 
@@ -189,16 +190,30 @@ or in your Visual Studio Package.appxmanifest capabilities.
 **RecognizerStart** Set this to determine whether the keyword recognizer will start immediately or if it should wait for your code to tell it to start.
 
 #### Voice
-##### KeywordManager.cs
-Allows you to specify keywords and methods in the Unity Inspector, instead of registering them explicitly in code.  
-**IMPORTANT**: Please make sure to add the microphone capability in your app, in Unity under  
+
+**IMPORTANT**: Please make sure to add the Microphone capabilities in your app, in Unity under  
 Edit -> Project Settings -> Player -> Settings for Windows Store -> Publishing Settings -> Capabilities  
 or in your Visual Studio Package.appxmanifest capabilities.
 
+##### KeywordManager.cs
+Allows you to specify keywords and methods in the Unity Inspector, instead of registering them explicitly in code.  
+
 **_KeywordsAndResponses_** Set the size as the number of keywords you'd like to listen for, then specify the keywords and method responses to complete the array.
 
 **RecognizerStart** Set this to determine whether the keyword recognizer will start immediately or if it should wait for your code to tell it to start.
 
+##### SpeechInputSource.cs
+Allows you to specify keywords and keyboard shortcuts in the Unity Inspector, instead of registering them explicitly in code. Keywords are handled by scripts that implement ISpeechHandler.cs.
+
+Check out Assets/HoloToolkit/Input/Tests/Scripts/SphereKeywords.cs and Assets/HoloToolkit/Input/Tests/Scripts/SphereGlobalKeywords.cs for an example of implementing these features, which is used in the demo scene at Assets/HoloToolkit/Input/Tests/SpeechInputSource.unity.
+
+**_KeywordsAndKeys_** Set the size as the number of keywords you'd like to listen for, then specify the keywords to complete the array.
+
+**RecognizerStart** Set this to determine whether the keyword recognizer will start immediately or if it should wait for your code to tell it to start.
+
+##### ISpeechHandler.cs
+Interface that a game object can implement to react to speech keywords.
+
 ### [Test Prefabs](TestPrefabs)
 
 Prefabs used in the various test scenes, which you can use as inspiration to build your own.
@@ -276,6 +291,19 @@ Gazing on an object and saying "Select Object" will persistently select that obj
 after which the user can also adjust object size with "Make Smaller" and "Make Bigger" voice commands and finally clear
 currently selected object by saying "Clear Selection".
 
+#### SpeechInputSource.unity
+
+Shows how to use the SpeechInputSource.cs script to add keywords to your scene.
+
+1. Select whether you want the recognizer to start automatically or when you manually start it.
+2. Specify the number of keywords you want.
+3. Type the word or phrase you'd like to register as the keyword and, if you want, set a key code to use in the Editor. You can also use an attached microphone with the Editor.
+4. Attach a script that implements ISpeechHandler.cs to the object in the scene that will require the gaze focus to execute the command. You should register this script with the InputManager.cs as a global listener to handle keywords that don't require a focused object.
+
+When you start the scene, your keywords will automatically be registered on a KeywordRecognizer, and the recognizer will be started (or not) based on your Recognizer Start setting.
+
+#### 
+
 ---
 ##### [Go back up to the table of contents.](../../../README.md)
 ---

Assets/HoloToolkit/Input/Scripts/InputEvents/ISpeechHandler.cs

@@ -0,0 +1,15 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.
+// Licensed under the MIT License. See LICENSE in the project root for license information.
+
+using UnityEngine.EventSystems;
+
+namespace HoloToolkit.Unity.InputModule
+{
+    /// <summary>
+    /// Interface to implement to react to speech recognition.
+    /// </summary>
+    public interface ISpeechHandler : IEventSystemHandler
+    {
+        void OnSpeechKeywordRecognized(SpeechKeywordRecognizedEventData eventData);
+    }
+}

Assets/HoloToolkit/Input/Scripts/InputEvents/ISpeechHandler.cs.meta

@@ -0,0 +1,55 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.
+// Licensed under the MIT License. See LICENSE in the project root for license information.
+
+using System;
+using UnityEngine;
+using UnityEngine.EventSystems;
+using UnityEngine.Windows.Speech;
+
+namespace HoloToolkit.Unity.InputModule
+{
+    /// <summary>
+    /// Describes an input event that involves keyword recognition.
+    /// </summary>
+    public class SpeechKeywordRecognizedEventData : InputEventData
+    {
+        /// <summary>
+        /// A measure of correct recognition certainty.
+        /// </summary>
+        public ConfidenceLevel Confidence { get; private set; }
+
+        /// <summary>
+        /// The time it took for the phrase to be uttered.
+        /// </summary>
+        public TimeSpan PhraseDuration { get; private set; }
+
+        /// <summary>
+        /// The moment in time when uttering of the phrase began.
+        /// </summary>
+        public DateTime PhraseStartTime { get; private set; }
+
+        /// <summary>
+        /// A semantic meaning of recognized phrase.
+        /// </summary>
+        public SemanticMeaning[] SemanticMeanings { get; private set; }
+
+        /// <summary>
+        /// The text that was recognized.
+        /// </summary>
+        public string RecognizedText { get; private set; }
+
+        public SpeechKeywordRecognizedEventData(EventSystem eventSystem) : base(eventSystem)
+        {
+        }
+
+        public void Initialize(IInputSource inputSource, uint sourceId, ConfidenceLevel confidence, TimeSpan phraseDuration, DateTime phraseStartTime, SemanticMeaning[] semanticMeanings, string recognizedText)
+        {
+            BaseInitialize(inputSource, sourceId);
+            Confidence = confidence;
+            PhraseDuration = phraseDuration;
+            PhraseStartTime = phraseStartTime;
+            SemanticMeanings = semanticMeanings;
+            RecognizedText = recognizedText;
+        }
+    }
+}

Assets/HoloToolkit/Input/Scripts/InputEvents/SpeechKeywordRecognizedEventData.cs

@@ -33,6 +33,7 @@ public class InputManager : Singleton<InputManager>
         private ManipulationEventData manipulationEventData;
         private NavigationEventData navigationEventData;
         private HoldEventData holdEventData;
+        private SpeechKeywordRecognizedEventData speechKeywordRecognizedEventData;
 
         /// <summary>
         /// Indicates if input is currently enabled or not.
@@ -185,6 +186,7 @@ public void RegisterInputSource(IInputSource inputSource)
             inputSource.NavigationCompleted += InputSource_NavigationCompleted;
             inputSource.NavigationStarted += InputSource_NavigationStarted;
             inputSource.NavigationUpdated += InputSource_NavigationUpdated;
+            inputSource.SpeechKeywordRecognized += InputSource_SpeechKeywordRecognized;
         }
 
         /// <summary>
@@ -210,6 +212,7 @@ public void UnregisterInputSource(IInputSource inputSource)
             inputSource.NavigationCompleted -= InputSource_NavigationCompleted;
             inputSource.NavigationStarted -= InputSource_NavigationStarted;
             inputSource.NavigationUpdated -= InputSource_NavigationUpdated;
+            inputSource.SpeechKeywordRecognized -= InputSource_SpeechKeywordRecognized;
         }
 
         private void Start()
@@ -231,6 +234,7 @@ private void InitializeEventDatas()
             manipulationEventData = new ManipulationEventData(EventSystem.current);
             navigationEventData = new NavigationEventData(EventSystem.current);
             holdEventData = new HoldEventData(EventSystem.current);
+            speechKeywordRecognizedEventData = new SpeechKeywordRecognizedEventData(EventSystem.current);
         }
 
         protected override void OnDestroy()
@@ -653,5 +657,21 @@ private void InputSource_NavigationCanceled(object sender, NavigationEventArgs e
             // Pass handler through HandleEvent to perform modal/fallback logic
             HandleEvent(navigationEventData, OnNavigationCanceledEventHandler);
         }
+
+        private static readonly ExecuteEvents.EventFunction<ISpeechHandler> OnSpeechKeywordRecognizedEventHandler =
+            delegate (ISpeechHandler handler, BaseEventData eventData)
+            {
+                SpeechKeywordRecognizedEventData casted = ExecuteEvents.ValidateEventData<SpeechKeywordRecognizedEventData>(eventData);
+                handler.OnSpeechKeywordRecognized(casted);
+            };
+
+        private void InputSource_SpeechKeywordRecognized(object sender, SpeechKeywordRecognizedEventArgs e)
+        {
+            // Create input event
+            speechKeywordRecognizedEventData.Initialize(e.InputSource, e.SourceId, e.Confidence, e.PhraseDuration, e.PhraseStartTime, e.SemanticMeanings, e.RecognizedText);
+
+            // Pass handler through HandleEvent to perform modal/fallback logic
+            HandleEvent(speechKeywordRecognizedEventData, OnSpeechKeywordRecognizedEventHandler);
+        }
     }
 }

Assets/HoloToolkit/Input/Scripts/InputEvents/SpeechKeywordRecognizedEventData.cs.meta

@@ -27,6 +27,7 @@ public abstract class BaseInputSource : MonoBehaviour, IInputSource
         public event EventHandler<NavigationEventArgs> NavigationUpdated;
         public event EventHandler<NavigationEventArgs> NavigationCompleted;
         public event EventHandler<NavigationEventArgs> NavigationCanceled;
+        public event EventHandler<SpeechKeywordRecognizedEventArgs> SpeechKeywordRecognized;
 
         public abstract SupportedInputEvents SupportedEvents { get; }
 
@@ -251,6 +252,15 @@ protected void RaiseNavigationCanceledEvent(NavigationEventArgs e)
             }
         }
 
+        protected void RaiseSpeechKeywordRecognizedEvent(SpeechKeywordRecognizedEventArgs e)
+        {
+            EventHandler<SpeechKeywordRecognizedEventArgs> handler = SpeechKeywordRecognized;
+            if (handler != null)
+            {
+                handler(this, e);
+            }
+        }
+
         #endregion
     }
 }

Assets/HoloToolkit/Input/Scripts/InputManager.cs

@@ -16,7 +16,8 @@ public enum SupportedInputEvents
         SourceClicked = 2,
         Hold = 4,
         Manipulation = 8,
-        Navigation = 16
+        Navigation = 16,
+        SpeechKeyword = 32
     }
 
     /// <summary>
@@ -118,6 +119,11 @@ public interface IInputSource
         /// </summary>
         event EventHandler<NavigationEventArgs> NavigationCanceled;
 
+        /// <summary>
+        /// Event triggered when a speech phrase is recognized.
+        /// </summary>
+        event EventHandler<SpeechKeywordRecognizedEventArgs> SpeechKeywordRecognized;
+
         /// <summary>
         /// Events supported by the input source.
         /// </summary>

Assets/HoloToolkit/Input/Scripts/InputSources/BaseInputSource.cs

@@ -0,0 +1,49 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.
+// Licensed under the MIT License. See LICENSE in the project root for license information.
+
+using System;
+using UnityEngine;
+using UnityEngine.Windows.Speech;
+
+namespace HoloToolkit.Unity.InputModule
+{
+    /// <summary>
+    /// Event args for a speech kwyword recognized event.
+    /// </summary>
+    public class SpeechKeywordRecognizedEventArgs : InputSourceEventArgs
+    {
+        /// <summary>
+        /// A measure of correct recognition certainty.
+        /// </summary>
+        public ConfidenceLevel Confidence { get; private set; }
+
+        /// <summary>
+        /// The time it took for the phrase to be uttered.
+        /// </summary>
+        public TimeSpan PhraseDuration { get; private set; }
+
+        /// <summary>
+        /// The moment in time when uttering of the phrase began.
+        /// </summary>
+        public DateTime PhraseStartTime { get; private set; }
+
+        /// <summary>
+        /// A semantic meaning of recognized phrase.
+        /// </summary>
+        public SemanticMeaning[] SemanticMeanings { get; private set; }
+
+        /// <summary>
+        /// The text that was recognized.
+        /// </summary>
+        public string RecognizedText { get; private set; }
+
+        public SpeechKeywordRecognizedEventArgs(IInputSource inputSource, uint sourceId, ConfidenceLevel confidence, TimeSpan phraseDuration, DateTime phraseStartTime, SemanticMeaning[] semanticMeanings, string recognizedText) : base(inputSource, sourceId)
+        {
+            Confidence = confidence;
+            PhraseDuration = phraseDuration;
+            PhraseStartTime = phraseStartTime;
+            SemanticMeanings = semanticMeanings;
+            RecognizedText = recognizedText;
+        }
+    }
+}