Skip to content

Commit ef13f90

Browse files
author
Lukas Tönne
authored
Input simulation docs (#3743)
* Documentation for the Input Simulation Service. * Embedded videos for input simulation docs. * Make a single video for camera control instead of multiple small ones. * Video to demo hand movement. * Sections for one- and two-hand manipulation. * Section on GGV interaction. * Nicer title cards for videos. * Remove unused files. * Move images and videos into External/Documentation/. * Update profile section to adjust for the new service registration. * Remove videos and replace them with embedded youtube links. * Disable autoplay for embedded documentation videos. * Nicer scaling and centering for images and videos. * Use "Hololens"/"Hololens 2" instead of "Hololens version 1/2". * Remove separators and fix typo.
1 parent 287870c commit ef13f90

13 files changed

+270
-110
lines changed

‎Documentation/InputSimulation/InputSimulationService.md‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
# Input Simulation Service
2+
3+
The Input Simulation Service emulates the behaviour of devices and platforms that may not be available in the Unity editor. Examples include:
4+
* Hololens or VR device head tracking
5+
* Hololens hand gestures
6+
* Hololens 2 articulated hand tracking
7+
8+
Users can use a conventional keyboard and mouse combination to control simulated devices at runtime. This allows testing of interactions in the Unity editor without first deploying to a device.
9+
10+
## Enabling the Input Simulation Service
11+
12+
Input simulation is enabled by default in MRTK.
13+
14+
Input simulation is an optional [Mixed Reality service](../../External/Documentation/MixedRealityServices.md). It can be added as a data provider in the [Input System profile](../TODO.md).
15+
* __Type__ must be _Microsoft.MixedReality.Toolkit.Input > InputSimulationService_.
16+
* __Platform(s)__ should always be _Windows Editor_ since the service depends on keyboard and mouse input.
17+
* __Profile__ has all settings for input simulation.
18+
19+
| __Warning__: Any type of profile can be assigned to services at the time of this writing. If you assign a different profile to the service, make sure to use a profile of type _Input Simulation_ or it will not work! |
20+
| --- |
21+
22+
<a target="_blank" href="../../External/Documentation/Images/MRTK_InputSimulation_InputSystemDataProviders.png">
23+
<img src="../../External/Documentation/Images/MRTK_InputSimulation_InputSystemDataProviders.png" title="Full Hand Mesh" width="80%" class="center" />
24+
</a>
25+
26+
Open the linked profile to access settings for input simulation.
27+
28+
<a target="_blank" href="../../External/Documentation/Images/MRTK_InputSimulation_InputSimulationProfile.png">
29+
<img src="../../External/Documentation/Images/MRTK_InputSimulation_InputSimulationProfile.png" title="Full Hand Mesh" width="80%" class="center" />
30+
</a>
31+
32+
# Camera Control
33+
34+
Head movement can be emulated by the Input Simulation Service.
35+
36+
<a target="_blank" href="../../External/Documentation/Images/MRTK_InputSimulation_CameraControlSettings.png">
37+
<img src="../../External/Documentation/Images/MRTK_InputSimulation_CameraControlSettings.png" title="Full Hand Mesh" width="80%" class="center" />
38+
</a>
39+
40+
## Rotating the camera
41+
42+
1. Hover over the viewport editor window.
43+
44+
_You may need to click the window to give it input focus if button presses don't work._
45+
46+
2. Press and hold the __Mouse Look Button__ (default: Right mouse button).
47+
3. Move the mouse in the viewport window to rotate the camera.
48+
49+
## Moving the camera
50+
51+
Press and hold the movement keys (W/A/S/D for forward/left/back/right).
52+
53+
<iframe width="560" height="315" src="https://www.youtube.com/embed/Z7L4I1ET7GU" class="center" frameborder="0" allow="accelerometer; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />
54+
55+
# Hand Simulation
56+
57+
The input simulation supports emulated hand devices. These virtual hands can interact with any object that supports regular hand devices, such as buttons or grabable objects.
58+
59+
<a target="_blank" href="../../External/Documentation/Images/MRTK_InputSimulation_HandSimulationMode.png">
60+
<img src="../../External/Documentation/Images/MRTK_InputSimulation_HandSimulationMode.png" title="Full Hand Mesh" width="80%" class="center" />
61+
</a>
62+
63+
The __Hand Simulation Mode__ switches between two distinct input models.
64+
65+
* _Articulated Hands_: Simulates a fully articulated hand device with joint position data.
66+
67+
Emulates Hololens 2 interaction model.
68+
69+
Interactions that are based on precise positioning of the hand or use touching can be simulated in this mode.
70+
71+
* _Gestures_: Simulates a simplified hand model with air tap and basic gestures.
72+
73+
Emulates [Hololens interaction model](https://docs.microsoft.com/en-us/windows/mixed-reality/gestures).
74+
75+
Focus is controlled using the Gaze pointer. The _Air Tap_ gesture is used to interact with buttons.
76+
77+
## Controlling hand movement
78+
79+
<a target="_blank" href="../../External/Documentation/Images/MRTK_InputSimulation_HandControlSettings.png">
80+
<img src="../../External/Documentation/Images/MRTK_InputSimulation_HandControlSettings.png" title="Full Hand Mesh" width="80%" class="center" />
81+
</a>
82+
83+
Press and hold the _Left/Right Hand Manipulation Key_ (default: Left Shift/Space for left/right respectively) to gain control of either hand. While the manipulation key is pressed, the hand will appear in the viewport. Mouse movement will move the hand in the view plane.
84+
85+
Once the manipulation key is released the hands will disappear after a short _Hand Hide Timeout_. To toggle hands on permanently, press the _Toggle Left/Right Hand Key_ (default: T/Y for left/right respectively). Press the toggle key again to hide the hands again.
86+
87+
<a target="_blank" href="../../External/Documentation/Images/MRTK_InputSimulation_HandPlacementSettings.png">
88+
<img src="../../External/Documentation/Images/MRTK_InputSimulation_HandPlacementSettings.png" title="Full Hand Mesh" width="80%" class="center" />
89+
</a>
90+
91+
Hands can be moved further or closer to the camera using the _mouse wheel_.
92+
93+
<a target="_blank" href="../../External/Documentation/Images/MRTK_InputSimulation_HandRotationSettings.png">
94+
<img src="../../External/Documentation/Images/MRTK_InputSimulation_HandRotationSettings.png" title="Full Hand Mesh" width="80%" class="center" />
95+
</a>
96+
97+
Hands can be rotated when precise direction is required.
98+
* Yaw rotates around the Y axis (default: E/Q keys for clockwise/counter-clockwise rotation)
99+
* Pitch rotates around the X axis (default: F/R keys for clockwise/counter-clockwise rotation)
100+
* Roll rotates around the Z axis (default: X/Z keys for clockwise/counter-clockwise rotation)
101+
102+
<iframe width="560" height="315" src="https://www.youtube.com/embed/uRYfwuqsjBQ" class="center" frameborder="0" allow="accelerometer; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />
103+
104+
## Hand Gestures
105+
106+
Hand gestures such as pinching, grabbing, poking, etc. can also be simulated.
107+
108+
<a target="_blank" href="../../External/Documentation/Images/MRTK_InputSimulation_HandGestureSettings.png">
109+
<img src="../../External/Documentation/Images/MRTK_InputSimulation_HandGestureSettings.png" title="Full Hand Mesh" width="80%" class="center" />
110+
</a>
111+
112+
1. First enable hand control using the manipulation keys (Left Shift/Space)
113+
114+
Alternatively toggle the hands on/off using the toggle keys (T/Y).
115+
116+
2. While manipulating, press and hold a mouse button to perform a hand gesture.
117+
118+
Each of the mouse buttons can be mapped to transform the hand shape into a different gesture using the _Left/Middle/Right Mouse Hand Gesture_ settings. The _Default Hand Gesture_ is the shape of the hand when no button is pressed.
119+
120+
| Note: The _Pinch_ gesture is the only gesture that performs the "Select" action at this point. |
121+
| --- |
122+
123+
## One-Hand Manipulation
124+
125+
1. Press and hold hand control key (Space/Left Shift)
126+
2. Point at object
127+
3. Hold mouse button to pinch
128+
4. Use mouse to move the object
129+
5. Release mouse button to stop interaction
130+
131+
<iframe width="560" height="315" src="https://www.youtube.com/embed/rM0xaHam6wM" class="center" frameborder="0" allow="accelerometer; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />
132+
133+
## Two-Hand Manipulation
134+
135+
For manipulating objects with two hands at the same time the persistent hand mode is recommended.
136+
137+
1. Toggle on both hands by pressing the toggle keys (T/Y).
138+
2. Manipulate one hand at a time:
139+
1. Hold _Space_ to control the right hand
140+
2. Move the hand to where you want to grab the object
141+
3. Press mouse button to activate the _Pinch_ gesture. In persistent mode the gesture will remain active when you release the mouse button.
142+
3. Repeat the process with the other hand, grabbing the same object in a second spot.
143+
4. Now that both hands are grabbing the same object, you can move either of them to perform two-handed manipulation.
144+
145+
<iframe width="560" height="315" src="https://www.youtube.com/embed/Qol5OFNfN14" class="center" frameborder="0" allow="accelerometer; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />
146+
147+
## GGV Interaction
148+
149+
1. Enable GGV simulation by switching __Hand Simulation Mode__ to _Gestures_ in the [Input Simulation Profile](#enabling-the-input-simulation-service)
150+
151+
<a target="_blank" href="../../External/Documentation/Images/MRTK_InputSimulation_SwitchToGGV.png">
152+
<img src="../../External/Documentation/Images/MRTK_InputSimulation_SwitchToGGV.png" title="Full Hand Mesh" width="80%" class="center" />
153+
</a>
154+
155+
2. Rotate the camera to point the gaze cursor at the interactable object (right mouse button)
156+
3. Hold _Space_ to control the right hand
157+
4. Click and hold _left mouse button_ to interact
158+
5. Rotate the camera again to manipulate the object
159+
160+
<iframe width="560" height="315" src="https://www.youtube.com/embed/6841rRMdqWw" class="center" frameborder="0" allow="accelerometer; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

‎Documentation/toc.yml‎

Lines changed: 109 additions & 109 deletions
Original file line numberDiff line numberDiff line change
@@ -1,109 +1,109 @@
1-
- name: About MRTK
2-
items:
3-
- name: MRTK v2
4-
href: ../MRTK-vNext.md
5-
- name: Mixed Reality Services
6-
href: ../External/Documentation/MixedRealityServices.md
7-
- name: How-tos
8-
href: ../External/HowTo/README.md
9-
items:
10-
- name: Getting Started
11-
href: GettingStartedWithTheMRTK.md
12-
items:
13-
- name: Downloading MRTK
14-
href: DownloadingTheMRTK.md
15-
- name: Core Features
16-
href: TODO.md
17-
items:
18-
- name: Packages
19-
href: TODO.md
20-
- name: Service Provider
21-
href: MixedRealityConfigurationGuide.md
22-
- name: Input
23-
href: TODO.md
24-
items:
25-
- name: Hands
26-
href: InputSystem/HandTracking.md
27-
- name: Voice
28-
href: TODO.md
29-
- name: Eye
30-
href: TODO.md
31-
- name: Gaze
32-
href: TODO.md
33-
- name: Controllers
34-
href: TODO.md
35-
- name: Interactions and UX Building Blocks
36-
href: TODO.md
37-
items:
38-
- name: Interactable
39-
href: README_Interactable.md
40-
- name: Obeject Manipulation
41-
href: README_ManipulationHandler.md
42-
- name: Fintertip Visualization
43-
href: README_FingertipVisualization.md
44-
- name: Hand Interaction Example
45-
href: README_HandInteractionExamples.md
46-
- name: App Bar
47-
href: README_AppBar.md
48-
- name: Bounding Box
49-
href: README_BoundingBox.md
50-
- name: Button
51-
href: README_Button.md
52-
- name: Object Collection
53-
href: README_ObjectCollection.md
54-
- name: Pointers
55-
href: README_Pointers.md
56-
- name: Slate
57-
href: README_Slate.md
58-
- name: System Keyboard
59-
href: README_SystemKeyboard.md
60-
- name: Tooltips
61-
href: README_Tooltip.md
62-
- name: Solvers
63-
href: README_Solver.md
64-
- name: Spatial Awareness
65-
href: SpatialAwareness/SpatialAwarenessGettingStarted.md
66-
items:
67-
- name: Configuring the Spatial Awareness Mesh Observer
68-
- name: In-Editor Input Simulation
69-
href: TODO.md
70-
- name: Diagnostics System
71-
href: Diagnostics/DiagnosticsSystemGettingStarted.md
72-
items:
73-
- name: Configuring the Diagnostics System
74-
href: Diagnostics/ConfiguringDiagnostics.md
75-
- name: Using the Visual Profiler
76-
href: TODO.md
77-
- name: Other Features
78-
href: TODO.md
79-
items:
80-
- name: QR Tracking
81-
href: TODO.md
82-
- name: Spatial Audio
83-
href: TODO.md
84-
- name: Advanced topics
85-
href: TODO.md
86-
items:
87-
- name: Performance
88-
href: TODO.md
89-
- name: Shared Experiences
90-
href: TODO.md
91-
- name: Moving your project from HTK to MRTK
92-
href: TODO.md
93-
- name: Contributing
94-
href: ../CONTRIBUTING.md
95-
items:
96-
- name: Feature Contribution Process
97-
href: ../Feature_Contribution_Process.md
98-
- name: Coding Guidelines
99-
href: ../CodingGuidelines.md
100-
- name: Documentation Guide
101-
href: ../DocumentationGuide.md
102-
- name: Developer Docs Generation
103-
href: DevDocGuide.md
104-
- name: Planning
105-
items:
106-
- name: Roadmap
107-
href: ../Roadmap.md
108-
- name: Notice
109-
href: ../NOTICE.md
1+
- name: About MRTK
2+
items:
3+
- name: MRTK v2
4+
href: ../MRTK-vNext.md
5+
- name: Mixed Reality Services
6+
href: ../External/Documentation/MixedRealityServices.md
7+
- name: How-tos
8+
href: ../External/HowTo/README.md
9+
items:
10+
- name: Getting Started
11+
href: GettingStartedWithTheMRTK.md
12+
items:
13+
- name: Downloading MRTK
14+
href: DownloadingTheMRTK.md
15+
- name: Core Features
16+
href: TODO.md
17+
items:
18+
- name: Packages
19+
href: TODO.md
20+
- name: Service Provider
21+
href: MixedRealityConfigurationGuide.md
22+
- name: Input
23+
href: TODO.md
24+
items:
25+
- name: Hands
26+
href: InputSystem/HandTracking.md
27+
- name: Voice
28+
href: TODO.md
29+
- name: Eye
30+
href: TODO.md
31+
- name: Gaze
32+
href: TODO.md
33+
- name: Controllers
34+
href: TODO.md
35+
- name: Interactions and UX Building Blocks
36+
href: TODO.md
37+
items:
38+
- name: Interactable
39+
href: README_Interactable.md
40+
- name: Obeject Manipulation
41+
href: README_ManipulationHandler.md
42+
- name: Fintertip Visualization
43+
href: README_FingertipVisualization.md
44+
- name: Hand Interaction Example
45+
href: README_HandInteractionExamples.md
46+
- name: App Bar
47+
href: README_AppBar.md
48+
- name: Bounding Box
49+
href: README_BoundingBox.md
50+
- name: Button
51+
href: README_Button.md
52+
- name: Object Collection
53+
href: README_ObjectCollection.md
54+
- name: Pointers
55+
href: README_Pointers.md
56+
- name: Slate
57+
href: README_Slate.md
58+
- name: System Keyboard
59+
href: README_SystemKeyboard.md
60+
- name: Tooltips
61+
href: README_Tooltip.md
62+
- name: Solvers
63+
href: README_Solver.md
64+
- name: Spatial Awareness
65+
href: SpatialAwareness/SpatialAwarenessGettingStarted.md
66+
items:
67+
- name: Configuring the Spatial Awareness Mesh Observer
68+
- name: In-Editor Input Simulation
69+
href: InputSimulation/InputSimulationService.md
70+
- name: Diagnostics System
71+
href: Diagnostics/DiagnosticsSystemGettingStarted.md
72+
items:
73+
- name: Configuring the Diagnostics System
74+
href: Diagnostics/ConfiguringDiagnostics.md
75+
- name: Using the Visual Profiler
76+
href: TODO.md
77+
- name: Other Features
78+
href: TODO.md
79+
items:
80+
- name: QR Tracking
81+
href: TODO.md
82+
- name: Spatial Audio
83+
href: TODO.md
84+
- name: Advanced topics
85+
href: TODO.md
86+
items:
87+
- name: Performance
88+
href: TODO.md
89+
- name: Shared Experiences
90+
href: TODO.md
91+
- name: Moving your project from HTK to MRTK
92+
href: TODO.md
93+
- name: Contributing
94+
href: ../CONTRIBUTING.md
95+
items:
96+
- name: Feature Contribution Process
97+
href: ../Feature_Contribution_Process.md
98+
- name: Coding Guidelines
99+
href: ../CodingGuidelines.md
100+
- name: Documentation Guide
101+
href: ../DocumentationGuide.md
102+
- name: Developer Docs Generation
103+
href: DevDocGuide.md
104+
- name: Planning
105+
items:
106+
- name: Roadmap
107+
href: ../Roadmap.md
108+
- name: Notice
109+
href: ../NOTICE.md

‎External/Documentation/Images/MRTK_InputSimulation_CameraControlSettings.png‎

48.4 KB
Loading

‎External/Documentation/Images/MRTK_InputSimulation_HandControlSettings.png‎

19.8 KB
Loading

‎External/Documentation/Images/MRTK_InputSimulation_HandGestureSettings.png‎

26.9 KB
Loading

‎External/Documentation/Images/MRTK_InputSimulation_HandPlacementSettings.png‎

10.8 KB
Loading

‎External/Documentation/Images/MRTK_InputSimulation_HandRotationSettings.png‎

25.8 KB
Loading

‎External/Documentation/Images/MRTK_InputSimulation_HandSimulationMode.png‎

6.07 KB
Loading

‎External/Documentation/Images/MRTK_InputSimulation_HandSimulationSettings.png‎

88 KB
Loading

‎External/Documentation/Images/MRTK_InputSimulation_InputSimulationProfile.png‎

84.2 KB
Loading

0 commit comments

Comments
 (0)