Aisv647 visualizations - Preview Cut (#40)

* Initial cut of labeler visualizers

* Added event system to the scene so that the UI would work

* Created a shader that can remove the background of the segmentation image

* Changed package for the shader

* Now with sliders in UI for transparency values

* Heads up display information now available

* PR fixes

Some fairly major UI architecture changes to address comments made during the initial cut PR. Things make more sense now.

* Fixes so that rendering is not a frame behind

* Graceful-ish handling of more than 1 perception camera for visualization

* Removal of expired visualization artifacts fixes for SynthDet

Removed bounding boxes and HUD entries for objects no longer being visualized, added performance improvements to handle complete scene changes every frame

* Fixes for scroll bars on HUD and UI Controls on top of UI objects

* Object pooling for HUD entries

* Adding tests for visualization init and destroy.

* Updated control panel to not need labels to add new controls

* Moving visualization to screen space - camera and unifying into a single canvas to ensure good behavior with multiple cameras.

* Bounding box visualizer fix to respond to screen resize

* Sort of working enable/disable visualizations

* Removing extra logic around enabling or disabling visualizations. Still have a bug with labelers being reset when serialized fields change, resulting in duplicate UI.

* Disabling visualizations in non-development players. Making it so the inspector is non-editable in play mode to work around the myriad of issues with inspector changes in play mode seemingly stemming from usages of SerializeReference

* Updated documentation

* Updated documentation

* Updated documentation

* Updated changelog to 0.3.0-preview.1

* Fixing pip install issues

* Fixed controls property getter

* Attempting to fix linux builds. Fixing line ending mismatches when checking out with linux file endings.

* Fixing semantic segmentation tests when visualizations are enabled

* Fixing some tests wreaked by newline issues. Commenting out renderdoc code to get semantic segmentation tests to compile in players.

* Removing final usage of UnityEditorInternal to fix tests

* Fix for _MainTex test error

* Fix for merge issue

Co-authored-by: Jon Hogins <[email protected]>

Author: StevenBorkman

.gitignore

@@ -34,4 +34,5 @@ upm-ci~/*
 **/.bin
 CodeCoverage
 /.download
-/TestProjects/PerceptionURP/Build
+**/Build/**
+**/Builds/**

TestProjects/PerceptionURP/Assets/Scenes/SampleScene.unity

@@ -440,7 +440,7 @@ Light:
   m_BounceIntensity: 1
   m_ColorTemperature: 6570
   m_UseColorTemperature: 0
-  m_BoundingSphereOverride: {x: 0, y: 1e-45, z: 1.8643e-41, w: 5.9e-44}
+  m_BoundingSphereOverride: {x: 4.7e-43, y: 0, z: 0, w: 0}
   m_UseBoundingSphereOverride: 0
   m_ShadowRadius: 0
   m_ShadowAngle: 0
@@ -484,6 +484,7 @@ GameObject:
   - component: {fileID: 963194229}
   - component: {fileID: 963194227}
   - component: {fileID: 963194231}
+  - component: {fileID: 963194232}
   m_Layer: 0
   m_Name: Main Camera
   m_TagString: MainCamera
@@ -608,6 +609,7 @@ MonoBehaviour:
   - id: 1
   - id: 2
   - id: 3
+  showVisualizations: 1
   references:
     version: 1
     00000000:
@@ -653,8 +655,20 @@ MonoBehaviour:
   m_Script: {fileID: 11500000, guid: 7c51d9f2c5784bb4aee3fdf021966e14, type: 3}
   m_Name: 
   m_EditorClassIdentifier: 
-  light: {fileID: 705507993}
+  targetLight: {fileID: 705507993}
   target: {fileID: 1640252278}
+--- !u!114 &963194232
+MonoBehaviour:
+  m_ObjectHideFlags: 0
+  m_CorrespondingSourceObject: {fileID: 0}
+  m_PrefabInstance: {fileID: 0}
+  m_PrefabAsset: {fileID: 0}
+  m_GameObject: {fileID: 963194225}
+  m_Enabled: 1
+  m_EditorHideFlags: 0
+  m_Script: {fileID: 11500000, guid: 53f4c974fdf704444959724a41de0cfe, type: 3}
+  m_Name: 
+  m_EditorClassIdentifier: 
 --- !u!1 &1640252278
 GameObject:
   m_ObjectHideFlags: 0

TestProjects/PerceptionURP/ProjectSettings/EditorBuildSettings.asset

@@ -4,5 +4,8 @@
 EditorBuildSettings:
   m_ObjectHideFlags: 0
   serializedVersion: 2
-  m_Scenes: []
+  m_Scenes:
+  - enabled: 1
+    path: Assets/Scenes/SampleScene.unity
+    guid: 9fc0d4010bbf28b4594072e72b8655ab
   m_configObjects: {}

com.unity.perception/CHANGELOG.md

@@ -1,4 +1,5 @@
 # Changelog
+
 All notable changes to this package will be documented in this file.
 
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
@@ -19,18 +20,34 @@ Fixed compilation warnings with latest com.unity.simulation.core package.
 
 ### Security
 
+## [0.3.0-preview.1] - 2020-07-31
+
+### Added
+
+Added realtime visualization capability to the perception package.
+
+Added visualizers for built-in labelers: Semantic Segmentation, 2D Bounding Boxes, Object Count, and Render Object Info.
+
 ## [0.2.0-preview.2] - 2020-07-15
+
 ### Fixed
+
 Fixed bug that prevented RGB captures to be written out to disk
 Fixed compatibility with [email protected]
 
 ## [0.2.0-preview.1] - 2020-07-02
+
 ### Added
+
 Added CameraLabeler, an extensible base type for all forms of dataset output from a camera.
 Added LabelConfig\<T\>, a base class for mapping labels to data used by a labeler. There are two new derived types - ID label config and semantic segmentation label config.
+
 ### Changed
+
 Moved the various forms of ground truth from PerceptionCamera into various subclasses of CameraLabeler.
 Renamed SimulationManager to DatasetCapture.
 Changed Semantic Segmentation to take a SemanticSegmentationLabelConfig, which maps labels to color pixel values.
+
 ## [0.1.0] - 2020-06-24
+
 ### This is the first release of the _Perception_ package

com.unity.perception/Documentation~/GettingStarted.md

@@ -1,10 +1,13 @@
 # Getting Started with Perception
+
 This walkthrough will provide creating a new scene for generating perception datasets including segmentation data and image captures.
 
 ## Step 1: Create a project and install Perception
+
 First, follow [this guide](SetupSteps.md) to install Perception in your project.
 
 ## Step 2 (for Universal Render Pipeline projects): Add GroundTruthRendererFeature
+
 This step can be skipped for HDRP projects.
 
 1. Select your project's `ScriptableRenderer` asset and open the inspector window. In most projects it is located at `Assets/Settings/ForwardRenderer.asset`.
@@ -13,17 +16,19 @@ This step can be skipped for HDRP projects.
 <img src="images/ScriptableRendererStep.png" align="middle"/>
 
 ## Step 3: Create a new scene and camera
+
 1. Create a new scene using File -> New Scene
 2. `ctrl+s` to save the scene and give it a name
-3. Select the Main Camera and reset the Position transform to 0 
+3. Select the Main Camera and reset the Position transform to 0
 4. In the Hierarchy window select the main camera
-	1. In the inspector panel of the main camera select Add Component
-	2. Add a **Perception Camera** component
+   1. In the inspector panel of the main camera select Add Component
+   2. Add a **Perception Camera** component
 
 <img src="images/PerceptionCameraFinished.png" align="middle"/>
 
 ## Step 4: Create labeled objects
-1. Create a cube by right-clicking in the Hierarchy window, select 3D Object -> Cube 
+
+1. Create a cube by right-clicking in the Hierarchy window, select 3D Object -> Cube
 2. Create 2 more cubes this way
 3. Change the names of the cubes to Cube, Box, and Crate
 4. Position the Cubes in front of the main Camera
@@ -49,19 +54,39 @@ This step can be skipped for HDRP projects.
 <img src="images/MainCameraLabelConfig.PNG" align="middle"/>
 
 ## Step 5: Run simulation and generate dataset
+
 1. Press play in the editor, allow the scene to run for a few seconds, and then exit playmode
 2. In the console log you will see a Shutdown in Progress message that will show a file path to the location of the generated dataset.
->Example file path on a Windows PC : `C:/Users/<User Name>/AppData/LocalLow/DefaultCompany/UnityTestFramework\2e10ec21-9d97-4cee-b5a2-7e95e299afa4\RGB18f61842-ef8d-4b31-acb5-cb1da36fb7b1`
+
+> Example file path on a Windows PC : `C:/Users/<User Name>/AppData/LocalLow/DefaultCompany/UnityTestFramework\2e10ec21-9d97-4cee-b5a2-7e95e299afa4\RGB18f61842-ef8d-4b31-acb5-cb1da36fb7b1`
+
 3. In the dataset folder you will find the following data:
-	- RGB captures 
-	- Semantic segmentation images
-	- Logs
-	- JSON dataset
-	
+   - RGB captures
+   - Semantic segmentation images
+   - Logs
+   - JSON dataset
+
 <img src="images/rgb_2.png" align="middle"/>
 
 _RGB image_
 
 <img src="images/segmentation_2.png" align="middle"/>
 
-_Example semantic segmentation image_
+_Example semantic segmentation image_
+
+## Optional Step: Realtime visualization of labelers
+
+The perception package now comes with the ability to show realtime results of the labeler in the scene. To enable this capability:
+
+<img src="images/visualized.png" align="middle"/>
+
+_Example of perception running with show visualizations on_
+
+1. To use the visualizer, verify that *Show Visualizations* is checked on in the Inspector pane. This turns on the built in labelers which includes segmentation data, 2D bounding boxes, pixel and object counts.
+2. Turning on the visualizer creates new UI controls in the editor's game view. These controls allow you to atomically control each of the individual visualizers. Each individual can be turned on/off on their own. Some visualizers also include controls to change their output.
+
+<img src="images/controls.gif" align="middle"/>
+
+_Visualization controls in action_
+
+***Important Note:*** The perception package takes advantage of asynchronous processing to ensure reasonable frame rates of a scene. A side effect of realtime visualization is that the labelers have to be applied to the capture in its actual frame, which will potentially adversely affect the scene's framerate.

com.unity.perception/Documentation~/images/controls.gif

@@ -78,14 +78,18 @@ void AddLabeler(Type labelerType)
 
         public override void OnInspectorGUI()
         {
-            EditorGUILayout.PropertyField(serializedObject.FindProperty(nameof(perceptionCamera.description)));
-            EditorGUILayout.PropertyField(serializedObject.FindProperty(nameof(perceptionCamera.period)));
-            EditorGUILayout.PropertyField(serializedObject.FindProperty(nameof(perceptionCamera.startTime)));
-            EditorGUILayout.PropertyField(serializedObject.FindProperty(nameof(perceptionCamera.captureRgbImages)));
-
-            //EditorGUILayout.PropertyField(serializedObject.FindProperty(nameof(PerceptionCamera.labelers)));
-            m_LabelersList.DoLayoutList();
-
+            using(new EditorGUI.DisabledScope(EditorApplication.isPlaying))
+            {
+                EditorGUILayout.PropertyField(serializedObject.FindProperty(nameof(perceptionCamera.description)));
+                EditorGUILayout.PropertyField(serializedObject.FindProperty(nameof(perceptionCamera.period)));
+                EditorGUILayout.PropertyField(serializedObject.FindProperty(nameof(perceptionCamera.startTime)));
+                EditorGUILayout.PropertyField(serializedObject.FindProperty(nameof(perceptionCamera.showVisualizations)));
+                EditorGUILayout.PropertyField(serializedObject.FindProperty(nameof(perceptionCamera.captureRgbImages)));
+                serializedObject.ApplyModifiedProperties();
+
+                //EditorGUILayout.PropertyField(serializedObject.FindProperty(nameof(PerceptionCamera.labelers)));
+                m_LabelersList.DoLayoutList();
+            }
             if (EditorSettings.asyncShaderCompilation)
             {
                 EditorGUILayout.HelpBox("Asynchronous shader compilation may result in invalid data in beginning frames. This can be disabled in Project Settings -> Edtior -> Asynchronous Shader Compilation", MessageType.Warning);

com.unity.perception/Documentation~/images/visualized.png

@@ -1,9 +1,11 @@
-using System;
+using System;
 using System.Collections.Generic;
 using System.Diagnostics.CodeAnalysis;
 using Unity.Collections;
 using Unity.Profiling;
 using UnityEngine.Serialization;
+using Unity.Simulation;
+using UnityEngine.UI;
 
 namespace UnityEngine.Perception.GroundTruth
 {
@@ -42,6 +44,10 @@ struct BoundingBoxValue
         AnnotationDefinition m_BoundingBoxAnnotationDefinition;
         BoundingBoxValue[] m_BoundingBoxValues;
 
+        private GameObject visualizationHolder = null;
+
+        private Vector2 originalScreenSize = Vector2.zero;
+
         /// <summary>
         /// Creates a new BoundingBox2DLabeler. Be sure to assign <see cref="idLabelConfig"/> before adding to a <see cref="PerceptionCamera"/>.
         /// </summary>
@@ -58,6 +64,9 @@ public BoundingBox2DLabeler(IdLabelConfig labelConfig)
             this.idLabelConfig = labelConfig;
         }
 
+        /// <inheritdoc/>
+        protected override bool supportsVisualization => true;
+
         /// <inheritdoc/>
         protected override void Setup()
         {
@@ -70,6 +79,12 @@ protected override void Setup()
                 "Bounding box for each labeled object visible to the sensor", id: new Guid(annotationId));
 
             perceptionCamera.RenderedObjectInfosCalculated += OnRenderedObjectInfosCalculated;
+
+            visualizationEnabled = supportsVisualization;
+
+            // Record the original screen size. The screen size can change during play, and the visual bounding
+            // boxes need to be scaled appropriately
+            originalScreenSize = new Vector2(Screen.width, Screen.height);
         }
 
         /// <inheritdoc/>
@@ -108,8 +123,80 @@ void OnRenderedObjectInfosCalculated(int frameCount, NativeArray<RenderedObjectI
                     };
                 }
 
+                if (!CaptureOptions.useAsyncReadbackIfSupported && frameCount != Time.frameCount)
+                    Debug.LogWarning("Not on current frame: " + frameCount + "(" + Time.frameCount + ")");
+
+                if (visualizationEnabled)
+                {
+                    Visualize();
+                }
+
                 asyncAnnotation.ReportValues(m_BoundingBoxValues);
             }
         }
+
+        /// <inheritdoc/>
+        protected override void PopulateVisualizationPanel(ControlPanel panel)
+        {
+            panel.AddToggleControl("BoundingBoxes", enabled => {
+                visualizationEnabled = enabled;
+            });
+
+            objectPool = new List<GameObject>();
+
+            visualizationHolder = new GameObject("BoundsHolder" + Time.frameCount);
+            visualizationCanvas.AddComponent(visualizationHolder);
+        }
+
+        void ClearObjectPool(int count)
+        {
+            for (int i = count; i < objectPool.Count; i++)
+            {
+                objectPool[i].SetActive(false);
+            }
+        }
+
+        List<GameObject> objectPool = null;
+
+        void Visualize()
+        {
+            ClearObjectPool(m_BoundingBoxValues.Length);
+
+            // The player screen can be dynamically resized during play, need to
+            // scale the bounding boxes appropriately from the original screen size
+            float screenRatioWidth = Screen.width / originalScreenSize.x;
+            float screenRatioHeight = Screen.height / originalScreenSize.y;
+
+            for (int i = 0; i < m_BoundingBoxValues.Length; i++)
+            {
+                var boxVal = m_BoundingBoxValues[i];
+
+                if (i >= objectPool.Count)
+                {
+                    var boundingBoxObject = GameObject.Instantiate(Resources.Load<GameObject>("BoundingBoxPrefab"));
+                    objectPool.Add(boundingBoxObject);
+                    var rectTransform = (RectTransform)boundingBoxObject.transform;
+                    rectTransform.SetParent(visualizationHolder.transform, false);
+                }
+
+                if (!objectPool[i].activeSelf) objectPool[i].SetActive(true);
+
+                string label = boxVal.label_name + "_" + boxVal.instance_id;
+                objectPool[i].GetComponentInChildren<Text>().text = label;
+
+                var rectTrans = objectPool[i].transform as RectTransform;
+
+                rectTrans.anchoredPosition = new Vector2(boxVal.x * screenRatioWidth, -boxVal.y * screenRatioHeight);
+                rectTrans.SetSizeWithCurrentAnchors(RectTransform.Axis.Horizontal, boxVal.width * screenRatioWidth);
+                rectTrans.SetSizeWithCurrentAnchors(RectTransform.Axis.Vertical, boxVal.height * screenRatioHeight);
+            }
+        }
+
+        /// <inheritdoc/>
+        override protected void OnVisualizerEnabledChanged(bool enabled)
+        {
+            if (visualizationHolder != null)
+                visualizationHolder.SetActive(enabled);
+        }
     }
 }