Fix keypoint issues (#259)

* Fixing two bugs in KeypointLabeler:
AISV-1254 - Some keypoints fall outside of the image bounds
AISV-1257 - Partially visible characters with no visible keypoints are not reported

* Adding JointLabelEditor  with message about Labeling requirement

* Reorganizing keypoint docs and elaborating on setup requirements.

* Updating PerceptionCamera.md

* Fixing issues on 2020.2 and failing test

* Add object filtering modes to keypoint labeler

* Updating docs for object filtering

* Adding link to tutorial

* Moving KeypointLabeler.md to proper directory

* Update TUTORIAL.md

* Updating capitalization

Co-authored-by: Mohsen K <[email protected]>

* Apply doc suggestions

Co-authored-by: Mohsen K <[email protected]>

* Update PerceptionCamera.md based on feedback

* Update CHANGELOG.md

Co-authored-by: Mohsen K <[email protected]>

Author: JonathanHUnity

com.unity.perception/CHANGELOG.md

@@ -14,6 +14,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 ### Added
 
 ### Changed
+Expanded documentation on the Keypoint Labeler
+Updated Keypoint Labeler logic to only report keypoints for visible objects by default
 
 ### Deprecated
 

com.unity.perception/Documentation~/GroundTruth/KeypointLabeler.md

@@ -0,0 +1,91 @@
+# Keypoint Labeler
+
+The Keypoint Labeler captures the screen locations of specific points on labeled GameObjects. The typical use of this Labeler is capturing human pose estimation data, but it can be used to capture points on any kind of object. The Labeler uses a [Keypoint Template](#KeypointTemplate) which defines the keypoints to capture for the model and the skeletal connections between those keypoints. The positions of the keypoints are recorded in pixel coordinates.
+
+## Data Format
+The keypoints captured each frame are in the following format:
+```
+keypoints {
+  label_id:      <int>   -- Integer identifier of the label
+  instance_id:   <str>   -- UUID of the instance.
+  template_guid: <str>   -- UUID of the keypoint template
+  pose:          <str>   -- Current pose
+  keypoints [            -- Array of keypoint data, one entry for each keypoint defined in associated template file.
+    {
+      index:     <int>   -- Index of keypoint in template
+      x:         <float> -- X pixel coordinate of keypoint
+      y:         <float> -- Y pixel coordinate of keypoint
+      state:     <int>   -- Visibility state
+    }, ...
+  ]
+}
+```
+
+The `state` entry has three possible values: 
+* 0 - the keypoint either does not exist or is outside of the image's bounds
+* 1 - the keypoint exists inside of the image bounds but cannot be seen because the object is not visible at its location in the image
+* 2 - the keypoint exists and the object is visible at its location
+
+The annotation definition, captured by the Keypoint Labeler once in each dataset, describes points being captured and their skeletal connections. These are defined by the [Keypoint Template](#KeypointTemplate).
+```
+annotation_definition.spec {
+  template_id:       <str>           -- The UUID of the template
+  template_name:     <str>           -- Human readable name of the template
+  key_points [                       -- Array of joints defined in this template
+    {
+      label:         <str>           -- The label of the joint
+      index:         <int>           -- The index of the joint
+    }, ...
+  ]
+  skeleton [                         -- Array of skeletal connections (which joints have connections between one another) defined in this template
+    {
+      joint1:        <int>           -- The first joint of the connection
+      joint2:        <int>           -- The second joint of the connection
+    }, ...
+  ]
+}
+```
+
+## Setup
+The Keypoint Labeler captures keypoints each frame from each object in the scene that meets the following conditions:
+
+* The object or its children are at least partially visible in the frame 
+  * The _Object Filter_ option on the Keypoint Labeler can be used to also include fully occluded or off-screen objects
+* The root object has a `Labeling` component
+* The object matches at least one entry in the Keypoint Template by either:
+  * Containing an Animator with a [humanoid avatar](https://docs.unity3d.com/Manual/ConfiguringtheAvatar.html) whose rig matches a keypoint OR
+  * Containing children with Joint Label components whose labels match keypoints
+
+For a tutorial on setting up your project for keypoint labeling, see the [Human Pose Labeling and Randomization Tutorial](../HPTutorial/TUTORIAL.md).
+
+## Keypoint Template
+
+Keypoint Templates are used to define the keypoints and skeletal connections captured by the Keypoint Labeler. The Keypoint Template takes advantage of Unity's humanoid animation rig, and allows the user to automatically associate template keypoints to animation rig joints. Additionally, the user can choose to ignore the rigged points, or add points not defined in the rig.
+
+A [COCO](https://cocodataset.org/#home) Keypoint Template is included in the Perception package.
+
+### Editor
+
+The Keypoint Template editor allows the user to create/modify a Keypoint Template. The editor consists of the header information, the keypoint array, and the skeleton array.
+
+![Header section of the keypoint template](../images/keypoint_template_header.png)
+<br/>_Header section of the keypoint template_
+
+In the header section, a user can change the name of the template and supply textures that they would like to use for the keypoint visualization.
+
+![The keypoint section of the keypoint template](../images/keypoint_template_keypoints.png)
+<br/>_Keypoint section of the keypoint template_
+
+The keypoint section allows the user to create/edit keypoints and associate them with Unity animation rig points. Each keypoint record
+has 4 fields: label (the name of the keypoint), Associate to Rig (a boolean value which, if true, automatically maps the keypoint to
+the GameObject defined by the rig), Rig Label (only needed if Associate To Rig is true, defines which rig component to associate with
+the keypoint), and Color (RGB color value of the keypoint in the visualization).
+
+![Skeleton section of the keypoint template](../images/keypoint_template_skeleton.png)
+<br/>_Skeleton section of the keypoint template_
+
+The skeleton section allows the user to create connections between joints, basically defining the skeleton of a labeled object.
+
+#### Animation Pose Label
+
+This file is used to define timestamps in an animation to a pose label.

com.unity.perception/Documentation~/HPTutorial/TUTORIAL.md

@@ -248,7 +248,7 @@ You can now check out the output dataset to see what the annotations look like.
 }
 ```
 
-In the above annotation, all of the 18 joints defined in the COCO template we used are listed. For each joint that is present in our character, you can see the X and Y coordinates within the captured frame. However, you may notice three of the joints are listed with (0,0) coordinates. These joints are not present in our character. A fact that is also denoted by the `state` field. A state of **0** means the joint was not present, **1** denotes a joint that is present but not visible (to be implemented in a later version of the package), and **2** means the joint was present and visible.
+In the above annotation, all of the 18 joints defined in the COCO template we used are listed. For each joint that is present in our character, you can see the X and Y coordinates within the captured frame. However, you may notice three of the joints are listed with (0,0) coordinates. These joints are not present in our character. A fact that is also denoted by the `state` field. A state of **0** means the joint either does not exist or is outside of the image's bounds, **1** denotes a joint that is inside of the image but cannot be seen because the part of the object it belongs to is not visible in the image, and **2** means the joint was present and visible.
 
 You may also note that the `pose` field has a value of `unset`. This is because we have not defined poses for our animation clip and `Perception Camera` yet. We will do this next.
 

com.unity.perception/Documentation~/PerceptionCamera.md

@@ -75,83 +75,13 @@ The ObjectCountLabeler records object counts for each label you define in the Id
 ```
 _Example rendered object info for a single object_
 
-The RenderedObjectInfoLabeler records a list of all objects visible in the Camera image, including its instance ID, resolved label ID and visible pixels. If Unity cannot resolve objects to a label in the IdLabelConfig, it does not record these objects.
+The `RenderedObjectInfoLabeler` records a list of all objects visible in the camera image, including their instance IDs, resolved label IDs, and visible pixel counts. If Unity cannot resolve objects to a label in the `IdLabelConfig`, it does not record these objects.
 
-### KeypointLabeler
+### Keypoint Labeler
 
-The keypoint labeler captures keypoints of a labeled gameobject. The typical use of this labeler is capturing human pose estimation data. The labeler uses a [keypoint template](#KeypointTemplate) which defines the keypoints to capture for the model and the skeletal connections between those keypoints. The positions of the keypoints are recorded in pixel coordinates. Each keypoint has a state value: 0 - the keypoint either does not exist or is outside of the image's bounds, 1 - the keypoint exists and is inside of the image's bounds but cannot be seen because it is occluded by another object, and 2 - the keypoint exists and is visible.
+The keypoint labeler captures the screen locations of specific points on labeled GameObjects. The typical use of this Labeler is capturing human pose estimation data, but it can be used to capture points on any kind of object. The Labeler uses a [Keypoint Template](#KeypointTemplate) which defines the keypoints to capture for the model and the skeletal connections between those keypoints. The positions of the keypoints are recorded in pixel coordinates.
 
-```
-keypoints {
-  label_id:      <int>   -- Integer identifier of the label
-  instance_id:   <str>   -- UUID of the instance.
-  template_guid: <str>   -- UUID of the keypoint template
-  pose:          <str>   -- Pose ground truth information
-  keypoints [            -- Array of keypoint data, one entry for each keypoint defined in associated template file.
-    {
-      index:     <int>   -- Index of keypoint in template
-      x:         <float> -- X pixel coordinate of keypoint
-      y:         <float> -- Y pixel coordinate of keypoint
-      state:     <int>   -- 0: keypoint does not exist, 1: keypoint exists but is not visible, 2: keypoint exists and is visible
-    }, ...
-  ]
-}
-```
-
-#### Keypoint Template
-
-keypoint templates are used to define the keypoints and skeletal connections captured by the KeypointLabeler. The keypoint
-template takes advantage of Unity's humanoid animation rig, and allows the user to automatically associate template keypoints
-to animation rig joints. Additionally, the user can choose to ignore the rigged points, or add points not defined in the rig.
-A Coco keypoint template is included in the perception package.
-
-##### Editor
-
-The keypoint template editor allows the user to create/modify a keypoint template. The editor consists of the header information,
-the keypoint array, and the skeleton array.
-
-![Header section of the keypoint template](images/keypoint_template_header.png)
-<br/>_Header section of the keypoint template_
-
-In the header section, a user can change the name of the template and supply textures that they would like to use for the keypoint
-visualization.
-
-![The keypoint section of the keypoint template](images/keypoint_template_keypoints.png)
-<br/>_Keypoint section of the keypoint template_
-
-The keypoint section allows the user to create/edit keypoints and associate them with Unity animation rig points. Each keypoint record
-has 4 fields: label (the name of the keypoint), Associate to Rig (a boolean value which, if true, automatically maps the keypoint to
-the gameobject defined by the rig), Rig Label (only needed if Associate To Rig is true, defines which rig component to associate with
-the keypoint), and Color (RGB color value of the keypoint in the visualization).
-
-![Skeleton section of the keypoint template](images/keypoint_template_skeleton.png)
-<br/>_Skeleton section of the keypoint template_
-
-The skeleton section allows the user to create connections between joints, basically defining the skeleton of a labeled object.
-
-##### Format
-```
-annotation_definition.spec {
-  template_id:       <str>           -- The UUID of the template
-  template_name:     <str>           -- Human readable name of the template
-  key_points [                       -- Array of joints defined in this template
-    {
-      label:         <str>           -- The label of the joint
-      index:         <int>           -- The index of the joint
-    }, ...
-  ]
-  skeleton [                         -- Array of skeletal connections (which joints have connections between one another) defined in this template
-    {
-      joint1:        <int>           -- The first joint of the connection
-      joint2:        <int>           -- The second joint of the connection
-    }, ...
-  ]
-}
-```
-
-#### Animation Pose Label
-
-This file is used to define timestamps in an animation to a pose label.
+For more information, see [Keypoint Labeler](GroundTruth/KeypointLabeler.md) or the [Human Pose Labeling and Randomization Tutorial](HPTutorial/TUTORIAL.md)
 
 ## Limitations
 

com.unity.perception/Editor/GroundTruth/JointLabelEditor.cs

@@ -0,0 +1,22 @@
+using System.Linq;
+using UnityEngine;
+using UnityEngine.Perception.GroundTruth;
+
+namespace UnityEditor.Perception.GroundTruth
+{
+    [CustomEditor(typeof(JointLabel))]
+    public class JointLabelEditor : Editor
+    {
+        public override void OnInspectorGUI()
+        {
+            base.OnInspectorGUI();
+#if UNITY_2020_1_OR_NEWER
+            //GetComponentInParent<T>(bool includeInactive) only exists on 2020.1 and later
+            if (targets.Any(t => ((Component)t).gameObject.GetComponentInParent<Labeling>(true) == null))
+#else
+            if (targets.Any(t => ((Component)t).GetComponentInParent<Labeling>() == null))
+#endif
+                EditorGUILayout.HelpBox("No Labeling component detected on parents. Keypoint labeling requires a Labeling component on the root of the object.", MessageType.Info);
+        }
+    }
+}