Skip to content

Commit b2dac9d

Browse files
committed
Updates for manual
More is probably needed, but this is a start
1 parent 6979ac1 commit b2dac9d

File tree

4 files changed

+79
-0
lines changed

4 files changed

+79
-0
lines changed

com.unity.perception/Documentation~/PerceptionCamera.md

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,85 @@ _Example rendered object info for a single object_
6262

6363
The RenderedObjectInfoLabeler records a list of all objects visible in the Camera image, including its instance ID, resolved label ID and visible pixels. If Unity cannot resolve objects to a label in the IdLabelConfig, it does not record these objects.
6464

65+
### KeypointLabeler
66+
67+
The keypoint labeler captures keypoints of a labeled gameobject. The typical use of this labeler is capturing human pose
68+
estimation data. The labeler uses a [keypoint template](#KeypointTemplate) which defines the keypoints to capture for the
69+
model and the skeletal connections between those keypoints. The positions of the keypoints are recorded in pixel coordinates
70+
and saved to the captures json file.
71+
72+
```
73+
keypoints {
74+
label_id: <int> -- Integer identifier of the label
75+
instance_id: <str> -- UUID of the instance.
76+
template_guid: <str> -- UUID of the keypoint template
77+
pose: <str> -- Pose ground truth information
78+
keypoints [ -- Array of keypoint data, one entry for each keypoint defined in associated template file.
79+
{
80+
index: <int> -- Index of keypoint in template
81+
x: <float> -- X pixel coordinate of keypoint
82+
y: <float> -- Y pixel coordinate of keypoint
83+
state: <int> -- 0: keypoint does not exist, 1 keypoint exists
84+
}, ...
85+
]
86+
}
87+
```
88+
89+
#### Keypoint Template
90+
91+
keypoint templates are used to define the keypoints and skeletal connections captured by the KeypointLabeler. The keypoint
92+
template takes advantage of Unity's humanoid animation rig, and allows the user to automatically associate template keypoints
93+
to animation rig joints. Additionally, the user can choose to ignore the rigged points, or add points not defined in the rig.
94+
A Coco keypoint template is included in the perception package.
95+
96+
##### Editor
97+
98+
The keypoint template editor allows the user to create/modify a keypoint template. The editor consists of the header information,
99+
the keypoint array, and the skeleton array.
100+
101+
![Header section of the keypoint template](images/keypoint_template_header.png)
102+
<br/>_Header section of the keypoint template_
103+
104+
In the header section, a user can change the name of the template and supply textures that they would like to use for the keypoint
105+
visualization.
106+
107+
![The keypoint section of the keypoint template](images/keypoint_template_keypoints.png)
108+
<br/>_Keypoint section of the keypoint template_
109+
110+
The keypoint section allows the user to create/edit keypoints and associate them with Unity animation rig points. Each keypoint record
111+
has 4 fields: label (the name of the keypoint), Associate to Rig (a boolean value which, if true, automatically maps the keypoint to
112+
the gameobject defined by the rig), Rig Label (only needed if Associate To Rig is true, defines which rig component to associate with
113+
the keypoint), and Color (RGB color value of the keypoint in the visualization).
114+
115+
![Skeleton section of the keypoint template](images/keypoint_template_skeleton.png)
116+
<br/>_Skeleton section of the keypoint template_
117+
118+
The skeleton section allows the user to create connections between joints, basically defining the skeleton of a labeled object.
119+
120+
##### Format
121+
```
122+
annotation_definition.spec {
123+
template_id: <str> -- The UUID of the template
124+
template_name: <str> -- Human readable name of the template
125+
key_points [ -- Array of joints defined in this template
126+
{
127+
label: <str> -- The label of the joint
128+
index: <int> -- The index of the joint
129+
}, ...
130+
]
131+
skeleton [ -- Array of skeletal connections (which joints have connections between one another) defined in this template
132+
{
133+
joint1: <int> -- The first joint of the connection
134+
joint2: <int> -- The second joint of the connection
135+
}, ...
136+
]
137+
}
138+
```
139+
140+
#### Animation Pose Label
141+
142+
This file is used to define timestamps in an animation to a pose label.
143+
65144
## Limitations
66145

67146
Ground truth is not compatible with all rendering features, especially those that modify the visibility or shape of objects in the frame.
29.7 KB
Loading
49.7 KB
Loading
22.3 KB
Loading

0 commit comments

Comments
 (0)