Skip to content

Commit 7b48f35

Browse files
makseqmakseq
authored
fix: Keypoints model_index should start with 0 (#637)
1 parent bbfbeee commit 7b48f35

File tree

9 files changed

+284
-221
lines changed

9 files changed

+284
-221
lines changed

‎label_studio_ml/examples/yolo/README.md‎

Lines changed: 39 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -221,7 +221,7 @@ Here is an example of a prompt for this. It includes 1000 labels from YOLOv8 cla
221221
222222
</details>
223223
224-
## YOLOv5 and other YOLO models
224+
## YOLOv5 and other YOLO versions
225225
226226
YOLOv8 models have been successfully tested with this ML backend.
227227
@@ -426,32 +426,32 @@ More info: [Ultralytics YOLO Keypoint Documentation](https://docs.ultralytics.co
426426
model_add_bboxes="true" model_point_size="1"
427427
model_path="yolov8n-pose.pt"
428428
>
429-
<Label value="nose" predicted_values="person" model_index="1" background="red" />
429+
<Label value="nose" predicted_values="person" model_index="0" background="red" />
430430
431-
<Label value="left_eye" predicted_values="person" model_index="2" background="yellow" />
432-
<Label value="right_eye" predicted_values="person" model_index="3" background="yellow" />
431+
<Label value="left_eye" predicted_values="person" model_index="1" background="yellow" />
432+
<Label value="right_eye" predicted_values="person" model_index="2" background="yellow" />
433433
434-
<Label value="left_ear" predicted_values="person" model_index="4" background="purple" />
435-
<Label value="right_ear" predicted_values="person" model_index="5" background="purple" />
434+
<Label value="left_ear" predicted_values="person" model_index="3" background="purple" />
435+
<Label value="right_ear" predicted_values="person" model_index="4" background="purple" />
436436
437437
<View>
438-
<Label value="left_shoulder" predicted_values="person" model_index="6" background="green" />
439-
<Label value="left_elbow" predicted_values="person" model_index="8" background="green" />
440-
<Label value="left_wrist" predicted_values="person" model_index="10" background="green" />
438+
<Label value="left_shoulder" predicted_values="person" model_index="5" background="green" />
439+
<Label value="left_elbow" predicted_values="person" model_index="7" background="green" />
440+
<Label value="left_wrist" predicted_values="person" model_index="9" background="green" />
441441
442-
<Label value="right_shoulder" predicted_values="person" model_index="7" background="blue" />
443-
<Label value="right_elbow" predicted_values="person" model_index="9" background="blue" />
444-
<Label value="right_wrist" predicted_values="person" model_index="11" background="blue" />
442+
<Label value="right_shoulder" predicted_values="person" model_index="6" background="blue" />
443+
<Label value="right_elbow" predicted_values="person" model_index="8" background="blue" />
444+
<Label value="right_wrist" predicted_values="person" model_index="10" background="blue" />
445445
</View>
446446
447447
<View>
448-
<Label value="left_hip" predicted_values="person" model_index="12" background="brown" />
449-
<Label value="left_knee" predicted_values="person" model_index="14" background="brown" />
450-
<Label value="left_ankle" predicted_values="person" model_index="16" background="brown" />
448+
<Label value="left_hip" predicted_values="person" model_index="11" background="brown" />
449+
<Label value="left_knee" predicted_values="person" model_index="13" background="brown" />
450+
<Label value="left_ankle" predicted_values="person" model_index="15" background="brown" />
451451
452-
<Label value="right_hip" predicted_values="person" model_index="13" background="orange" />
453-
<Label value="right_knee" predicted_values="person" model_index="15" background="orange" />
454-
<Label value="right_ankle" predicted_values="person" model_index="17" background="orange" />
452+
<Label value="right_hip" predicted_values="person" model_index="12" background="orange" />
453+
<Label value="right_knee" predicted_values="person" model_index="14" background="orange" />
454+
<Label value="right_ankle" predicted_values="person" model_index="16" background="orange" />
455455
</View>
456456
</KeyPointLabels>
457457
@@ -463,12 +463,12 @@ More info: [Ultralytics YOLO Keypoint Documentation](https://docs.ultralytics.co
463463

464464
| Parameter | Type | Default | Description |
465465
|-------------------------|--------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
466-
| `model_path` | string | None | Path to the custom YOLO model. See more in the section "Custom YOLO Models." |
466+
| `model_path` | string | None | Path to the custom YOLO model. See more in the section "Custom YOLO Models." |
467467
| `model_score_threshold` | float | 0.5 | Sets the minimum confidence threshold for bounding box detections. Keypoints that are related to the detected bbox with a confidence below this threshold will be disregarded. |
468468
| `model_point_threshold` | float | 0.0 | Minimum confidence threshold for keypoints. Keypoints with confidence below this value will be ignored. |
469469
| `model_add_bboxes` | bool | True | Adds bounding boxes for detected keypoints. All keypoints will be grouped by parent bounding boxes on the region panel. See details in the tip below. |
470470
| `model_point_size` | float | 1 | Size of the keypoints in pixels. Just a visual parameter. |
471-
| `model_index` | int | None | Index of the keypoint in the YOLO model output. It's used in `Label` tags only to build mapping between a Label and an output point. |
471+
| `model_index` | int | None | Index of the keypoint in the YOLO model output starting from 0. It's used in `Label` tags only to build mapping between a Label and an output point. |
472472
473473
For example:
474474
@@ -509,32 +509,32 @@ You can use this labeling configuration to get rid of bounding boxes and keep on
509509
model_path="yolov8n-pose.pt" model_point_size="1"
510510
model_add_bboxes="false"
511511
>
512-
<Label value="nose" predicted_values="person" model_index="1" background="red" />
512+
<Label value="nose" predicted_values="person" model_index="0" background="red" />
513513
514-
<Label value="left_eye" predicted_values="person" model_index="2" background="yellow" />
515-
<Label value="right_eye" predicted_values="person" model_index="3" background="yellow" />
514+
<Label value="left_eye" predicted_values="person" model_index="1" background="yellow" />
515+
<Label value="right_eye" predicted_values="person" model_index="2" background="yellow" />
516516
517-
<Label value="left_ear" predicted_values="person" model_index="4" background="purple" />
518-
<Label value="right_ear" predicted_values="person" model_index="5" background="purple" />
517+
<Label value="left_ear" predicted_values="person" model_index="3" background="purple" />
518+
<Label value="right_ear" predicted_values="person" model_index="4" background="purple" />
519519
520520
<View>
521-
<Label value="left_shoulder" predicted_values="person" model_index="6" background="green" />
522-
<Label value="left_elbow" predicted_values="person" model_index="8" background="green" />
523-
<Label value="left_wrist" predicted_values="person" model_index="10" background="green" />
521+
<Label value="left_shoulder" predicted_values="person" model_index="5" background="green" />
522+
<Label value="left_elbow" predicted_values="person" model_index="7" background="green" />
523+
<Label value="left_wrist" predicted_values="person" model_index="9" background="green" />
524524
525-
<Label value="right_shoulder" predicted_values="person" model_index="7" background="blue" />
526-
<Label value="right_elbow" predicted_values="person" model_index="9" background="blue" />
527-
<Label value="right_wrist" predicted_values="person" model_index="11" background="blue" />
525+
<Label value="right_shoulder" predicted_values="person" model_index="6" background="blue" />
526+
<Label value="right_elbow" predicted_values="person" model_index="8" background="blue" />
527+
<Label value="right_wrist" predicted_values="person" model_index="10" background="blue" />
528528
</View>
529529
530530
<View>
531-
<Label value="left_hip" predicted_values="person" model_index="12" background="brown" />
532-
<Label value="left_knee" predicted_values="person" model_index="14" background="brown" />
533-
<Label value="left_ankle" predicted_values="person" model_index="16" background="brown" />
531+
<Label value="left_hip" predicted_values="person" model_index="11" background="brown" />
532+
<Label value="left_knee" predicted_values="person" model_index="13" background="brown" />
533+
<Label value="left_ankle" predicted_values="person" model_index="15" background="brown" />
534534
535-
<Label value="right_hip" predicted_values="person" model_index="13" background="orange" />
536-
<Label value="right_knee" predicted_values="person" model_index="15" background="orange" />
537-
<Label value="right_ankle" predicted_values="person" model_index="17" background="orange" />
535+
<Label value="right_hip" predicted_values="person" model_index="12" background="orange" />
536+
<Label value="right_knee" predicted_values="person" model_index="14" background="orange" />
537+
<Label value="right_ankle" predicted_values="person" model_index="16" background="orange" />
538538
</View>
539539
</KeyPointLabels>
540540
<Image name="image" value="$image" />
@@ -550,8 +550,8 @@ Each keypoint can be associated with a specific part of a person or object,
550550
and you can define this mapping using the `model_index` and `predicted_values` attributes.
551551
552552
```xml
553-
<Label value="left_eye" predicted_values="person" model_index="2" />
554-
<Label value="right_eye" predicted_values="person" model_index="3" />
553+
<Label value="left_eye" predicted_values="person" model_index="1" />
554+
<Label value="right_eye" predicted_values="person" model_index="2" />
555555
```
556556
557557
This configuration ensures that the keypoints detected by the YOLO model are correctly labeled in the Label Studio interface.

‎label_studio_ml/examples/yolo/control_models/base.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -198,4 +198,4 @@ def __str__(self):
198198

199199
class Config:
200200
arbitrary_types_allowed = True
201-
protected_namespaces = ('__.*__', '_.*') # Excludes 'model_'
201+
protected_namespaces = ("__.*__", "_.*") # Excludes 'model_'

‎label_studio_ml/examples/yolo/control_models/keypoint_labels.py‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ def is_control_matched(cls, control) -> bool:
3636

3737
def build_point_mapping(self):
3838
"""Build a mapping between points and Label Studio labels, e.g.
39-
<Label value="left_eye" predicted_values="person" model_index="2" /> => {"person::2": "left_eye"}
39+
<Label value="nose" predicted_values="person" model_index="0" /> => {"person::0": "nose"}
4040
"""
4141
mapping = {}
4242
for value, label_tag in self.control.labels_attrs.items():
@@ -80,12 +80,15 @@ def create_keypoints(self, results, path):
8080
) # Convert normalized keypoints to percentages
8181
model_label = model_names[int(results[0].boxes.cls[bbox_index])]
8282

83+
point_logs = "\n".join(
84+
[f' model_index="{i}", xy={xyn}' for i, xyn in enumerate(point_xyn)]
85+
)
8386
logger.debug(
8487
"----------------------\n"
8588
f"task id > {path}\n"
8689
f"type: {self.control}\n"
87-
f"keypoints > {point_xyn}\n"
8890
f"model label > {model_label}\n"
91+
f"keypoints >\n{point_logs}\n"
8992
f"confidences > {bbox_conf}\n"
9093
)
9194

@@ -115,7 +118,7 @@ def create_keypoints(self, results, path):
115118
logger.warning(
116119
f"Point {index_name} not found in point map, "
117120
f"you have to define it in the labeling config, e.g.:\n"
118-
f'<Label value="nose" predicted_values="person" index="1" />'
121+
f'<Label value="nose" predicted_values="person" model_index="0" />'
119122
)
120123
continue
121124
point_label = self.point_map[index_name]
@@ -126,10 +129,10 @@ def create_keypoints(self, results, path):
126129
"to_name": self.to_name,
127130
"type": "keypointlabels",
128131
"value": {
129-
"keypointlabels": [point_label], # Keypoint label
130-
"width": self.point_size
131-
/ image_width
132-
* 100, # Keypoint width, just visual styling
132+
# point label
133+
"keypointlabels": [point_label],
134+
# point width, just visual styling
135+
"width": self.point_size / image_width * 100,
133136
"x": x,
134137
"y": y,
135138
},

‎label_studio_ml/examples/yolo/control_models/timeline_labels.py‎

Lines changed: 34 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
BaseNN,
88
MultiLabelLSTM,
99
cached_feature_extraction,
10-
cached_yolo_predict
10+
cached_yolo_predict,
1111
)
1212
from utils.converter import (
1313
get_label_map,
@@ -22,7 +22,7 @@
2222
class TimelineLabelsModel(ControlModel):
2323
"""
2424
Class representing a TimelineLabels control tag for YOLO model.
25-
See README_TIMELINE_LABELS.md for more details.
25+
See README_TIMELINE_LABELS.md for more details.
2626
"""
2727

2828
type = "TimelineLabels"
@@ -50,7 +50,7 @@ def create(cls, *args, **kwargs):
5050
f"TimelinesLabels model works in simple mode (without training), "
5151
f"but no labels from YOLO model names are matched:\n{instance.control.name}\n"
5252
f"Add labels from YOLO model names to the labeling config or use `predicted_values` to map them. "
53-
f"As alternative option, you can set `model_trainable=\"true\"` in the TimelineLabels control tag "
53+
f'As alternative option, you can set `model_trainable="true"` in the TimelineLabels control tag '
5454
f"to train the model on the labels from the labeling config."
5555
)
5656
return instance
@@ -64,15 +64,21 @@ def predict_regions(self, video_path) -> List[Dict]:
6464
def create_timelines_simple(self, video_path):
6565
logger.debug(f"create_timelines_simple: {self.from_name}")
6666
# get yolo predictions
67-
frame_results = cached_yolo_predict(self.model, video_path, self.model.model_name)
67+
frame_results = cached_yolo_predict(
68+
self.model, video_path, self.model.model_name
69+
)
6870

6971
# Initialize a dictionary to keep track of ongoing segments for each label
7072
model_names = self.model.names
7173
needed_ids = [i for i, name in model_names.items() if name in self.label_map]
72-
needed_labels = [name for i, name in model_names.items() if name in self.label_map]
74+
needed_labels = [
75+
name for i, name in model_names.items() if name in self.label_map
76+
]
7377

7478
probs = [frame.probs.data[needed_ids].cpu().numpy() for frame in frame_results]
75-
label_map = {self.label_map[label]: idx for idx, label in enumerate(needed_labels)}
79+
label_map = {
80+
self.label_map[label]: idx for idx, label in enumerate(needed_labels)
81+
}
7682

7783
return convert_probs_to_timelinelabels(
7884
probs, label_map, self.control.name, self.model_score_threshold
@@ -81,7 +87,9 @@ def create_timelines_simple(self, video_path):
8187
def create_timelines_trainable(self, video_path):
8288
logger.debug(f"create_timelines_trainable: {self.from_name}")
8389
# extract features based on pre-trained yolo classification model
84-
frame_results = cached_feature_extraction(self.model, video_path, self.model.model_name)
90+
frame_results = cached_feature_extraction(
91+
self.model, video_path, self.model.model_name
92+
)
8593

8694
yolo_probs = [frame.probs for frame in frame_results]
8795
path = self.get_classifier_path(self.project_id)
@@ -95,7 +103,10 @@ def create_timelines_trainable(self, video_path):
95103
# run predict and convert to timelinelabels
96104
probs = classifier.predict(yolo_probs)
97105
regions = convert_probs_to_timelinelabels(
98-
probs, classifier.get_label_map(), self.control.name, self.model_score_threshold
106+
probs,
107+
classifier.get_label_map(),
108+
self.control.name,
109+
self.model_score_threshold,
99110
)
100111

101112
return regions
@@ -109,17 +120,20 @@ def fit(self, event, data, **kwargs):
109120
)
110121

111122
if event in ("ANNOTATION_CREATED", "ANNOTATION_UPDATED"):
112-
features, labels, label_map, project_id = self.load_features_and_labels(data)
123+
features, labels, label_map, project_id = self.load_features_and_labels(
124+
data
125+
)
113126
classifier, path = self.load_classifier(features, label_map, project_id)
114127
return self.train_classifier(classifier, features, labels, path)
115128

116129
def train_classifier(self, classifier, features, labels, path):
117-
""" Train the classifier model for timelinelabels using incremental partial learning.
118-
"""
130+
"""Train the classifier model for timelinelabels using incremental partial learning."""
119131
# Stop training when accuracy or f1 score reaches this threshold, it helps to avoid overfitting
120132
# because we partially train it on a small dataset from one annotation only
121133
get = self.control.attr.get
122-
epochs = int(get("model_classifier_epochs", 1000)) # Maximum number of training epochs
134+
epochs = int(
135+
get("model_classifier_epochs", 1000)
136+
) # Maximum number of training epochs
123137
f1_threshold = float(get("model_classifier_f1_threshold", 0.95))
124138
accuracy_threshold = float(get("model_classifier_accuracy_threshold", 1.00))
125139

@@ -129,13 +143,13 @@ def train_classifier(self, classifier, features, labels, path):
129143
labels,
130144
epochs=epochs,
131145
f1_threshold=f1_threshold,
132-
accuracy_threshold=accuracy_threshold
146+
accuracy_threshold=accuracy_threshold,
133147
)
134148
classifier.save_and_cache(path)
135149
return result
136150

137151
def load_classifier(self, features, label_map, project_id):
138-
""" Load or create a classifier model for timelinelabels.
152+
"""Load or create a classifier model for timelinelabels.
139153
1. Load neural network parameters from labeling config.
140154
2. Try loading classifier model from memory cache, then from disk.
141155
3. Or create a new classifier instance if there wasn't successful loading, or if parameters have changed.
@@ -155,11 +169,11 @@ def load_classifier(self, features, label_map, project_id):
155169
# Create a new classifier instance if it doesn't exist
156170
# or if labeling config has changed
157171
if (
158-
not classifier
159-
or classifier.label_map != label_map
160-
or classifier.sequence_size != sequence_size
161-
or classifier.hidden_size != hidden_size
162-
or classifier.num_layers != num_layers
172+
not classifier
173+
or classifier.label_map != label_map
174+
or classifier.sequence_size != sequence_size
175+
or classifier.hidden_size != hidden_size
176+
or classifier.num_layers != num_layers
163177
):
164178
logger.info("Creating a new classifier model for timelinelabels")
165179
input_size = len(features[0])
@@ -176,7 +190,7 @@ def load_classifier(self, features, label_map, project_id):
176190
return classifier, path
177191

178192
def load_features_and_labels(self, data):
179-
""" Load features and labels from the annotation
193+
"""Load features and labels from the annotation
180194
Args:
181195
data: event data, dictionary with keys 'task' and 'annotation'
182196
Returns:

0 commit comments

Comments
 (0)