Skip to content

Commit 3beea87

Browse files
talmoclaude
andauthored
Port Python sleap-io PRs #366-373: BoundingBox, GeoJSON, ROI enhancements, bug fixes (#74)
* Add GeoJSON I/O, new geometry types, ROI instance serialization (format 1.6) Port Python PRs #366, #367, #371: - Extend Geometry union with LineString, MultiPoint, GeometryCollection - Add WKB encode/decode for all new geometry types - Add fromMultiPolygon() static factory and explode() instance method - Add toGeoJSON() method on ROI for Feature serialization - Create src/io/geojson.ts with roisToGeoJSON, roisFromGeoJSON, writeGeoJSON, readGeoJSON - Add ROI instance column to SLP format (v1.6) for read/write - Add _instanceIdx deferred resolution in Labels.materialize() - Update bounds, area, _allPoints, rasterizeGeometry for new geometry types - Export geojson module from both Node and browser entry points - Add comprehensive tests for all new functionality (43 tests in roi.test.ts, 5 in geojson.test.ts) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix numpy output truncation and negative frame handling in lazy/dict codecs Port two Python bug fixes: - PR #368: numpy() output was truncated at last labeled frame instead of spanning the full video length. Fixed in both Labels.numpy() and LazyDataStore.toNumpy() by checking video.shape[0]. - PR #369: negative frames were silently dropped in lazy loading and dictionary codec. Fixed by reading /negative_frames in readSlpLazy(), passing them through LazyDataStore, preserving them in dictionary toDict/fromDict round-trips, and keeping negative frames when skipEmptyFrames is enabled. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add first-class BoundingBox type with full SLP I/O support Port of Python PR #373. Introduces UserBoundingBox and PredictedBoundingBox model classes, simplifies ROI/SegmentationMask by removing annotationType and score fields, adds bboxes field to Labels with getBboxes() query method, implements SLP read/write for bboxes (format v1.7), and migrates old bbox-type ROIs to BoundingBox objects on read. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix review issues: xywh consistency, geojson test nesting, separator alignment - BoundingBox.xywh now returns AABB dimensions for rotated bboxes - Fix fromPolygon call in geojson test (wrong coordinate nesting) - Align negative frame key separator to use ':' consistently - Remove redundant type assertion in Labels.getBboxes() Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Address code review: clean up redundant code, add type safety, add ROI instance test - Remove unnecessary intermediate variable in readSlpLazy destructuring - Add @internal annotation to BoundingBox._instanceIdx for consistency with ROI - Add explicit SegmentationMask return type to BoundingBox.toMask() - Use String() coercion instead of unsafe `as string` casts in GeoJSON parser - Extract readRoisAndBboxes() helper to deduplicate eager/lazy read paths - Add ROI instance association (format 1.6) round-trip tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Update docs for BoundingBox, GeoJSON I/O, and format 1.6/1.7 - docs/api.md: Add BoundingBox/GeoJSON sections, remove stale annotationType/score references, document new ROI methods - docs/usage.md: Update ROI section with bbox and GeoJSON examples - docs/index.md: Update format version range to 1.0-1.7 - README.md: Add bbox/GeoJSON to feature list Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 22a5399 commit 3beea87

23 files changed

+1537
-173
lines changed

README.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,11 @@ JavaScript/TypeScript utilities for reading and writing SLEAP `.slp` files with
1111

1212
## Features
1313

14-
- SLP read/write with format compatibility (including embedded frames via HDF5 video datasets).
14+
- SLP read/write with format compatibility (format 1.0–1.7, including embedded frames via HDF5 video datasets).
1515
- Browser-compatible SLP writing via `saveSlpToBytes()`.
1616
- Streaming-friendly file access (URL, `File`, `FileSystemFileHandle`, `Blob`).
1717
- Core data model (`Labels`, `LabeledFrame`, `Instance`, `Skeleton`, `Video`, etc.).
18+
- ROI, segmentation mask, and bounding box annotations with GeoJSON I/O.
1819
- Video backends accept `string`, `File`, or `Blob` sources.
1920
- Browser-safe: Node.js-only dependencies (`skia-canvas`, `child_process`) are dynamically imported, so bundlers can tree-shake them.
2021
- Dictionary and numpy codecs for interchange.

docs/api.md

Lines changed: 85 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -18,12 +18,18 @@ import {
1818
LabelsSet,
1919
SuggestionFrame,
2020
ROI,
21-
AnnotationType,
2221
SegmentationMask,
22+
BoundingBox,
23+
UserBoundingBox,
24+
PredictedBoundingBox,
2325
toDict,
2426
fromDict,
2527
toNumpy,
2628
fromNumpy,
29+
roisToGeoJSON,
30+
roisFromGeoJSON,
31+
writeGeoJSON,
32+
readGeoJSON,
2733
} from "@talmolab/sleap-io.js";
2834
```
2935

@@ -142,7 +148,8 @@ Key types:
142148
- `Video`, `SuggestionFrame`, `LabelsSet`
143149
- `Camera`, `CameraGroup`, `RecordingSession` (camera utilities)
144150
- `LazyDataStore`, `LazyFrameList` (lazy loading)
145-
- `ROI`, `SegmentationMask`, `AnnotationType` (spatial annotations)
151+
- `ROI`, `SegmentationMask` (spatial annotations)
152+
- `BoundingBox`, `UserBoundingBox`, `PredictedBoundingBox` (detection/tracking)
146153
- `Point` has fields `{ xy, visible, complete, score? }` where `score` is an optional confidence value.
147154
- `LabeledFrame.isNegative` (`boolean`, default `false`): marks negative-annotated frames.
148155
- `SuggestionFrame.group` (`string`, default `"default"`): the suggestion group name.
@@ -179,16 +186,15 @@ Key classes:
179186

180187
## ROI & Segmentation Masks
181188

182-
Spatial annotations stored alongside pose data in SLP format 1.5.
189+
Spatial annotations stored alongside pose data (SLP format 1.5+).
183190

184191
### `ROI`
185-
Region of interest with GeoJSON-like geometry.
192+
Region of interest with GeoJSON-like geometry. Supports `Polygon`, `Point`, `MultiPolygon`, `MultiPoint`, `LineString`, and `GeometryCollection` types.
186193

187194
```ts
188195
// Create from bounding box
189196
const roi = ROI.fromBbox(100, 200, 50, 80, {
190197
category: "arena",
191-
annotationType: AnnotationType.ARENA,
192198
video: labels.videos[0],
193199
});
194200

@@ -197,12 +203,24 @@ const roi = ROI.fromPolygon([[0,0], [100,0], [100,100], [0,100]], {
197203
category: "region",
198204
});
199205

206+
// Create from multi-polygon
207+
const roi = ROI.fromMultiPolygon([
208+
[[[0,0], [10,0], [10,10], [0,10], [0,0]]],
209+
[[[20,20], [30,20], [30,30], [20,30], [20,20]]],
210+
]);
211+
200212
// Properties
201213
roi.bounds; // { minX, minY, maxX, maxY }
202214
roi.area; // polygon area
203215
roi.centroid; // { x, y }
204216
roi.isBbox; // true if axis-aligned rectangle
205217

218+
// Explode multi-geometries into individual ROIs
219+
const parts = roi.explode(); // ROI[]
220+
221+
// Convert to GeoJSON Feature
222+
const feature = roi.toGeoJSON();
223+
206224
// Convert to mask
207225
const mask = roi.toMask(480, 640);
208226
```
@@ -225,21 +243,77 @@ mask.bbox; // { x, y, width, height }
225243
const roi = mask.toPolygon();
226244
```
227245

228-
### `Labels` ROI/Mask Access
246+
### `BoundingBox`
247+
Axis-aligned or rotated bounding box for detection/tracking workflows (SLP format 1.7).
248+
249+
```ts
250+
// Create user-annotated bbox
251+
const bbox = new UserBoundingBox({
252+
xCenter: 50, yCenter: 60, width: 100, height: 80,
253+
video: labels.videos[0], frameIdx: 3, category: "animal",
254+
});
255+
256+
// Create predicted bbox with confidence score
257+
const bbox = new PredictedBoundingBox({
258+
xCenter: 50, yCenter: 60, width: 100, height: 80,
259+
score: 0.95,
260+
});
261+
262+
// Factory methods
263+
const bbox = BoundingBox.fromXyxy(10, 20, 110, 100); // corner coords
264+
const bbox = BoundingBox.fromXywh(10, 20, 100, 80); // top-left + size
265+
266+
// Properties
267+
bbox.xyxy; // [x1, y1, x2, y2] (axis-aligned)
268+
bbox.xywh; // { x, y, width, height }
269+
bbox.corners; // number[][] (4 corner points, respects rotation)
270+
bbox.bounds; // { minX, minY, maxX, maxY }
271+
bbox.area; // width * height
272+
bbox.centroid; // { x, y }
273+
bbox.isPredicted; // true for PredictedBoundingBox
274+
bbox.isStatic; // true if no frameIdx
275+
bbox.isRotated; // true if angle != 0
276+
277+
// Conversion
278+
const roi = bbox.toRoi(); // Polygon ROI
279+
const mask = bbox.toMask(480, 640); // SegmentationMask
280+
```
281+
282+
### `Labels` ROI/Mask/BBox Access
229283

230284
```ts
231285
// Direct access
232-
labels.rois; // ROI[]
233-
labels.masks; // SegmentationMask[]
234-
labels.staticRois; // ROIs without frame index
286+
labels.rois; // ROI[]
287+
labels.masks; // SegmentationMask[]
288+
labels.bboxes; // BoundingBox[]
289+
labels.staticRois; // ROIs without frame index
235290
labels.temporalRois; // ROIs with frame index
291+
labels.staticBboxes; // BBoxes without frame index
292+
labels.temporalBboxes; // BBoxes with frame index
236293

237294
// Filtered queries
238295
labels.getRois({ video, frameIdx: 0, category: "arena" });
239-
labels.getMasks({ video, annotationType: AnnotationType.SEGMENTATION });
296+
labels.getMasks({ video, category: "segmentation" });
297+
labels.getBboxes({ video, frameIdx: 0, predicted: true });
240298
```
241299

242-
ROIs and masks are read/written in SLP format 1.5 files. The format ID is automatically set to 1.5 when ROIs or masks are present.
300+
The format ID is set automatically: 1.5 for ROIs/masks, 1.6 for ROIs with instance associations, 1.7 when bounding boxes are present.
301+
302+
## GeoJSON I/O
303+
304+
Convert ROIs to/from GeoJSON format.
305+
306+
```ts
307+
// Convert ROIs to GeoJSON FeatureCollection
308+
const geojson = roisToGeoJSON(labels.rois);
309+
310+
// Serialize to JSON string
311+
const json = writeGeoJSON(labels.rois);
312+
313+
// Parse GeoJSON back to ROIs
314+
const rois = roisFromGeoJSON(geojson);
315+
const rois = readGeoJSON(jsonString);
316+
```
243317

244318
## Skeleton Codecs
245319

docs/index.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,11 +24,11 @@ await saveSlp(labels, "/tmp/session-roundtrip.slp", { embed: false });
2424

2525
## Features
2626

27-
- SLP read/write with embedded frame support (format 1.0–1.5).
28-
- ROI and segmentation mask annotations (format 1.5).
27+
- SLP read/write with embedded frame support (format 1.0–1.7).
28+
- ROI and segmentation mask annotations (format 1.5), ROI-instance associations (format 1.6), bounding boxes (format 1.7).
2929
- Browser-compatible SLP writing via `saveSlpToBytes()`.
3030
- Streaming inputs (URL, `File`, `FileSystemFileHandle`, `Blob`).
31-
- Data model types (`Labels`, `LabeledFrame`, `Instance`, `Skeleton`, `Video`, `ROI`, `SegmentationMask`).
31+
- Data model types (`Labels`, `LabeledFrame`, `Instance`, `Skeleton`, `Video`, `ROI`, `SegmentationMask`, `BoundingBox`).
3232
- Video backends accept `string`, `File`, or `Blob` sources.
3333
- Browser-safe: Node.js-only code is fully isolated from the browser bundle.
3434
- Dictionary and numpy codecs.

docs/usage.md

Lines changed: 24 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -301,30 +301,46 @@ const trainLabels = set.get("train");
301301
await saveSlpSet(set);
302302
```
303303
304-
## ROI & Segmentation Masks
304+
## ROI, Segmentation Masks & Bounding Boxes
305305
306-
SLP format 1.5 supports spatial annotations (regions of interest and segmentation masks) alongside pose data:
306+
SLP format 1.5+ supports spatial annotations alongside pose data: ROIs (format 1.5), ROI-instance associations (format 1.6), and bounding boxes (format 1.7).
307307
308308
```ts
309-
import { loadSlp, ROI, SegmentationMask, AnnotationType } from "@talmolab/sleap-io.js";
309+
import {
310+
loadSlp, ROI, SegmentationMask,
311+
UserBoundingBox, PredictedBoundingBox, BoundingBox,
312+
writeGeoJSON, readGeoJSON,
313+
} from "@talmolab/sleap-io.js";
310314
311315
const labels = await loadSlp("dataset.slp");
312316
313-
// Access ROIs and masks
314-
console.log(`${labels.rois.length} ROIs, ${labels.masks.length} masks`);
317+
// Access ROIs, masks, and bounding boxes
318+
console.log(`${labels.rois.length} ROIs, ${labels.masks.length} masks, ${labels.bboxes.length} bboxes`);
315319
316320
// Query by video and frame
317321
const frameRois = labels.getRois({ video: labels.videos[0], frameIdx: 0 });
318322
const frameMasks = labels.getMasks({ video: labels.videos[0], frameIdx: 0 });
323+
const frameBboxes = labels.getBboxes({ video: labels.videos[0], frameIdx: 0 });
319324
320325
// Create new ROIs
321-
const bbox = ROI.fromBbox(100, 200, 50, 80, {
326+
const roi = ROI.fromBbox(100, 200, 50, 80, {
322327
category: "arena",
323328
video: labels.videos[0],
324329
});
325-
labels.rois.push(bbox);
330+
labels.rois.push(roi);
326331
327-
// Save — format 1.5 is used automatically when ROIs/masks are present
332+
// Create bounding boxes
333+
const bbox = new UserBoundingBox({
334+
xCenter: 150, yCenter: 240, width: 50, height: 80,
335+
category: "animal", video: labels.videos[0], frameIdx: 0,
336+
});
337+
labels.bboxes.push(bbox);
338+
339+
// Export ROIs to GeoJSON
340+
const geojsonStr = writeGeoJSON(labels.rois);
341+
const restoredRois = readGeoJSON(geojsonStr);
342+
343+
// Save — format version is set automatically based on content
328344
await saveSlp(labels, "output.slp");
329345
```
330346

src/codecs/dictionary.ts

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ export type LabelsDict = {
2626
frame_idx: number;
2727
video_idx: number;
2828
instances: Array<Record<string, unknown>>;
29+
is_negative?: boolean;
2930
}>;
3031
suggestions: Array<Record<string, unknown>>;
3132
provenance: Record<string, unknown>;
@@ -61,13 +62,14 @@ export function toDict(
6162
const labeledFrames: LabelsDict["labeled_frames"] = [];
6263
for (const frame of labels.labeledFrames) {
6364
if (videoFilter && !frame.video.matchesPath(videoFilter.video, true)) continue;
64-
if (options?.skipEmptyFrames && frame.instances.length === 0) continue;
65+
if (options?.skipEmptyFrames && frame.instances.length === 0 && !frame.isNegative) continue;
6566
const videoIdx = videos.indexOf(frame.video);
6667
if (videoIdx < 0) continue;
6768
labeledFrames.push({
6869
frame_idx: frame.frameIdx,
6970
video_idx: videoIdx,
7071
instances: frame.instances.map((instance) => instanceToDict(instance, labels, trackIndex)),
72+
...(frame.isNegative ? { is_negative: true } : {}),
7173
});
7274
}
7375

@@ -121,7 +123,7 @@ export function fromDict(data: LabelsDict): Labels {
121123
const labeledFrames = data.labeled_frames.map((frame) => {
122124
const video = videos[frame.video_idx];
123125
const instances = frame.instances.map((inst) => dictToInstance(inst, skeletons, tracks));
124-
return new LabeledFrame({ video, frameIdx: frame.frame_idx, instances });
126+
return new LabeledFrame({ video, frameIdx: frame.frame_idx, instances, isNegative: frame.is_negative ?? false });
125127
});
126128

127129
const suggestions = data.suggestions.map((suggestion) => {

0 commit comments

Comments
 (0)