Skip to content

Conversation

yiyujin
Copy link
Contributor

@yiyujin yiyujin commented Jun 26, 2025

Bring back object detection in ml5.js

To start, I added legacy object detection code) and p5js example code. Example code is from github reference doc

History

Make code compatible with ml5 1.0
  • load the model in preload() for basic example. (this will change later with p5.js 2.0 integration) -> 7deb4c8
  • new detectStart() and detectStop() functions (instead of calling detect() recursively over and over. -> ce82f51
  • Swap order of error and results argument for results callback. This way the error is optional and handling is not required in basic example. -> 4db442f

Make code scalable to support more models

  • refactor ObjectDetector Class (ObjectDetector wraps model -> ObjectDetector as model) -> 9375570

Next Steps

- p5 2.0 examples : Having this coco-ssd work with p5.js 2.0 is priority over supporting Transformers.js.
img
p5js 2.0 error (although it doesn't halt the detection)

@shiffman
Copy link
Member

Yay! Great job @yiyujin! As discussed, if it's helpful here's a quick list of things that we changed with ml5.js 1.0.

  • load the model in preload() for basic example. (this will change later with p5.js 2.0 integration)
  • new detectStart() and detectStop() functions (instead of calling detect() recursively over and over.
  • Swap order of error and results argument for results callback. This way the error is optional and handling is not required in basic example.

yiyujin added 2 commits July 1, 2025 22:28
…ct (instance.model), added mediaReady utils so that detection waits for video to be ready before detecting, updated tf.backend to webgl to avoid webgpu warning
@yiyujin
Copy link
Contributor Author

yiyujin commented Jul 10, 2025

Thank you @shiffman for the neat instructions!

  • load the model in preload() for basic example. (this will change later with p5.js 2.0 integration) -> 7deb4c8

  • updated ObjectDetector Class to return object (instance.model) so it can be loaded in preload()

  • added mediaReady utils so that detection waits for video to be ready before detecting.

  • added tf.backend(webgl) to avoid webgpu warning

  • new detectStart() and detectStop() functions (instead of calling detect() recursively over and over. -> ce82f51

  • Swap order of error and results argument for results callback. This way the error is optional and handling is not required in basic example. -> 4db442f

@yiyujin
Copy link
Contributor Author

yiyujin commented Jul 10, 2025

Based on today's meetup with @shiffman cc.@nasif-co

After all, final output for the project would be the p5js example code.

Example code (in order of priority) : bcd1788

  • object detection on image
  • object detection on webcam
  • object detection on video (would be interesting for curation purpose. ex. blazePose - was easy to demo with a shot video)

I guess next will be the docs, which I can draft out in parallel with the code !

@yiyujin
Copy link
Contributor Author

yiyujin commented Jul 11, 2025

I made the 3 changes to code so it works with ml5.js 1.0

💪 This week, I want to digest code better and draft a docs that explains the common pattern (functions and its roles) used across models : start, stop etc. I need it for myself anyways + could be a useful onboarding docs for future dev team.

Some notes on naming conventions :
Last meeting (w Dan, Nasif) I recall we briefly talked about naming conventions, how tensorflow.js uses a general word "predict" while ml5.js uses more model specific words like classify, detect, etc. I guess it's also different by context : user code vs. source code (such as results vs. detections / classifications..).
Anyhow, consistency matters so maybe a central docs can include some naming convention info too. Not much opinion on this so far, but just wanted to mention before I forget :-)

(speaking of which.. I think my branch should have been object-detector, not object-detection..? 😭)

  • Object Detection - Dev Notes (meant for onboarding)

@yiyujin
Copy link
Contributor Author

yiyujin commented Jul 11, 2025

After a brief code review from @pearmini 🙏 (wow I mindlessly copied the code to begin with then didn't think through!)

  • ObjectDetector is a redundant layer now because it just has a constructor.

  • Class is meant for methods or shared behavior across models.

  • refactor ObjectDetector Class (ObjectDetector wraps model -> ObjectDetector as model) -> 9375570

@pearmini can you take a glance at this please?

@yiyujin
Copy link
Contributor Author

yiyujin commented Jul 15, 2025

Just dropping notes for future tasks :

  • converting JSDoc to Typescript ?
  • object segmentation model

@yiyujin
Copy link
Contributor Author

yiyujin commented Jul 15, 2025

Copy link
Member

@shiffman shiffman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @yiyujin fantastic work! I left a few small comments in the basic example!

@yiyujin
Copy link
Contributor Author

yiyujin commented Jul 23, 2025

I updated the three example codes -> bcd1788

- webcam (examples/objectDetection-webcam/sketch.js)
- single image file (examples/objectDetection-sigle-image/sketch.js)
- video file (examples/objectDetection-video/sketch.js)

I'm esp. not sure if the video file example is at its best, since I'm adding a bunch of video eventListeners to check if video is loadedmetadata/play/pause/ended. I don't think p5 has a method on video to do that if I'm correct..

Copy link
Member

@shiffman shiffman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple last small comments on the video example and then some discoveries about video resizing! Great work @yiyujin!

@yiyujin yiyujin marked this pull request as ready for review July 29, 2025 20:45
@yiyujin
Copy link
Contributor Author

yiyujin commented Jul 30, 2025

  • rename test.mp4 to sth more meaningful (thank you @gohai )

946b084
Not too creative, changed it to ball_lifting.mp4

@yiyujin
Copy link
Contributor Author

yiyujin commented Jul 31, 2025

  • create p5 2.0 examples

https://github.com/ml5js/ml5-next-gen/tree/main/examples#p5js-20-examples

-> will be handled in another PR

Removed unused typedefs for options and predictions.
pearmini
pearmini previously approved these changes Oct 8, 2025
Copy link
Member

@pearmini pearmini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@pearmini pearmini merged commit 57a91db into main Oct 8, 2025
@yiyujin
Copy link
Contributor Author

yiyujin commented Oct 9, 2025

@pearmini It has npm install error. Can you look ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants