You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: projects/make-a-webcam-controlled-game-with-tensorflowjs/make-a-webcam-controlled-game-with-tensorflowjs.mdx
+19-13Lines changed: 19 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -27,11 +27,11 @@ In this project tutorial, we’ll build **Air Juggler**, a webcam gesture-contro
27
27
28
28
You will learn about the following concepts in this tutorial:
29
29
30
-
What TensorFlow.js is and how machine learning works in the browser.
31
-
How MediaPipe Hands detects hand landmarks using computer vision.
32
-
How to access your webcam using the `getUserMedia` API.
33
-
How to process video frames in real-time and extract hand positions.
34
-
How to use hand tracking data to create interactive experiences.
30
+
-What TensorFlow.js is and how machine learning works in the browser.
31
+
-How MediaPipe Hands detects hand landmarks using computer vision.
32
+
-How to access your webcam using the `getUserMedia` API.
33
+
-How to process video frames in real-time and extract hand positions.
34
+
-How to use hand tracking data to create interactive experiences.
35
35
36
36
By the end of this tutorial, we’ll have a fully functional gesture-controlled game that looks like this:
37
37
@@ -66,11 +66,17 @@ This approach has major problems:
66
66
- How do we handle different hand orientations? Should we measure the width looking down at the palm side of the hand or from the thumb side of the hand?
67
67
- How to account for hand size when depending on the age, gender, and other factors the hands may look completely different in dimensions?
68
68
69
-
To sum it up, the idea of programming with traditional computer programming is that the developers have some input that they want to process, they write the rules to process that input and the output of these rules is what they expect. In this case, they have an image(s) of a hand that they want to process, the rules determine what part of the hand is a finger, palm,etc. and as the output, they get the coordinates of these areas that they were looking for in the image.
69
+
To sum it up, the idea of programming with traditional computer programming is that the developers have some input that they want to process, they write the rules to process that input and the output of these rules is what they expect.
70
70
71
-
**Machine Learning** takes a completely different approach. Instead of writing rules, we train a model on thousands of images in different positions, lighting conditions, and angles. The model learns patterns that we engineers might not even consciously notice. Then, we send an image to this model and let it decide based on what it was trained upon to determine whether it was a hand or not! Computers are exceptional at spotting patterns and we’re trying to leverage that potential, by having them generate rules of what a hand looks like based on hundreds of thousands of images of hands and the coordinates of different areas on the hand.
71
+
In this case, they have an image(s) of a hand that they want to process, the rules determine what part of the hand is a finger, palm, etc. and as the output, they get the coordinates of these areas that they were looking for in the image.
72
72
73
-
So, in summary, the idea of programming with machine learning is that the developers have some input (an image of a hand), they provide the expected output (labeled coordinates for the hand/fingers) to a model, and the model automatically generates the internal rules/patterns necessary to transform the input into that output. In this case, the model is trained with images of hands and the expected coordinates of their landmarks, and as the output, it automatically predicts the coordinates of the landmarks in a new, unseen image.
73
+
**Machine Learning** takes a completely different approach. Instead of writing rules, we train a model on thousands of images in different positions, lighting conditions, and angles. The model learns patterns that we engineers might not even consciously notice.
74
+
75
+
Then, we send an image to this model and let it decide based on what it was trained upon to determine whether it was a hand or not! Computers are exceptional at spotting patterns and we’re trying to leverage that potential, by having them generate rules of what a hand looks like based on hundreds of thousands of images of hands and the coordinates of different areas on the hand.
76
+
77
+
So, in summary, the idea of programming with machine learning is that the developers have some input (an image of a hand), they provide the expected output (labeled coordinates for the hand/fingers) to a model, and the model automatically generates the internal rules/patterns necessary to transform the input into that output.
78
+
79
+
In this case, the model is trained with images of hands and the expected coordinates of their landmarks, and as the output, it automatically predicts the coordinates of the landmarks in a new, unseen image.
- Pass the `model` variable as an argument. This is the MediaPipeHands model seen on the previous code block.
307
313
- Add the `detectorConfig` as the second argument.
308
314
@@ -548,8 +554,8 @@ Let's recap what we learned:
548
554
549
555
Now that you understand hand tracking, you can build even more exciting projects such as:
550
556
551
-
- **Gesture Recognition** - Detect specific hand gestures (peace sign, thumbs up, etc.).
552
-
- **Sign Language Translator** - Recognize sign language alphabet.
557
+
- **Gesture Recognition** - Detect specific hand gestures (peace sign, thumbs up, etc.). You can use the same model as this project to build this!
558
+
- **Sign Language Translator** - Recognize sign language alphabet. After building the gesture recognition, assign it meanings in your code.
553
559
- Or build on this game by adding paddles using specific fingers such as index finger paddle, adding a high score system that saves the highest score of all the games, and more!
0 commit comments