You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: projects/make-a-webcam-controlled-game-with-tensorflowjs/make-a-webcam-controlled-game-with-tensorflowjs.mdx
+45-58Lines changed: 45 additions & 58 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -43,8 +43,10 @@ By the end of this tutorial, we’ll have a fully functional gesture-controlled
43
43
**Note:** The main focus of this tutorial is understanding how hand tracking works with TensorFlow.js, not building a complex game. The game mechanics are kept simple so we can focus on the machine learning concepts.
44
44
45
45
Let’s start!
46
+
46
47
## How Does Hand Tracking Work?
47
-
Before we jump into code, we need to understand what’s happening behind the scenes when we track hands using TensorFlow.js!
48
+
49
+
Before we jump into code, we need to understand what’s happening behind the scenes when we track hands using TensorFlow.js!
48
50
49
51
There are three main concepts I want to talk about: How Machine Learning is different from traditional computer programming, what TensorFlow.js and MediaPipe Hands libraries are, and how the detection process works.
50
52
@@ -77,10 +79,9 @@ So, in summary, the idea of programming with machine learning is that the develo
77
79
78
80
So, instead of the developers writing the rules, the machine writes the logic for us. Super cool, right?
79
81
80
-
81
82
### What is TensorFlow.js?
82
83
83
-
[TensorFlow.js](https://www.tensorflow.org/js) is a JavaScript library that lets you run machine learning models directly in the browser.
84
+
[TensorFlow.js](https://www.tensorflow.org/js) is a JavaScript library that lets you run machine learning models directly in the browser.
@@ -122,11 +124,11 @@ Here’s what happens every time a frame of the live video is processed (You can
122
124
Okay! Now that we have a general idea of what is happening behind the scenes, let’s get onto building it!
123
125
124
126
## Setup
127
+
125
128
Let’s begin with the starter code by downloading the project files from this repository: [Air Juggler using TensorFlow.js](https://github.com/Goku-kun/air-juggler-using-tensorflowjs)
126
129
127
130
The folder should have the following structure:
128
131
129
-
130
132
```
131
133
air-juggler-with-tensorflowjs/
132
134
├── starter/ # Start here - incomplete code with TODOs
@@ -221,15 +221,13 @@ if (document.readyState === "loading") {
221
221
}
222
222
```
223
223
224
-
225
224
**How this works:**
226
225
227
226
Once the HTML has loaded in the page, call the `checkTensorFlowLoaded` function
228
227
This function will keep running and continue showing the loading state until both the libraries load.
229
228
230
229
**Test it:** Open **starter/index.html** in your browser by double clicking on the file. You should see a loading spinner that disappears after 2-3 seconds!
231
230
232
-
233
231
## Step 3: Request Webcam Access
234
232
235
233
Now we'll access the user's webcam using the `getUserMedia` API.
@@ -238,11 +236,10 @@ Open **starter/handTracking.js** and find **Step 3a and Step 3b** in the `setupH
## Step 6: Transform Hand Landmarks to Coordinates
353
347
354
348
The model gives us 21 detailed keypoints, but our game only needs one position per hand—the palm center. Let's transform this complex data into simple (x, y) coordinates.
355
349
356
350
At **Step 6a**, process the detected landmarks:
357
351
358
-
359
352
```javascript
360
-
361
353
// Transform hand landmarks to canvas coordinates
362
-
consthandPositions=hands.map(hand=> {
363
-
// Get palm center (average of wrist and thumb 4 points)
- We use the 5 keypoints (wrist + base of index, middle, ring and pinky fingers) to find the approximate palm center. Look at the points: 0, 5, 9, 13 and 17 in the image above.
@@ -389,17 +380,18 @@ At **Step 6b**, call the function:
389
380
```javascript
390
381
// Call sendHandsCallback with new hand positions
391
382
if (sendHandsCallback) {
392
-
sendHandsCallback(handPositions);
383
+
sendHandsCallback(handPositions);
393
384
}
394
385
```
395
386
396
387
**Data Flow:**
397
-
1. It is important to understand how the whole process is working together, i.e. where the data originates, how its passed around, transforming it and getting us our hand information.
398
-
2. Video frame made available from video gets passed to the model
388
+
389
+
1. It is important to understand how the whole process is working together, i.e. where the data originates, how its passed around, transforming it and getting us our hand information.
390
+
2. Video frame made available from video gets passed to the model
399
391
3. Model generates and outputs 21 landmarks per hand
400
392
4. Landmarks are then used for the palm center calculation
401
393
5. Palm center calculation are then used to get the mirrored coordinates
402
-
6. These coordinates are passed to the sendHandsCallback which in turn updates the game state.
394
+
6. These coordinates are passed to the sendHandsCallback which in turn updates the game state.
403
395
404
396
**The callback pattern:** Every 30 FPS, we call the `sendHandsCallback` function with new hand positions, updating the game in real-time!
405
397
@@ -411,7 +403,6 @@ Open **starter/game.js** and find **Step 10** in the `startGame` function.
411
403
412
404
Add this code:
413
405
414
-
415
406
```javascript
416
407
// Initialize hand tracking if not already done
417
408
if (!window.handTrackingInitialized) {
@@ -427,7 +418,7 @@ if (!window.handTrackingInitialized) {
sendHandsCallback = sendHands // this function will be used to send updated hands position which will be received in the game.js file via the receiveHands function
@@ -510,11 +501,10 @@ if (!success) {
510
501
- If camera permission is denied, setup returns `false`.
511
502
- Shows friendly error message.
512
503
- Prevent the game from starting without hand tracking.
0 commit comments