-
Notifications
You must be signed in to change notification settings - Fork 40
Sequential NeuralNetwork #252
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
mop9047
wants to merge
47
commits into
main
Choose a base branch
from
timeseries-base
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 4 commits
Commits
Show all changes
47 commits
Select commit
Hold shift + click to select a range
0355d38
added_instantiation_files
mop9047 b8f031e
solved convergence issue with compile()
mop9047 bf67b69
instantiation code form TSDATA
mop9047 1228a59
added previous examples and refactored layers
mop9047 bedb655
added helper functions, simplified examples
mop9047 61c7d27
changes based on PR comments
mop9047 494ce16
convert linear/spatial to flag
mop9047 576d54c
Code Clean up and Simplified examples
mop9047 60d0eb4
Merged timeseries-as-task to timeseries-base and bug fixes
mop9047 1356185
merged timeseries as task to timeseries-base and bug fixes
mop9047 c5f78cf
Transferred files into NeuralNetwork Folder and code cleanup
mop9047 b9a1959
Fixed comment formatting
mop9047 a6afd6d
Changed Class names to Sequential
mop9047 174b1d8
changes based on PR comments
mop9047 3fc035a
chore: changed gestures to gesture
mop9047 e70ab0e
chore: replace Conv to withCNN
mop9047 8caffd3
Tweaks to neuralNetwork-sequence-weather-prediction example
gohai 03d3a18
Fix typo
gohai 790a162
Open-code sliding window in example
gohai f702700
Tweak variable placement and add comment
gohai 4c9594e
fix: sliding window bug
mop9047 e3d95f2
chore: slidingwindow to verb and rename samplewindow outputs
mop9047 002213a
chore: change padCoordinate name to setFixedLength
mop9047 b0c968c
fix: tslayers naming bug
mop9047 b5adac3
chore: clarify error when task is unknown
mop9047 c083beb
Rework the neuralNetwork-sequence-mouse-gesture-rdp example
gohai ec08ad9
Construct NN inside setup() in weather-prediction example
gohai e8f6958
Shave off two unneeded lines
gohai f580e64
fix: error capitalization from seqUtils
mop9047 0998b5b
chore: remove rdp visualizer
mop9047 0febf59
Rework the neuralNetwork-sequence-hand-gesture example
gohai 17a1cc2
Fix grammar in comment
gohai 21d3054
Use canvasDiv everywhere
gohai 155fde6
Use training/predicting in examples
gohai 9e53b97
Whitespace change
gohai d4a26a4
Flip for order
gohai 0e71849
Rework neuralNetwork-sequence-hand-gesture-load-model example
gohai 452ed2d
Tweaks to language
gohai 46924ba
Rename targetLength to sequenceLength and other tweaks
gohai 1308865
Revert part of last commit
gohai 024d91f
Rename "predicting" to "classifiying" in classification tasks
gohai 43b9198
feat: filter input/output by labels
mop9047 0d50690
chore: add new pre-trained models
mop9047 a7f8eea
chore: trainOptions, mm, barHeight
mop9047 8fd81c2
chore: friendlier error messages
mop9047 e84709f
fix: friendly error message normalize
mop9047 7776f28
chore: error for different lengths
mop9047 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
<!-- | ||
👋 Hello! This is an ml5.js example made and shared with ❤️. | ||
Learn more about the ml5.js project: https://ml5js.org/ | ||
ml5.js license and Code of Conduct: https://github.com/ml5js/ml5-next-gen/blob/main/LICENSE.md | ||
|
||
This example demonstrates training a Sign Language classifier through ml5.TimeSeries. | ||
--> | ||
|
||
<html> | ||
<head> | ||
<meta charset="UTF-8" /> | ||
<meta http-equiv="X-UA-Compatible" content="IE=edge" /> | ||
<meta name="viewport" content="width=device-width, initial-scale=1.0" /> | ||
<title>ml5.js Time Series Hand Gesture Train and Save</title> | ||
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/1.9.4/p5.min.js"></script> | ||
<script src="../../dist/ml5.js"></script> | ||
</head> | ||
|
||
<body> | ||
<script src="sketch.js"></script> | ||
</body> | ||
</html> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,192 @@ | ||
/* | ||
* 👋 Hello! This is an ml5.js example made and shared with ❤️. | ||
* Learn more about the ml5.js project: https://ml5js.org/ | ||
* ml5.js license and Code of Conduct: https://github.com/ml5js/ml5-next-gen/blob/main/LICENSE.md | ||
* | ||
* This example demonstrates training a Hand Gesture classifier through ml5.TimeSeries. | ||
*/ | ||
|
||
let seqLength = 50; | ||
|
||
let handPose; | ||
let video; | ||
|
||
let hands = []; | ||
let sequence = []; | ||
|
||
let recordingFinished = false; | ||
let predictedWord = ""; | ||
|
||
// UI variables | ||
let trainingWords = {}; | ||
|
||
function preload() { | ||
// Load the handPose model | ||
handPose = ml5.handPose(); | ||
|
||
// setup the timeseries neural network | ||
let options = { | ||
outputs: ["label"], | ||
task: "classification", | ||
dataMode: "spatial", | ||
debug: "true", | ||
learningRate: 0.001, | ||
mop9047 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
}; | ||
model = ml5.timeSeries(options); | ||
} | ||
|
||
function setup() { | ||
createCanvas(640, 480); | ||
|
||
// setup video capture | ||
video = createCapture(VIDEO); | ||
video.size(640, 480); | ||
video.hide(); | ||
|
||
// place UI elements | ||
UI(); | ||
|
||
// use handpose model on video | ||
handPose.detectStart(video, gotHands); | ||
} | ||
|
||
function draw() { | ||
// draw video on frame | ||
image(video, 0, 0, width, height); | ||
|
||
drawPredictedWord(); | ||
|
||
// if hands are found then start recording | ||
if (hands.length > 0 && recordingFinished == false) { | ||
if (sequence.length <= seqLength) { | ||
// get coordinates from hands (21 points) | ||
handpoints = drawPoints(); | ||
sequence.push(handpoints); | ||
|
||
// once sequence reaches the seqLength, add sequence as just one X value | ||
} else if (sequence.length > 0) { | ||
// get the training word from the input box | ||
let train_word = nameField.value(); | ||
mop9047 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
// if there is a word currently in the box then add data with that label | ||
if (train_word.length > 0) { | ||
// add data to the model | ||
let target = { label: train_word }; | ||
model.addData(sequence, target); | ||
trainingWordsUpdate(); | ||
|
||
// if there is no word in the box then classify instead | ||
} else { | ||
// classify the data | ||
model.classify(sequence, gotResults); | ||
} | ||
|
||
// reset the sequence | ||
sequence = []; | ||
recordingFinished = true; | ||
} | ||
|
||
// can only record again when hand is out of frame | ||
} else { | ||
if (hands.length == 0) { | ||
recordingFinished = false; | ||
} | ||
} | ||
} | ||
|
||
function drawPoints() { | ||
let handpoints = []; | ||
// iterate through both hands | ||
for (let i = 0; i < hands.length; i++) { | ||
let hand = hands[i]; | ||
for (let j = 0; j < hand.keypoints.length; j++) { | ||
// access the keypoints in the hand | ||
let keypoint = hand.keypoints[j]; | ||
handpoints.push(keypoint.x, keypoint.y); | ||
|
||
fill(0, 255, 0); | ||
noStroke(); | ||
circle(keypoint.x, keypoint.y, 5); | ||
} | ||
} | ||
// assign to a different variable before clearing | ||
let output = handpoints; | ||
handpoints = []; | ||
|
||
return output; | ||
} | ||
|
||
// Callback function for when handPose outputs data | ||
function gotHands(results) { | ||
// save the output to the hands variable | ||
hands = results; | ||
} | ||
|
||
function trainModelAndSave() { | ||
model.normalizeData(); | ||
let options = { | ||
epochs: 100, | ||
}; | ||
model.train(options, whileTraining, finishedTraining); | ||
nameField.value(""); | ||
} | ||
|
||
function whileTraining(epoch) { | ||
console.log(epoch); | ||
} | ||
|
||
function finishedTraining() { | ||
console.log("finished training."); | ||
model.save("model"); | ||
} | ||
|
||
function gotResults(results) { | ||
predictedWord = results[0].label; | ||
console.log(predictedWord); | ||
text(predictedWord, 200, 200); | ||
} | ||
|
||
function UI() { | ||
nameField = createInput(""); | ||
nameField.attribute("placeholder", "Type the word to train"); | ||
nameField.position(110, 500); | ||
nameField.size(250); | ||
|
||
instructionP = createP( | ||
'I want to train: <br><br> 1.) Type any word you want to pair with a gesture, e.g. "HELLO" <br> 2.) Do the gesture associated to the word, make sure to do it until the points disappear. <br> 3.) Move your hand out of the frame and repeat the gesture, do this multiple times <br> 4.) Do the same for other words e.g. "BYE" <br> 5.) Once all data is collected, press Train and Save<br><br> Tip: have at least 5 datasets for each word' | ||
); | ||
instructionP.style("width", "640px"); | ||
dataCountsP = createP("-> After the gesture a tally will appear here <-"); | ||
|
||
train_but = createButton("Train and Save"); | ||
train_but.mouseClicked(trainModelAndSave); | ||
train_but.style("font-family", "Georgia"); | ||
train_but.style("font-size", "20px"); | ||
train_but.position(500, 490); | ||
} | ||
|
||
function drawPredictedWord() { | ||
textSize(100); | ||
fill(255); | ||
text(predictedWord, 100, height / 2); | ||
} | ||
|
||
function trainingWordsUpdate() { | ||
let tempWord = nameField.value(); | ||
console.log(Object.keys(trainingWords)); | ||
if (!(tempWord in trainingWords)) { | ||
trainingWords[tempWord] = 1; | ||
} else { | ||
trainingWords[tempWord]++; | ||
} | ||
|
||
let counts = ""; | ||
let keys = Object.keys(trainingWords); | ||
console.log("keys", keys); | ||
gohai marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
for (let k of keys) { | ||
counts += k + " : " + trainingWords[k] + "<br>"; | ||
} | ||
|
||
dataCountsP.html(counts); | ||
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
<!-- | ||
👋 Hello! This is an ml5.js example made and shared with ❤️. | ||
Learn more about the ml5.js project: https://ml5js.org/ | ||
ml5.js license and Code of Conduct: https://github.com/ml5js/ml5-next-gen/blob/main/LICENSE.md | ||
|
||
This example demonstrates loading a Sign Language classifier through ml5.TimeSeries. | ||
--> | ||
|
||
<html> | ||
<head> | ||
<meta charset="UTF-8" /> | ||
<meta http-equiv="X-UA-Compatible" content="IE=edge" /> | ||
<meta name="viewport" content="width=device-width, initial-scale=1.0" /> | ||
<title>ml5.js Time Series Hand Gesture load model</title> | ||
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/1.9.4/p5.min.js"></script> | ||
<script src="../../dist/ml5.js"></script> | ||
</head> | ||
|
||
<body> | ||
<script src="sketch.js"></script> | ||
<div id="canvasDiv"></div> | ||
<p> | ||
This example loads a model that is trained with ASL hand gestures for | ||
Hello and Goodbye. <br /> | ||
<br /> | ||
|
||
Instructions: <br /> | ||
1.) Use one hand to do a gesture in front of the camera <br /> | ||
2.) Wait for the points to disappear or the prediction appears on | ||
screen<br /> | ||
3.) To predict again, remove your hands in the frame and do the gesture | ||
again<br /><br /> | ||
|
||
How to do gestures for Hello and Goodbye in ASL: <br /> | ||
Hello: | ||
<a href="https://babysignlanguage.com/dictionary/hello/" | ||
>https://babysignlanguage.com/dictionary/hello/ </a | ||
><br /> | ||
Goodbye: | ||
<a href="https://babysignlanguage.com/dictionary/goodbye/" | ||
>https://babysignlanguage.com/dictionary/goodbye/ </a | ||
><br /> | ||
</p> | ||
</body> | ||
</html> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
{"modelTopology":{"class_name":"Sequential","config":{"name":"sequential_1","layers":[{"class_name":"Conv1D","config":{"filters":8,"kernel_initializer":{"class_name":"VarianceScaling","config":{"scale":1,"mode":"fan_avg","distribution":"normal","seed":null}},"kernel_regularizer":null,"kernel_constraint":null,"kernel_size":[3],"strides":[1],"padding":"valid","dilation_rate":[1],"activation":"relu","use_bias":true,"bias_initializer":{"class_name":"Zeros","config":{}},"bias_regularizer":null,"activity_regularizer":null,"bias_constraint":null,"name":"conv1d_Conv1D1","trainable":true,"batch_input_shape":[null,51,42],"dtype":"float32"}},{"class_name":"MaxPooling1D","config":{"pool_size":[2],"padding":"valid","strides":[2],"name":"max_pooling1d_MaxPooling1D1","trainable":true}},{"class_name":"Conv1D","config":{"filters":16,"kernel_initializer":{"class_name":"VarianceScaling","config":{"scale":1,"mode":"fan_avg","distribution":"normal","seed":null}},"kernel_regularizer":null,"kernel_constraint":null,"kernel_size":[3],"strides":[1],"padding":"valid","dilation_rate":[1],"activation":"relu","use_bias":true,"bias_initializer":{"class_name":"Zeros","config":{}},"bias_regularizer":null,"activity_regularizer":null,"bias_constraint":null,"name":"conv1d_Conv1D2","trainable":true,"batch_input_shape":[null,51,42],"dtype":"float32"}},{"class_name":"MaxPooling1D","config":{"pool_size":[2],"padding":"valid","strides":[2],"name":"max_pooling1d_MaxPooling1D2","trainable":true}},{"class_name":"Flatten","config":{"name":"flatten_Flatten1","trainable":true}},{"class_name":"Dense","config":{"units":16,"activation":"relu","use_bias":true,"kernel_initializer":{"class_name":"VarianceScaling","config":{"scale":1,"mode":"fan_avg","distribution":"normal","seed":null}},"bias_initializer":{"class_name":"Zeros","config":{}},"kernel_regularizer":null,"bias_regularizer":null,"activity_regularizer":null,"kernel_constraint":null,"bias_constraint":null,"name":"dense_Dense1","trainable":true}},{"class_name":"Dense","config":{"units":2,"activation":"softmax","use_bias":true,"kernel_initializer":{"class_name":"VarianceScaling","config":{"scale":1,"mode":"fan_avg","distribution":"normal","seed":null}},"bias_initializer":{"class_name":"Zeros","config":{}},"kernel_regularizer":null,"bias_regularizer":null,"activity_regularizer":null,"kernel_constraint":null,"bias_constraint":null,"name":"dense_Dense2","trainable":true}}]},"keras_version":"tfjs-layers 4.8.0","backend":"tensor_flow.js"},"weightsManifest":[{"paths":["./hello.weights.bin"],"weights":[{"name":"conv1d_Conv1D1/kernel","shape":[3,42,8],"dtype":"float32"},{"name":"conv1d_Conv1D1/bias","shape":[8],"dtype":"float32"},{"name":"conv1d_Conv1D2/kernel","shape":[3,8,16],"dtype":"float32"},{"name":"conv1d_Conv1D2/bias","shape":[16],"dtype":"float32"},{"name":"dense_Dense1/kernel","shape":[176,16],"dtype":"float32"},{"name":"dense_Dense1/bias","shape":[16],"dtype":"float32"},{"name":"dense_Dense2/kernel","shape":[16,2],"dtype":"float32"},{"name":"dense_Dense2/bias","shape":[2],"dtype":"float32"}]}]} | ||
mop9047 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
Binary file not shown.
1 change: 1 addition & 0 deletions
1
examples/timeSeries-load-model-hand-gestures/model/model_meta.json
mop9047 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
{"inputUnits":[42],"outputUnits":2,"inputs":{"label_0":{"dtype":"number","min":4.151249399907168,"max":586.4725394909854},"label_1":{"dtype":"number","min":186.47223882383636,"max":496.34918695509003},"label_2":{"dtype":"number","min":12.818880217505907,"max":564.7860747522525},"label_3":{"dtype":"number","min":160.9460986889124,"max":478.89482602620234},"label_4":{"dtype":"number","min":20.681431005110262,"max":557.1173870582799},"label_5":{"dtype":"number","min":135.1274696802808,"max":454.0862355189599},"label_6":{"dtype":"number","min":29.375938053231934,"max":562.4826339023859},"label_7":{"dtype":"number","min":113.22511415628927,"max":455.15365538508894},"label_8":{"dtype":"number","min":37.27265551578051,"max":573.3838980891996},"label_9":{"dtype":"number","min":98.00531862273047,"max":473.4382341601794},"label_10":{"dtype":"number","min":2.706973037101564,"max":599.2858408346702},"label_11":{"dtype":"number","min":117.7350326456234,"max":453.76022921684716},"label_12":{"dtype":"number","min":11.635752695869659,"max":612.8243751678727},"label_13":{"dtype":"number","min":91.05094143918305,"max":481.6467136241304},"label_14":{"dtype":"number","min":22.9353041163117,"max":621.0127886598051},"label_15":{"dtype":"number","min":61.619264849841635,"max":499.63536096409143},"label_16":{"dtype":"number","min":33.53953084457643,"max":626.4181148091915},"label_17":{"dtype":"number","min":28.455718477478662,"max":512.7953875856006},"label_18":{"dtype":"number","min":-2.8065139589559984,"max":617.7828981986556},"label_19":{"dtype":"number","min":117.6886729722432,"max":459.5357193516273},"label_20":{"dtype":"number","min":3.7782929928570064,"max":633.7038985044576},"label_21":{"dtype":"number","min":86.77279076496669,"max":486.0751342925063},"label_22":{"dtype":"number","min":16.177018651157255,"max":642.8366376068107},"label_23":{"dtype":"number","min":51.687144639081325,"max":502.64037741142846},"label_24":{"dtype":"number","min":28.1461509145229,"max":650.2419536370577},"label_25":{"dtype":"number","min":15.922382743702723,"max":516.9301399988833},"label_26":{"dtype":"number","min":-6.382516546058305,"max":630.7077663350849},"label_27":{"dtype":"number","min":120.16376158664924,"max":461.0881814514869},"label_28":{"dtype":"number","min":-1.4074379536407533,"max":647.5041251714117},"label_29":{"dtype":"number","min":90.58035685591811,"max":485.04491883378125},"label_30":{"dtype":"number","min":10.174906800459325,"max":658.4893875478738},"label_31":{"dtype":"number","min":71.76407331703523,"max":500.55112323964187},"label_32":{"dtype":"number","min":21.11718120932074,"max":668.566957655395},"label_33":{"dtype":"number","min":39.557348432978586,"max":514.4287318106208},"label_34":{"dtype":"number","min":-7.9534800405596595,"max":641.3232619371444},"label_35":{"dtype":"number","min":126.31599791044414,"max":465.6320514399833},"label_36":{"dtype":"number","min":-3.8369034650104927,"max":658.2044139172733},"label_37":{"dtype":"number","min":103.73604938021917,"max":481.03793223993495},"label_38":{"dtype":"number","min":3.7075645592075435,"max":668.8017566330357},"label_39":{"dtype":"number","min":88.76136006394765,"max":494.63688258092407},"label_40":{"dtype":"number","min":6.9609311353376135,"max":676.9525074586147},"label_41":{"dtype":"number","min":75.97401514052241,"max":506.7948506427954}},"outputs":{"label":{"dtype":"string","min":0,"max":1,"uniqueValues":["hello","bye"],"legend":{"hello":[1,0],"bye":[0,1]}}},"isNormalized":true,"seriesShape":[51,42]} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,131 @@ | ||
/* | ||
* 👋 Hello! This is an ml5.js example made and shared with ❤️. | ||
* Learn more about the ml5.js project: https://ml5js.org/ | ||
* ml5.js license and Code of Conduct: https://github.com/ml5js/ml5-next-gen/blob/main/LICENSE.md | ||
* | ||
* This example demonstrates loading a Hand Gesture classifier through ml5.TimeSeries. | ||
* This example is trained with the ASL gestures for Hello and Goodbye | ||
* | ||
* Reference to sign hello and goodbye in ASL: | ||
* Hello: https://babysignlanguage.com/dictionary/hello/ | ||
* Goodbye: https://babysignlanguage.com/dictionary/goodbye/ | ||
*/ | ||
|
||
// change this to make the recording longer | ||
let seqLength = 50; | ||
|
||
let handPose; | ||
let video; | ||
let hands = []; | ||
let sequence = []; | ||
let recordingFinished = false; | ||
let predictedWord = ""; | ||
|
||
function preload() { | ||
// Load the handPose model | ||
handPose = ml5.handPose(); | ||
|
||
// setup the timeseries neural network | ||
let options = { | ||
task: "classification", | ||
dataMode: "spatial", | ||
}; | ||
|
||
model = ml5.timeSeries(options); | ||
} | ||
|
||
function setup() { | ||
let canvas = createCanvas(640, 480); | ||
canvas.parent("canvasDiv"); | ||
|
||
// create video capture | ||
video = createCapture(VIDEO); | ||
video.size(640, 480); | ||
video.hide(); | ||
|
||
handPose.detectStart(video, gotHands); | ||
|
||
// setup the model files to load | ||
let modelDetails = { | ||
model: "model/model.json", | ||
metadata: "model/model_meta.json", | ||
weights: "model/model.weights.bin", | ||
}; | ||
|
||
// load the model and call modelLoaded once finished | ||
model.load(modelDetails, modelLoaded); | ||
} | ||
// call back for load model | ||
function modelLoaded() { | ||
console.log("model loaded!"); | ||
} | ||
|
||
function draw() { | ||
// draw video on the canvas | ||
image(video, 0, 0, width, height); | ||
|
||
// put the text on screen after a prediction | ||
placePredictedText(); | ||
|
||
// if hands are found then start recording | ||
if (hands.length > 0 && recordingFinished == false) { | ||
if (sequence.length <= seqLength) { | ||
// get coordinates from hands (21 points) | ||
handpoints = drawPoints(); | ||
sequence.push(handpoints); | ||
|
||
// once sequence reaches the seqLength, add sequence as just one X value | ||
} else if (sequence.length > 0) { | ||
// classify based on the collected data | ||
model.classify(sequence, gotResults); | ||
|
||
// reset the sequence | ||
sequence = []; | ||
recordingFinished = true; | ||
} | ||
|
||
// can only record again when hand is out of frame | ||
} else { | ||
if (hands.length == 0) { | ||
recordingFinished = false; | ||
} | ||
} | ||
} | ||
|
||
// draw the points on the hands | ||
function drawPoints() { | ||
let handpoints = []; | ||
for (let i = 0; i < hands.length; i++) { | ||
let hand = hands[i]; | ||
for (let j = 0; j < hand.keypoints.length; j++) { | ||
let keypoint = hand.keypoints[j]; | ||
fill(0, 255, 0); | ||
noStroke(); | ||
circle(keypoint.x, keypoint.y, 5); | ||
handpoints.push(keypoint.x, keypoint.y); | ||
} | ||
} | ||
let output = handpoints; | ||
handpoints = []; | ||
return output; | ||
} | ||
|
||
// Callback function for when handPose outputs data | ||
function gotHands(results) { | ||
// save the output to the hands variable | ||
hands = results; | ||
} | ||
|
||
// call back for accessing the results | ||
function gotResults(results) { | ||
predictedWord = results[0].label; | ||
console.log(predictedWord); | ||
text(predictedWord, 100, 100); | ||
} | ||
|
||
// for drawing text on screen | ||
function placePredictedText() { | ||
textSize(100); | ||
fill(255); | ||
text(predictedWord, 100, height / 2); | ||
} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.