preventing experiment crashes when using eyetracking #2490

ffigari · 2022-02-03T23:49:40Z

ffigari
Feb 3, 2022

I'm looking to run eye tracking over a couple of blocks (~3) of many repetitions each (~150) of short (~2.5 seconds) tasks. This results in needing to run the eye tracker continuously for around 20 minutes. But when using JSPsych and its WebGazer fork I'm getting crashes after ~10 minutes, some times even ~4 minutes.
At the end there is a minimum working code example with which I got to replicate this issue.

The culprit here is most probably WebGazer but there is already an opened issue there and it is not getting much attention.
Did someone run into similar issues? Were you able to solve it somehow? Or was it not possible?
Any tip on how to debug this? Or any debugging done already?

On Firefox I'm not getting any error output when the crash occurs, but on Chrome I got

Error: Unable to create WebGLTexture.
    at throwIfNull (webgazer.js:51610)
    at createTexture (webgazer.js:51524)
    at createAndConfigureTexture (webgazer.js:56732)
    at createFloat32MatrixTexture (webgazer.js:56745)
    at gpgpu_context_GPGPUContext.createFloat32MatrixTexture (webgazer.js:56955)
    at texture_manager_TextureManager.acquireTexture (webgazer.js:60224)
    at backend_webgl_MathBackendWebGL.acquireTexture (webgazer.js:62833)
    at backend_webgl_MathBackendWebGL.uploadToGPU (webgazer.js:62811)
    at backend_webgl_MathBackendWebGL.runWebGLProgram (webgazer.js:62662)
    at backend_webgl_MathBackendWebGL.compileAndRun (webgazer.js:62689)

and some other webgl and tensorflowjs related errors.
Crashes also seem to occur faster on Firefox (~180 seconds) than on Chromium (~400).

tensorflow js's face landmark model's current demo run smoothly (~18 fps) for >20 minutes on this same notebook I'm getting the crashes.
Webgazer uses @tensorflow-models/facemesh v0.0.3 which has been marked as deprecated on its npm page in favor of @tensorflow-models/facemesh.
I doubt this versions detail is strictly related to the crashing tabs. At the end of the day both packages share the same goal of providing the facemesh.

Any ideas?

The provided code mimics the real use case. At the beginning and every 50 trials of the same short task a "free calibration" step is provided in which at least one calibration points has to be given.

index.html:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">

    <script src="https://unpkg.com/[email protected]"></script>
    <link rel="stylesheet" href="https://unpkg.com/[email protected]/css/jspsych.css" />
    <script src="https://unpkg.com/@jspsych/[email protected]"></script>
    <script src="https://unpkg.com/@jspsych/[email protected]"></script>
    <script src="https://unpkg.com/@jspsych/[email protected]"></script>

    <script src="./vendor/webgazer.js"></script>

    <script src="./vendor/jspsych-psychophysics-3.1.0/jspsych-psychophysics.js"></script>
  </head>
  <script src="./index.js"></script>
</html>

index.js:

const jsPsych = initJsPsych({
  on_finish: function() {
    jsPsych.data.get().localSave('json','wg-duration-experiment.json');
  },
  extensions: [
    {type: jsPsychExtensionWebgazer}
  ],
})

const timeline = [{
  type: jsPsychWebgazerInitCamera,
}];

const taskExpectedDurationInSeg = (750 + 750 + 200 + 700) / 1000;
const desiredAmountOfTrials = 4 * 250;
const trialsPerCalibration = Math.max(Math.ceil(desiredAmountOfTrials / 20), 3);
const timeBetweenReportsInSegs = 20;
let startTs;
let lastReportTs;
for (let i = 0; i < desiredAmountOfTrials; i++) {
  if (i % trialsPerCalibration === 0) {
    let calibrated = false;
    const markAsCalibrated = () => {
      calibrated = true;
      document.removeEventListener('click', markAsCalibrated);
    };
    const calibratePoint = ({ clientX, clientY }) => {
      console.log(`mapping <${clientX}, ${clientY}> to gaze`);
      webgazer.recordScreenPosition(clientX, clientY, 'click');
    }
    timeline.push({
      async on_start() {
        if (i === 0) {
          startTs = new Date;
          lastReportTs = new Date;
          webgazer.showPredictionPoints(true);
          await webgazer.resume();
          console.log(`starting trials`);
        }
      },
      type: jsPsychHtmlKeyboardResponse,
      choices: [' '],
      stimulus: `
        <div>
          <h3>Free calibration</h3>
          <p>
            On the next screen you will be allowed to freely calibrate
            WebGazer. Each click you make will map your gaze to the coordinate
            of the click.
            <br>
            As many times as you want, stare at your cursor and perform a
            click. Do this in the regions of the screen you want WebGazer to
            estimate your gaze.
          </p>
          <p>
            You will be shown the gaze estimation while you add calibration
            points. Press <i>Space</i> to start. When you are satisfied press
            <i>Space</i> again to finish the calibration process.
          </p>
        </div>
      `,
    }, {
      timeline: [{
        on_start() {
          console.log('clearing WG data')
          webgazer.clearData();
          document.addEventListener('click', markAsCalibrated);
          document.addEventListener('click', calibratePoint);
        },
        type: jsPsychHtmlKeyboardResponse,
        choices: [' '],
        stimulus: '',
        on_finish() {
          document.removeEventListener('click', markAsCalibrated);
          document.removeEventListener('click', calibratePoint);
        }
      }],
      loop_function() {
        if (!calibrated) {
          console.error('no calibration points were passed')
        }
        return !calibrated;
      }
    });
  }
  preTime = 500 + Math.round(Math.random() * 500);
  fixTime = 500 + Math.round(Math.random() * 500);
  midTime = 150 + Math.round(Math.random() * 100);
  endTime = 700;

  const startX = (Math.random() < 0.5 ? 1 : -1) * Math.floor(window.innerWidth / 3)
  const color = 'blue';
  const radius = 20;
  timeline.push({
    type: jsPsychPsychophysics,
    stimuli: [{
      obj_type: 'text',
      origin_center: true,
      startX: 0, 
      startY: 30,
      show_start_time: 0,
      show_end_time: preTime + fixTime + midTime + endTime,
      content: `${i + 1} / ${desiredAmountOfTrials}`,
    }, {
      obj_type: 'cross',
      origin_center: true,
      startX: 0,
      startY: 0,
      show_start_time: preTime,
      show_end_time: preTime + fixTime,
      line_length: radius * 2,
    }, {
      obj_type: 'circle',
      origin_center: true,
      startX, 
      startY: 0,
      show_start_time:
      preTime + fixTime + midTime,
      show_end_time:
      preTime + fixTime + midTime + endTime,
      radius,
      line_color: color,
      fill_color: color,
    }],
    response_ends_trial: false,
    trial_duration: preTime + fixTime + midTime + endTime,
    extensions: [{ type: jsPsychExtensionWebgazer, params: { targets: [] } }],
    on_finish() {
      if ((new Date - lastReportTs) / 1000 > timeBetweenReportsInSegs) {
        console.log(
          `reached the end of ${
            i + 1
          }-th stimulus; ${
            (
              (new Date - startTs) / 1000
            ).toFixed(2)
          }s elapsed`
        );
        lastReportTs = new Date;
      }
    },
  });
}

console.log(`Task expected duration: ${taskExpectedDurationInSeg}s;
Amount of trials: ${desiredAmountOfTrials};
Experiment expected duration:
  ${((taskExpectedDurationInSeg * desiredAmountOfTrials ) / 60).toFixed(2)} minutes
  + time of one calibration every ${trialsPerCalibration} trials`);

jsPsych.run(timeline);

install script in case it's useful:

#!/bin/bash

set -euo pipefail

INSTALL_PATH=vendor
rm -rf $INSTALL_PATH
mkdir $INSTALL_PATH

# webgazer
rm -f ./webgazer.js
rm -rf ./webgazer
git clone -b master https://github.com/jspsych/WebGazer webgazer

(
  cd webgazer
  npm i
  npm run build
  cp dist/webgazer.js ../$INSTALL_PATH/
)

# Psychophysics
PSYCHOPHYSICS_VERSION=3.1.0
PSYCHOPHYSICS_ZIP_NAME=jspsych-psychophysics-$PSYCHOPHYSICS_VERSION.zip

wget \
  -O $INSTALL_PATH/$PSYCHOPHYSICS_ZIP_NAME \
  https://github.com/kurokida/jspsych-psychophysics/archive/refs/tags/v$PSYCHOPHYSICS_VERSION.zip
unzip $INSTALL_PATH/$PSYCHOPHYSICS_ZIP_NAME \
  -d $INSTALL_PATH
rm $INSTALL_PATH/$PSYCHOPHYSICS_ZIP_NAME

echo Done installing JSPsych-Psychophysics $PSYCHOPHYSICS_VERSION and JSPsych fork of WG

edit 14/02

We kept doing some experiments on this topic and turns out that the crash does not occur in all machines.
I actually got the following error on FF after ~7,5 minutes:

Uncaught (in promise) 
Exception { name: "NS_ERROR_OUT_OF_MEMORY", message: "", result: 2147942414, filename: "file:///home/francisco/eye-tracking/rastreador-ocular/www/vendor/webgazer.js", lineNumber: 86221, columnNumber: 0, data: null, stack: "webgazer</TFFaceMesh.prototype.getEyePatches@file:///home/francisco/eye-tracking/rastreador-ocular/www/vendor/webgazer.js:86221:52\nasync*getPupilFeatures@file:///home/francisco/eye-tracking/rastreador-ocular/www/vendor/webgazer.js:87960:23\neyePatchesEmissionLoop@file:///home/francisco/eye-tracking/rastreador-ocular/www/vendor/webgazer.js:87986:29\nFrameRequestCallback*eyePatchesEmissionLoop@file:///home/francisco/eye-tracking/rastreador-ocular/www/vendor/webgazer.js:87992:24

I also added a couple of prints before and after the call and the "after" console log would not get printed in the last iteration.

Besides, for our use case the re-calibration on mouse move was not needed. Because of that the ridge regression coefficients would remain the same over lot of iterations. I tried computing those coefficients explicitly when needed but got similar crash results.
This last thing discards some hypothesis we had about the CPU getting over used and this being the cause of the crash. The crash seems to happen after some constant amount of estimations. We tried adding a couple of "while True" loops in parallel (to force CPU usage) and counter-intuitively this would delay the crash in terms of time, the amount of iterations however would remain the same as before the change. It's worth noting all these experiments were very informal.

These things we tried would suggest that the issue is actually with the facemesh model. The one used by WG is a deprecated model so maybe updating it to the one that TF now offers helps? The strangest thing to me is how this seems to happen after a constant amount of estimations, but that seems to depend heavily on the architecture

edit 18/02

I ran https://github.com/jodeleeuw/219-eyetrack-example-full and it worked ok. I then increased the amount of iterations from 100 to 500 and it crashed on FF after ~12 minutes which seems consistent with the other reports.

GEJ1 · 2022-02-17T14:36:04Z

GEJ1
Feb 17, 2022

@jodeleeuw Do you have any idea about this? We are working together with @ffigari and we have no clue :(

0 replies

jodeleeuw · 2022-02-17T14:58:12Z

jodeleeuw
Feb 17, 2022
Maintainer

Is there a chance that this is based on the number of samples that are being stored in jsPsych's data structure?

FWIW, I've been able to run eye tracking experiments using the forked copy that have recorded data for similar, and sometimes longer, periods of time. But the last study I ran was using v6.3.1 and was about 10 months ago, so things may have changed since then.

My summer project, fwiw, is to dive into the eyetracking stuff more and try and get a better solution built for the kinds of tracking that people typically want to do in experiments. But I don't have enough time available to do any serious work on it before May.

5 replies

ffigari Feb 17, 2022
Author

We are running lot of short trials so maybe that's related. The thing is we also built a playground which does not use jspsych but does use the jspsych's fork and we got similar results.
And to make it harder to debug, it seems to depend on the used computer. While this experiment above crashed in 5 minutes in one computer, it ran fine for over 30 minutes in another one.
The current plan is to update the deprecated facemesh model to the current one tensorflow suggests to use. I'll keep updating as we get results.

jodeleeuw Feb 17, 2022
Maintainer

Any idea if this is limited to the WebGazer fork, or does the main repo have this issue too? I know you linked to the unresolved issue from their repo, but it's not clear to me whether this is an upstream problem or if it is a problem that I created.

ffigari Feb 17, 2022
Author

imo it's an upstream problem. I actually did a quick experiment using only the original version of webgazer and I also got crashes after a similar amount of time. But given the mentioned issue was not getting much attention and that our potential use cases are similar to JSPsych's ones we thought about asking here in the forum. It is possible someone went through the same path and encountered some kind of fix.

jeffhuang Mar 19, 2022

I tried to pin this down but it's a little tricky. If someone wants to test this with a more recent version of facemesh, please be my guest. The facemesh used is 0.0.3 which is a little outdated, and either moving to 0.0.5 or the new tensorflow 2.0 could be nice.

ffigari Mar 21, 2022
Author

@jeffhuang, we ended up updating the facemesh model to the new one from TFJS and it seems it to have solved the issue. At least in my machine and in @GEJ1 's machine the crashes are not occurring anymore. The way the eye's bounding boxes are computed changed a bit since there is no guarantee the keypoints indexes are the same. You can check it here ffigari/WebGazer@e5df9f9 . If it looks good I can open a PR to the upstream repo

jeffhuang · 2022-03-21T21:48:32Z

jeffhuang
Mar 21, 2022

Amazing! I will check this out very soon. And the code looks cleaner. Thank you!

0 replies

jeffhuang · 2022-03-25T05:47:47Z

jeffhuang
Mar 25, 2022

hi all, I think this should be resolved in the current version of webgazer now? Thanks so much @ffigari for your help identifying and also solving the issue

0 replies

Uh oh!

preventing experiment crashes when using eyetracking #2490

Uh oh!

Uh oh!

ffigari Feb 3, 2022

Replies: 4 comments · 5 replies

Uh oh!

GEJ1 Feb 17, 2022

Uh oh!

jodeleeuw Feb 17, 2022 Maintainer

Uh oh!

ffigari Feb 17, 2022 Author

Uh oh!

jodeleeuw Feb 17, 2022 Maintainer

Uh oh!

ffigari Feb 17, 2022 Author

Uh oh!

jeffhuang Mar 19, 2022

Uh oh!

Uh oh!

ffigari Mar 21, 2022 Author

Uh oh!

jeffhuang Mar 21, 2022

Uh oh!

jeffhuang Mar 25, 2022

ffigari
Feb 3, 2022

Replies: 4 comments 5 replies

GEJ1
Feb 17, 2022

jodeleeuw
Feb 17, 2022
Maintainer

ffigari Feb 17, 2022
Author

jodeleeuw Feb 17, 2022
Maintainer

ffigari Feb 17, 2022
Author

ffigari Mar 21, 2022
Author

jeffhuang
Mar 21, 2022

jeffhuang
Mar 25, 2022