Webgazer extension params.targets - css selector struggles

[myrtevos]

Hi all,

I've got a webgazer experiment that an actual programmer built last year by writing and putting together custom jsPsych plugins. This works great, and/but I'm keen to see how the "official" jsPsych plugins compare, so I've been trying to create the same experiment using the new jsPsych webgazer extension. So far so good, but the extension doesn't seem to 'find' my CSS selectors.

Here is the relevant section of the timeline:

                    type: 'audio-betty-response',
                    stimulus: jsPsych.timelineVariable('audio'),
                    trial_ends_after_audio: false,
                    css_classes: ['trial-content'],
                    choices: [jsPsych.timelineVariable('left'), jsPsych.timelineVariable('right')],
                    button_html: [
                        '<img src="%choice%" class="img-fluid selection-enabled"/>',
                        '<img src="%choice%" class="img-fluid selection-enabled"/>'
                    ],
                    margin_vertical: '0px',
                    margin_horizontal: '0px',
                    response_allowed_while_playing: false,
                    extensions: [
                        {type: 'webgazer', params: {targets: ['#button-left', '#button-right']}}
                    ],

(Yes, this is a modified audio-button-response plugin named "Betty", it's a long story)

And here is the relevant section of the plugin:

        //button html modified to display exactly 2 picture-buttons, so not robust to a different # of choices!
        var left_img = buttons[0].replace(/%choice%/g, trial.choices[0]);
        var right_img = buttons[1].replace(/%choice%/g, trial.choices[1]);
        var html = '<div id="jspsych-audio-betty-response-btngroup" class="container-fluid" style="height: 100%">' +
                    '<div class="row" style="height: 100%">' +
                        '<div class="col-md-5 align-self-center">' +
                            '<div class="jspsych-audio-betty-response-button" style="display: inline-block; margin:' + trial.margin_vertical + ' ' + trial.margin_horizontal + '" id="button-left" data-choice="left">' + left_img + '</div>' +
                        '</div>' +
                        '<div class="col-md-2"></div>' +
                        '<div class="col-md-5 align-self-center">' +
                            '<div class="jspsych-audio-betty-response-button" style="display: inline-block; margin:' + trial.margin_vertical + ' ' + trial.margin_horizontal + '" id="button-right" data-choice="right">' + right_img + '</div>' +
                        '</div>' +
                    '</div>' +
                '</div>';

There's a few CSS classes from bootstrap in there that I think aren't relevant to the problem. Just in case, this is what the CSS class .trial-content looks like:

    .trial-content{    
        max-width: 100vw;
        width: 100vw;                
        height: 100vh;
        border-left: solid #cccccc 0.5vw;
        border-right: solid #cccccc 0.5vw;
        border-top: solid #cccccc 4vh;
        border-bottom: solid #cccccc 4vh;
        background-color: #cccccc;
    }

Anyone with slightly better (read: any) CSS intuitions who can see where I've gone wrong?
Many thanks in advance!

[jodeleeuw]

Hi @myrtevos,

One possibility is that the buttons aren't yet rendered on the DOM when the webgazer extension is trying to find them. This is a possibility for custom plugins with asynchronous or event-based displays. I'm trying to think about how to create a general purpose solution, but the current code is certainly not general.

Can you post the full plugin code?

[myrtevos]

Wow, that was fast! Thanks! :D

This is the full plugin code (vast majority of changes in the section I already posted):

/**
 * jspsych-audio-betty-response
 * A minimally modified version of audio-button_response (Kristin Diep)
 *
 * plugin for playing an audio file and getting a button response, in the Betty experiment
 *
 * documentation: docs.jspsych.org
 *
 **/

jsPsych.plugins["audio-betty-response"] = (function () {
    var plugin = {};
  
    jsPsych.pluginAPI.registerPreload('audio-betty-response', 'stimulus', 'audio');
  
    plugin.info = {
      name: 'audio-betty-response',
      description: '',
      parameters: {
        stimulus: {
          type: jsPsych.plugins.parameterType.AUDIO,
          pretty_name: 'Stimulus',
          default: undefined,
          description: 'The audio to be played.'
        },
        choices: {
          type: jsPsych.plugins.parameterType.STRING,
          pretty_name: 'Choices',
          default: undefined,
          array: true,
          description: 'The button labels.'
        },
        button_html: {
          type: jsPsych.plugins.parameterType.HTML_STRING,
          pretty_name: 'Button HTML',
          default: '<button class="jspsych-btn">%choice%</button>',
          array: true,
          description: 'Custom button. Can make your own style.'
        },
        prompt: {
          type: jsPsych.plugins.parameterType.STRING,
          pretty_name: 'Prompt',
          default: null,
          description: 'Any content here will be displayed below the stimulus.'
        },
        trial_duration: {
          type: jsPsych.plugins.parameterType.INT,
          pretty_name: 'Trial duration',
          default: null,
          description: 'The maximum duration to wait for a response.'
        },
        margin_vertical: {
          type: jsPsych.plugins.parameterType.STRING,
          pretty_name: 'Margin vertical',
          default: '0px',
          description: 'Vertical margin of button.'
        },
        margin_horizontal: {
          type: jsPsych.plugins.parameterType.STRING,
          pretty_name: 'Margin horizontal',
          default: '8px',
          description: 'Horizontal margin of button.'
        },
        response_ends_trial: {
          type: jsPsych.plugins.parameterType.BOOL,
          pretty_name: 'Response ends trial',
          default: true,
          description: 'If true, the trial will end when user makes a response.'
        },
        trial_ends_after_audio: {
          type: jsPsych.plugins.parameterType.BOOL,
          pretty_name: 'Trial ends after audio',
          default: false,
          description: 'If true, then the trial will end as soon as the audio file finishes playing.'
        },
        response_allowed_while_playing: {
          type: jsPsych.plugins.parameterType.BOOL,
          pretty_name: 'Response allowed while playing',
          default: true,
          description: 'If true, then responses are allowed while the audio is playing. ' +
            'If false, then the audio must finish playing before a response is accepted.'
        }
      }
    }
  
    plugin.trial = function (display_element, trial) {
  
      // setup stimulus
      var context = jsPsych.pluginAPI.audioContext();
      var audio;
  
      // store response
      var response = {
        rt: null,
        button: null
      };
  
      // record webaudio context start time
      var startTime;
  
      // load audio file
      jsPsych.pluginAPI.getAudioBuffer(trial.stimulus)
        .then(function (buffer) {
          if (context !== null) {
            audio = context.createBufferSource();
            audio.buffer = buffer;
            audio.connect(context.destination);
          } else {
            audio = buffer;
            audio.currentTime = 0;
          }
          setupTrial();
        })
        .catch(function (err) {
          console.error(`Failed to load audio file "${trial.stimulus}". Try checking the file path. We recommend using the preload plugin to load audio files.`)
          console.error(err)
        });
  
      function setupTrial() {
        // set up end event if trial needs it
        if (trial.trial_ends_after_audio) {
          audio.addEventListener('ended', end_trial);
        }
  
        // enable buttons after audio ends if necessary
        if ((!trial.response_allowed_while_playing) & (!trial.trial_ends_after_audio)) {
          audio.addEventListener('ended', enable_buttons);
        }
  
        //display buttons
        var buttons = [];
        if (Array.isArray(trial.button_html)) {
          if (trial.button_html.length == trial.choices.length) {
            buttons = trial.button_html;
          } else {
            console.error('Error in audio-betty-response plugin. The length of the button_html array does not equal the length of the choices array');
          }
        } else {
          for (var i = 0; i < trial.choices.length; i++) {
            buttons.push(trial.button_html);
          }
        }
  
        //button html modified to display exactly picture-buttons, so not robust to a different # of choices!
        var left_img = buttons[0].replace(/%choice%/g, trial.choices[0]);
        var right_img = buttons[1].replace(/%choice%/g, trial.choices[1]);
        var html = '<div id="jspsych-audio-betty-response-btngroup" class="container-fluid" style="height: 100%">' +
                    '<div class="row" style="height: 100%">' +
                        '<div class="col-md-5 align-self-center">' +
                            '<div class="jspsych-audio-betty-response-button" style="display: inline-block; margin:' + trial.margin_vertical + ' ' + trial.margin_horizontal + '" id="button-left" data-choice="left">' + left_img + '</div>' +
                        '</div>' +
                        '<div class="col-md-2"></div>' +
                        '<div class="col-md-5 align-self-center">' +
                            '<div class="jspsych-audio-betty-response-button" style="display: inline-block; margin:' + trial.margin_vertical + ' ' + trial.margin_horizontal + '" id="button-right" data-choice="right">' + right_img + '</div>' +
                        '</div>' +
                    '</div>' +
                '</div>';     
  
        //show prompt if there is one
        if (trial.prompt !== null) {
          html += trial.prompt;
        }
  
        display_element.innerHTML = html;
  
        if (trial.response_allowed_while_playing) {
          enable_buttons();
        } else {
          disable_buttons();
        }
  
        // start time
        startTime = performance.now();
  
        // start audio
        if (context !== null) {
          startTime = context.currentTime;
          audio.start(startTime);
        } else {
          audio.play();
        }
  
        // end trial if time limit is set
        if (trial.trial_duration !== null) {
          jsPsych.pluginAPI.setTimeout(function () {
            end_trial();
          }, trial.trial_duration);
        }
      }
  
  
  
      // function to handle responses by the subject
      function after_response(choice) {
  
        // measure rt
        var endTime = performance.now();
        var rt = endTime - startTime;
        if (context !== null) {
          endTime = context.currentTime;
          rt = Math.round((endTime - startTime) * 1000);
        }
        response.button = choice; //parseInt(choice);
        response.rt = rt;
  
        // disable all the buttons after a response
        disable_buttons();
  
        if (trial.response_ends_trial) {
          end_trial();
        }
      }
  
      // function to end trial when it is time
      function end_trial() {
  
        // kill any remaining setTimeout handlers
        jsPsych.pluginAPI.clearAllTimeouts();
  
        // stop the audio file if it is playing
        // remove end event listeners if they exist
        if (context !== null) {
          audio.stop();
        } else {
          audio.pause();
        }
  
        audio.removeEventListener('ended', end_trial);
        audio.removeEventListener('ended', enable_buttons);
  
        // gather the data to store for the trial
        var trial_data = {
          rt: response.rt,
          stimulus: trial.stimulus,
          response: response.button
        };
  
        // clear the display
        display_element.innerHTML = '';
  
        // move on to the next trial
        jsPsych.finishTrial(trial_data);
      }
  
      function button_response(e) {
        var choice = e.currentTarget.getAttribute('data-choice'); // don't use dataset for jsdom compatibility
        after_response(choice);
      }
  
      function disable_buttons() {
        var btns = document.querySelectorAll('.jspsych-audio-betty-response-button');
        for (var i = 0; i < btns.length; i++) {
          var btn_el = btns[i].querySelector('button');
          if (btn_el) {
            btn_el.disabled = true;
          }
          btns[i].removeEventListener('click', button_response);
        }
      }
  
      function enable_buttons() {
        var btns = document.querySelectorAll('.jspsych-audio-betty-response-button');
        for (var i = 0; i < btns.length; i++) {
          var btn_el = btns[i].querySelector('button');
          if (btn_el) {
            btn_el.disabled = false;
          }
          btns[i].addEventListener('click', button_response);
        }
      }
  
  
  
    };
  
    return plugin;
  })();

[myrtevos]

Honestly relieved to hear there may be a 'legitimate' problem and not just me fundamentally misunderstanding HTML/CSS or overlooking a spelling mistake for 5 hours straight 😬

[jodeleeuw]

Hey @myrtevos, this is now fixed in the code and will be included in 6.3.1. The release will be out in the next day or so.

[myrtevos]

Thank you so much!! I'm really grateful for the fast turnaround, and looking forward to learning something from the solution you implemented.

[myrtevos]

Hi,

Thanks again for the super quick bug fix on this! However, I'm still struggling a bit. With the code as given above, the bounding rectangle coordinates for the image-buttons are way off. Width and height are both 0, and the left & right / top & bottom coordinates are identical. Getting the coordinates for the parent div helps somewhat, but not completely: the width/height dimensions especially are way too small. After playing around with it for a while and trying different things I wonder if the problem is that jsPsych's CSS (for an array of buttons, which is what I'm coercing my images into) is interfering with my own CSS. I'll try stripping out the whole 'button' part of the plugin to see if that's the case, but since I'd really like to see if I can get what I need out of the Webgazer extension using only/mostly the standard jsPsych plugins, I thought I'd post here and ask for help again.

Code is still as given in the thread above. Many thanks in advance!

[jodeleeuw]

Width and height are both 0, and the left & right / top & bottom coordinates are identical

This sounds like you need to find a different element to select. Have you tried using the inspector tool to find the right element in the rendered page? I'm wondering if your img-fluid class might be making the image not take up any space in the parent container?

[myrtevos]

Yeah, I've played around with selecting different elements and using the inspector tool to try to understand what's going on. I've tried dropping the img-fluid class, that didn't help. I'm thinking I'll just have to scrap all this HTML/CSS and rewrite it without the fluid container etc. It's slightly perplexing to me, because this code worked fine with getBoundingClientRect() within a jspsych body before and it would have been nice to keep it, but oh well.
Thanks again for your time - I've attached a pic of my cat, Magnus, as recompense. As I write this he is hunting his own tail, which is a mood.

[jodeleeuw]

I just had one more idea....

The way that I fixed the original problem was to add a MutationObserver that listens for the creation of the target elements on the screen. This way if the elements don't exist at the start of the plugin.trial method it will still pick them up when they are created. It occurred to me that if you are putting your targets on the container of the <img> instead of the <img> itself then it could be that the image is not loaded and the container is smaller than it ultimately will be. To be honest I'm not optimistic that this is the answer, but it seems plausible enough to mention.

[myrtevos]

I'd already tried putting the targets on the image itself and got the exact same results - but your comment did lead me to check whether the problem was that the coordinates were being retrieved before the element had rendered. I stuck the console.log($("#img")[0].getBoundingClientRect()) command I've been testing with in a function that runs long after the images are rendered, and voilá: legit coordinates!

So if I've understood you correctly, the webgazer extension is still collecting coordinates too soon in this instance. At any rate I can use this trick to add the coordinates to the trial data through the plugin itself; for now, that will do :)