Trying to sample multiple responses from video-slider-response

[Alexpathy]

Hi!
Could someone help me with a plugin change for the video-slider-response? I want to continuously sample participants' slider responses while they are watching video content.

I tried to implement @becky-gilbert's code from this thread on how to change the audio-slider-response: #2244

My code is below. I got to a point where the code runs but responses are not saved in the data. My changes are on lines:
329-333
344-348
359-361
365

var jsPsychVideoSliderResponse = (function (jspsych) {
  'use strict';

  const info = {
      name: "video-slider-response",
      parameters: {
          /** Array of the video file(s) to play. Video can be provided in multiple file formats for better cross-browser support. */
          stimulus: {
              type: jspsych.ParameterType.VIDEO,
              pretty_name: "Video",
              default: undefined,
              array: true,
          },
          /** Any content here will be displayed below the stimulus. */
          prompt: {
              type: jspsych.ParameterType.HTML_STRING,
              pretty_name: "Prompt",
              default: null,
          },
          /** The width of the video in pixels. */
          width: {
              type: jspsych.ParameterType.INT,
              pretty_name: "Width",
              default: "",
          },
          /** The height of the video display in pixels. */
          height: {
              type: jspsych.ParameterType.INT,
              pretty_name: "Height",
              default: "",
          },
          /** If true, the video will begin playing as soon as it has loaded. */
          autoplay: {
              type: jspsych.ParameterType.BOOL,
              pretty_name: "Autoplay",
              default: true,
          },
          /** If true, the subject will be able to pause the video or move the playback to any point in the video. */
          controls: {
              type: jspsych.ParameterType.BOOL,
              pretty_name: "Controls",
              default: false,
          },
          /** Time to start the clip. If null (default), video will start at the beginning of the file. */
          start: {
              type: jspsych.ParameterType.FLOAT,
              pretty_name: "Start",
              default: null,
          },
          /** Time to stop the clip. If null (default), video will stop at the end of the file. */
          stop: {
              type: jspsych.ParameterType.FLOAT,
              pretty_name: "Stop",
              default: null,
          },
          /** The playback rate of the video. 1 is normal, <1 is slower, >1 is faster. */
          rate: {
              type: jspsych.ParameterType.FLOAT,
              pretty_name: "Rate",
              default: 1,
          },
          /** Sets the minimum value of the slider. */
          min: {
              type: jspsych.ParameterType.INT,
              pretty_name: "Min slider",
              default: 0,
          },
          /** Sets the maximum value of the slider. */
          max: {
              type: jspsych.ParameterType.INT,
              pretty_name: "Max slider",
              default: 100,
          },
          /** Sets the starting value of the slider. */
          slider_start: {
              type: jspsych.ParameterType.INT,
              pretty_name: "Slider starting value",
              default: 50,
          },
          /** Sets the step of the slider. */
          step: {
              type: jspsych.ParameterType.INT,
              pretty_name: "Step",
              default: 1,
          },
          /** Array containing the labels for the slider. Labels will be displayed at equidistant locations along the slider. */
          labels: {
              type: jspsych.ParameterType.HTML_STRING,
              pretty_name: "Labels",
              default: [],
              array: true,
          },
          /** Width of the slider in pixels. */
          slider_width: {
              type: jspsych.ParameterType.INT,
              pretty_name: "Slider width",
              default: null,
          },
          /** Label of the button to advance. */
          button_label: {
              type: jspsych.ParameterType.STRING,
              pretty_name: "Button label",
              default: "Continue",
          },
          /** If true, the participant will have to move the slider before continuing. */
          require_movement: {
              type: jspsych.ParameterType.BOOL,
              pretty_name: "Require movement",
              default: false,
          },
          /** If true, the trial will end immediately after the video finishes playing. */
          trial_ends_after_video: {
              type: jspsych.ParameterType.BOOL,
              pretty_name: "End trial after video finishes",
              default: false,
          },
          /** How long to show trial before it ends. */
          trial_duration: {
              type: jspsych.ParameterType.INT,
              pretty_name: "Trial duration",
              default: null,
          },
          /** If true, the trial will end when subject makes a response. */
          response_ends_trial: {
              type: jspsych.ParameterType.BOOL,
              pretty_name: "Response ends trial",
              default: true,
          },
          /** If true, then responses are allowed while the video is playing. If false, then the video must finish playing before a response is accepted. */
          response_allowed_while_playing: {
              type: jspsych.ParameterType.BOOL,
              pretty_name: "Response allowed while playing",
              default: true,
          },
      },
  };
  /**
   * **video-slider-response**
   *
   * jsPsych plugin for playing a video file and getting a slider response
   *
   * @author Josh de Leeuw
   * @see {@link https://www.jspsych.org/plugins/jspsych-video-slider-response/ video-slider-response plugin documentation on jspsych.org}
   */
  class VideoSliderResponsePlugin {
      constructor(jsPsych) {
          this.jsPsych = jsPsych;
      }
      trial(display_element, trial) {
          if (!Array.isArray(trial.stimulus)) {
              throw new Error(`
        The stimulus property for the video-slider-response plugin must be an array
        of files. See https://www.jspsych.org/latest/plugins/video-slider-response/#parameters
      `);
          }
          // half of the thumb width value from jspsych.css, used to adjust the label positions
          var half_thumb_width = 7.5;
          // setup stimulus
          var video_html = '<video id="jspsych-video-slider-response-stimulus-video"';
          if (trial.width) {
              video_html += ' width="' + trial.width + '"';
          }
          if (trial.height) {
              video_html += ' height="' + trial.height + '"';
          }
          if (trial.autoplay && trial.start == null) {
              // if autoplay is true and the start time is specified, then the video will start automatically
              // via the play() method, rather than the autoplay attribute, to prevent showing the first frame
              video_html += " autoplay ";
          }
          if (trial.controls) {
              video_html += " controls ";
          }
          if (trial.start !== null) {
              // hide video element when page loads if the start time is specified,
              // to prevent the video element from showing the first frame
              video_html += ' style="visibility: hidden;"';
          }
          video_html += ">";
          var video_preload_blob = this.jsPsych.pluginAPI.getVideoBuffer(trial.stimulus[0]);
          if (!video_preload_blob) {
              for (var i = 0; i < trial.stimulus.length; i++) {
                  var file_name = trial.stimulus[i];
                  if (file_name.indexOf("?") > -1) {
                      file_name = file_name.substring(0, file_name.indexOf("?"));
                  }
                  var type = file_name.substr(file_name.lastIndexOf(".") + 1);
                  type = type.toLowerCase();
                  if (type == "mov") {
                      console.warn("Warning: video-slider-response plugin does not reliably support .mov files.");
                  }
                  video_html += '<source src="' + file_name + '" type="video/' + type + '">';
              }
          }
          video_html += "</video>";
          var html = '<div id="jspsych-video-slider-response-wrapper" style="margin: 100px 0px;">';
          html += '<div id="jspsych-video-slider-response-stimulus">' + video_html + "</div>";
          html +=
              '<div class="jspsych-video-slider-response-container" style="position:relative; margin: 0 auto 3em auto; width:';
          if (trial.slider_width !== null) {
              html += trial.slider_width + "px;";
          }
          else {
              html += "auto;";
          }
          html += '">';
          html +=
              '<input type="range" class="jspsych-slider" value="' +
                  trial.slider_start +
                  '" min="' +
                  trial.min +
                  '" max="' +
                  trial.max +
                  '" step="' +
                  trial.step +
                  '" id="jspsych-video-slider-response-response"';
          if (!trial.response_allowed_while_playing) {
              html += " disabled";
          }
          html += "></input><div>";
          for (var j = 0; j < trial.labels.length; j++) {
              var label_width_perc = 100 / (trial.labels.length - 1);
              var percent_of_range = j * (100 / (trial.labels.length - 1));
              var percent_dist_from_center = ((percent_of_range - 50) / 50) * 100;
              var offset = (percent_dist_from_center * half_thumb_width) / 100;
              html +=
                  '<div style="border: 1px solid transparent; display: inline-block; position: absolute; ' +
                      "left:calc(" +
                      percent_of_range +
                      "% - (" +
                      label_width_perc +
                      "% / 2) - " +
                      offset +
                      "px); text-align: center; width: " +
                      label_width_perc +
                      '%;">';
              html += '<span style="text-align: center; font-size: 80%;">' + trial.labels[j] + "</span>";
              html += "</div>";
          }
          html += "</div>";
          html += "</div>";
          html += "</div>";
          // add prompt if there is one
          if (trial.prompt !== null) {
              html += "<div>" + trial.prompt + "</div>";
          }
          // add submit button
          var next_disabled_attribute = "";
          if (trial.require_movement || !trial.response_allowed_while_playing) {
              next_disabled_attribute = "disabled";
          }
          html +=
              '<button id="jspsych-video-slider-response-next" class="jspsych-btn" ' +
                  next_disabled_attribute +
                  ">" +
                  trial.button_label +
                  "</button>";
          display_element.innerHTML = html;
          var video_element = display_element.querySelector("#jspsych-video-slider-response-stimulus-video");
          if (video_preload_blob) {
              video_element.src = video_preload_blob;
          }
          video_element.onended = () => {
              if (trial.trial_ends_after_video) {
                  end_trial();
              }
              else if (!trial.response_allowed_while_playing) {
                  enable_slider();
              }
          };
          video_element.playbackRate = trial.rate;
          // if video start time is specified, hide the video and set the starting time
          // before showing and playing, so that the video doesn't automatically show the first frame
          if (trial.start !== null) {
              video_element.pause();
              video_element.onseeked = () => {
                  video_element.style.visibility = "visible";
                  video_element.muted = false;
                  if (trial.autoplay) {
                      video_element.play();
                  }
                  else {
                      video_element.pause();
                  }
                  video_element.onseeked = () => { };
              };
              video_element.onplaying = () => {
                  video_element.currentTime = trial.start;
                  video_element.onplaying = () => { };
              };
              // fix for iOS/MacOS browsers: videos aren't seekable until they start playing, so need to hide/mute, play,
              // change current time, then show/unmute
              video_element.muted = true;
              video_element.play();
          }
          let stopped = false;
          if (trial.stop !== null) {
              video_element.addEventListener("timeupdate", (e) => {
                  var currenttime = video_element.currentTime;
                  if (currenttime >= trial.stop) {
                      video_element.pause();
                      if (trial.trial_ends_after_video && !stopped) {
                          // this is to prevent end_trial from being called twice, because the timeupdate event
                          // can fire in quick succession
                          stopped = true;
                          end_trial();
                      }
                      if (!trial.response_allowed_while_playing) {
                          enable_slider();
                      }
                  }
              });
          }
          if (trial.require_movement) {
              const enable_button = () => {
                  display_element.querySelector("#jspsych-video-slider-response-next").disabled = false;
              };
              display_element
                  .querySelector("#jspsych-video-slider-response-response")
                  .addEventListener("mousedown", enable_button);
              display_element
                  .querySelector("#jspsych-video-slider-response-response")
                  .addEventListener("touchstart", enable_button);
              display_element
                  .querySelector("#jspsych-video-slider-response-response")
                  .addEventListener("change", enable_button);
          }
          var startTime = performance.now();
          // for storing the set of slider values sampled every second
            var responses = [];   
        // global variable for the response interval timer
            var response_interval;
          // function to end trial when it is time
          const end_trial = () => {
              // kill any remaining setTimeout handlers
              this.jsPsych.pluginAPI.clearAllTimeouts();
              // stop the video file if it is playing
              // remove any remaining end event handlers
              display_element
                  .querySelector("#jspsych-video-slider-response-stimulus-video")
                  .pause();
              display_element.querySelector("#jspsych-video-slider-response-stimulus-video").onended = () => { };
              // gather the data to store for the trial
              var trial_data = {
                responses: responses,  // <-- add new responses array to the data
                stimulus: trial.stimulus,
                slider_start: trial.slider_start,
              };
              // clear the display
              display_element.innerHTML = "";
              // move on to the next trial
              this.jsPsych.finishTrial(trial_data);
          };
          display_element
              .querySelector("#jspsych-video-slider-response-next")
              .addEventListener("click", () => {
              // measure response time
              var endTime = performance.now();
              response_interval = setInterval(function(){
                responses.push(display_element.querySelector('#jspsych-video-slider-response-response').valueAsNumber);
            }, 1000)
         end_trial();
          });
          
          clearInterval(response_interval);
          // function to enable slider after video ends
          function enable_slider() {
              document.querySelector("#jspsych-video-slider-response-response").disabled = false;
              if (!trial.require_movement) {
                  document.querySelector("#jspsych-video-slider-response-next").disabled = false;
              }
          }
          // end trial if time limit is set
          if (trial.trial_duration !== null) {
              this.jsPsych.pluginAPI.setTimeout(end_trial, trial.trial_duration);
          }
      }
      simulate(trial, simulation_mode, simulation_options, load_callback) {
          if (simulation_mode == "data-only") {
              load_callback();
              this.simulate_data_only(trial, simulation_options);
          }
          if (simulation_mode == "visual") {
              this.simulate_visual(trial, simulation_options, load_callback);
          }
      }
      create_simulation_data(trial, simulation_options) {
          const default_data = {
              stimulus: trial.stimulus,
              slider_start: trial.slider_start,
              response: this.jsPsych.randomization.randomInt(trial.min, trial.max),
              rt: this.jsPsych.randomization.sampleExGaussian(500, 50, 1 / 150, true),
              start: trial.start,
          };
          const data = this.jsPsych.pluginAPI.mergeSimulationData(default_data, simulation_options);
          this.jsPsych.pluginAPI.ensureSimulationDataConsistency(trial, data);
          return data;
      }
      simulate_data_only(trial, simulation_options) {
          const data = this.create_simulation_data(trial, simulation_options);
          this.jsPsych.finishTrial(data);
      }
      simulate_visual(trial, simulation_options, load_callback) {
          const data = this.create_simulation_data(trial, simulation_options);
          const display_element = this.jsPsych.getDisplayElement();
          this.trial(display_element, trial);
          load_callback();
          const video_element = display_element.querySelector("#jspsych-video-button-response-stimulus");
          const respond = () => {
              if (data.rt !== null) {
                  const el = display_element.querySelector("input[type='range']");
                  setTimeout(() => {
                      this.jsPsych.pluginAPI.clickTarget(el);
                      el.valueAsNumber = data.response;
                  }, data.rt / 2);
                  this.jsPsych.pluginAPI.clickTarget(display_element.querySelector("button"), data.rt);
              }
          };
          if (!trial.response_allowed_while_playing) {
              video_element.addEventListener("ended", respond);
          }
          else {
              respond();
          }
      }
  }
  VideoSliderResponsePlugin.info = info;

  return VideoSliderResponsePlugin;

})(jsPsychModule);

Then I run the plugin through the code below.

<!DOCTYPE html>
<html>
  <head>
    <title>My experiment</title>
    <script src="jsPysch/jspsych.js"></script>
    <script src="https://unpkg.com/@jspsych/[email protected]"></script>
    <script src="jsPysch/plugin-video-slider-response-c.js"></script>
    <link href="https://unpkg.com/[email protected]/css/jspsych.css" rel="stylesheet" type="text/css" />
    <link rel="stylesheet" href="https://unpkg.com/@jspsych/[email protected]/css/survey.css">
    
  </head>
  <style>
  </style>
  <body></body>
  <script>

    /* initialize jsPsych */
    var jsPsych = initJsPsych({
      show_progress_bar: true,
  on_finish: function() {
    jsPsych.data.displayData();
  },
});

/* create timeline */
var timeline = [];

/* define welcome trial */
var welcome = {
      type: jsPsychHtmlKeyboardResponse,
      stimulus: `
      <h1>Welcome to an experiment about emotions and decision-making.</h1> <p>The experiment consist of different elements relating to emotions and decision-making, e.g. text, pictures and videos. Your task is to pay close attention to the material presented and answer corresponding questions, as you deem appropriate. You will go through several trials with different material and you will be notified when the experiment is finished.</p>
      <p>The approximate time to finish the experiment is between 20 – 30 minutes.</p>
      <p>Click any key to continue and start the experiment.</p>
      `,
       post_trial_gap: 500
    };
    timeline.push(welcome);

var test = {
    type: jsPsychVideoSliderResponse,
    stimulus: ['stimuli/ID124_vid3_crop_downsized.mp4'],
    labels: ["Hated it", "Loved it"],
  prompt: '<p>Please rate your enjoyment of the video clip.</p>'
};

timeline.push(test);


    /* start the experiment */
    jsPsych.run(timeline);

  </script>
</html>

I get this data after my experiment:

[
	{
		"rt": 1294,
		"stimulus": "\n      <h1>Welcome to an experiment about emotions and decision-making.</h1> <p>The experiment consist of different elements relating to emotions and decision-making, e.g. text, pictures and videos. Your task is to pay close attention to the material presented and answer corresponding questions, as you deem appropriate. You will go through several trials with different material and you will be notified when the experiment is finished.</p>\n      <p>The approximate time to finish the experiment is between 20 – 30 minutes.</p>\n      <p>Click any key to continue and start the experiment.</p>\n      ",
		"response": " ",
		"trial_type": "html-keyboard-response",
		"trial_index": 0,
		"time_elapsed": 1339,
		"internal_node_id": "0.0-0.0"
	},
	{
		"responses": [],
		"stimulus": [
			"stimuli/ID124_vid3_crop_downsized.mp4"
		],
		"slider_start": 50,
		"trial_type": "video-slider-response",
		"trial_index": 1,
		"time_elapsed": 3638,
		"internal_node_id": "0.0-1.0"
	}
]

Obviously, the response- and responses array are empty. At this point I am not sure if I managed to save multiple slider responses and failed to display the data or if the multiple slides responses failed to be logged at all.

Also, if you have any ideas on how to enable the continue button first after the video content finished, while still sampling responses as long as participant interacts with the slider, that would be helpful. I managed to turn off the button but then I was not able to continue :P

Lastly, some of my queries have been disabled. I don't feel like I break code of conduct, so am not sure why they are disabled. If any of the admins could say something about this I can make more efficient inqueries in the future.

Thanks!

[jodeleeuw]

Lastly, some of my queries have been disabled. I don't feel like I break code of conduct, so am not sure why they are disabled. If any of the admins could say something about this I can make more efficient inqueries in the future.

Hi @Alexpathy,

I don't think any of us have disabled any queries. Perhaps this is something on GitHub's end? Is there a thread you can point to so I can investigate?

[Alexpathy]

Sry if I'm incorrect about the disabling bit, but as soon as I post, my inquery becomes transparent in the Discussions list. This was not the case when I started posting. Since I did not receive any comments on my last posts I assumed the transparency meant the posts have been disabled.