Playing an audio over a video

[miguelo93]

Hi all,

I'm trying to build an experiment where an audio and a video are presented one next to the other in time. I could easily use the audio-keyboard-response and the video-keyboard-response plugins. I've implemented it and it works just fine. I just think it would be nicer if the video could stay on screen while I'm presenting the audio. Is there a way to overlap different plugins ? The responses are collected in a next html-button-response plugin.

Thanks!

[becky-gilbert]

Hi @miguelo93, sorry that there isn't an easy way to do this yet. But we have added it to an issue so that we can add it to a future release: #1640

If I understand correctly, you want the audio and video to play one after another (rather than at the same time). Is that right? And how do the two files start playing - does the participant use the controls to play each file, or do you want them to start automatically? If you can provide some more information about exactly what you want to happen, then hopefully we can help you figure out how to do it.

[miguelo93]

Hi @becky-gilbert , thanks for the reply! Yes I need different audio and videos playing one after another (and viceversa in a block-desing). Participants don't have to use the control, just passivley looking/hearing at them. The response is then collected in a response plugin. I now think that I might find a way to combine the audio and the video with some video editing software, but since I would have to create various combinations, I'm still looking for a possible solution. The #1640 issue seems exactly what I would need. thanks again.

[mkissine]

Hi, I'm trying a similar thing, I need to trials to simultaneously combine a video and an audio file. Do I understand correctly that there is no easy way as yet to do it in jsPsych?

[becky-gilbert]

Sorry for the delay @miguelo93! This might be a little tricky to set up, depending on your level of JavaScript experience. What I would do is use one of the video-* plugins as a starting point, and open one of the audio-* plugins and incorporate the audio-related code into the video plugin to create your combined audio-video plugin. You'll need to add a parameter for the audio stimulus. You'll also either need to add another parameter that tells the plugin whether the audio should play before the video or vice versa, or create two separate plugins for these two playback orders (audio-then-video and video-then-audio). To get one file to play after another, what you can do is use the 'ended' event for one of the media elements to trigger the playback for the other. Then, assuming you want the trial to end after the second media element finishes playing, you would call the end_trial function in that element's 'ended' event. So part of the logic might look something like this, for a plugin that plays the audio followed by the video:

audio.addEventListener('ended', function() {
  // code here to play the video
});

video.addEventListener('ended', end_trial);

// start trial by playing audio
if (context !== null) {
  startTime = context.currentTime;
  audio.start(startTime);
} else {
  audio.play();
}

Or, if you're creating one plugin that plays the files in either order, depending on a parameter called play_audio_first (true or false), then it might look like this:

audio.addEventListener('ended', function() {
  if (trial.play_audio_first) {
    // code here to play the video
  } else {
    end_trial();
  }
});

video.addEventListener('ended', function() {
if (trial.play_audio_first) {
    end_trial();
  } else {
    // code here to play audio
  }
});

if (trial.play_audio_first) {
  // code here to start trial by playing audio
} else {
  // code here to start trial by playing video
}

Does that make sense? If you're able to make a start on this and post your code, then I'd be happy to help you debug any problems.

[becky-gilbert]

@mkissine no, there isn't an easy way to do this yet. We are planning to add an audio extension, which will allow you to add audio 'on top' of any other plugin, but we haven't done this yet.

I suggest following the steps in my other comment, except that instead of having the end of one media element trigger the start of the other, you can just have the audio and video start playing at the same time. Keep in mind that it will be tricky to ensure perfect synchronization between them - the best way to do that would be to put the audio into the video file. But if you make sure to preload all of your files, then hopefully you won't have any major issues with playback delays that throw off the synchronization. Feel free to post your code if you'd like any help debugging.