How to data tag video trials? Possible to tag following trials with previous trials' tags or combine data tags?

[LangDevLab]

For my experiment, participants are presented a series of videos followed by series of audio clips. I am wondering how to tag video trials in the same way audio is tagged?

This is how my audio trials are tagged at the moment and what the audio procedure looks like:

var prac_audio_stimuli5 = [
  {stimulus: 'audio/bd05_minus.wav', data: {test_part: 'practice', level: '5m'}},
  {stimulus: 'audio/bd05.wav', data: {test_part: 'practice', level: '5'}},
  {stimulus: 'audio/bd05_plus.wav', data: {test_part: 'practice', level: '5p'}},
  {stimulus: 'audio/bd05_minus.wav', data: {test_part: 'practice', level: '5m'}},
  {stimulus: 'audio/bd05.wav', data: {test_part: 'practice', level: '5'}},
  {stimulus: 'audio/bd05_plus.wav', data: {test_part: 'practice', level: '5p'}},
];

var prac_audioDL = {
      type: "audio-keyboard-response",
      stimulus: jsPsych.timelineVariable('stimulus'),
      choices: jsPsych.NO_KEYS,
      trial_ends_after_audio: true
    }

    var prac_responseDL = {
          type: "html-keyboard-response",
          stimulus: "<div style='font-size:25px';'><div style='width: 500px;'><div style='float:left';'><b>ADA</b></div>" +
          "<div style='font-size:25px';'><div style='width: 500px;'><div style='float:right';'><b>ABA</b></div>",
          choices: ['leftarrow', 'rightarrow'],
          response_ends_trial: true,
          trial_duration: null,
          data: jsPsych.timelineVariable('data'), // adds tagged data from stimuli bank "prac_audio_stimuli5" to results
          on_finish: function(data){ //this should add 'resp' column, converting key_press output 37,39 to 0,1 respectively
            if (data.key_press == 37) {
              data.resp = 1;
            } else if (data.key_press == 39) {
              data.resp = 0;
            }
          }
        }

        var prac_audioDL5 = {
              timeline: [prac_audioDL, prac_responseDL], // alternates the audio trial and response trial
              timeline_variables: prac_audio_stimuli5, // takes stimuli from "prac_audio_stimuli"
              randomize_order: true // randomize order of stimuli presented
            }

I tried tagging the video trials this way, but it doesn't seem to work. The videos don't play during the experiment and I receive an error in the console:

var prac_incon5ADA = {
timeline: [
    {
        type: 'html-keyboard-response',
        stimulus: '',
        trial_duration: 1
    },
    {
        type: 'video-keyboard-response',
        sources: jsPsych.timelineVariable('sources'),
        data: jsPsych.timelineVariable('data'),
        choices: jsPsych.NO_KEYS,
        trial_ends_after_video: true
    },

],
timeline_variables: [
  { sources: ['video/VdA0.mp4']},
  { sources: ['video/VdA5.mp4'], data: {test_part: 'practice', blocktype: 'ADA', congruency: 'incongruent', level: 'VdA5'}},
  { sources: ['video/VdA0.mp4']},
  { sources: ['video/VdA5.mp4'], data: {test_part: 'practice', blocktype: 'ADA', congruency: 'incongruent', level: 'VdA5'}},
  { sources: ['video/VdA0.mp4']},
  { sources: ['video/VdA5.mp4'], data: {test_part: 'practice', blocktype: 'ADA', congruency: 'incongruent', level: 'VdA5'}},
  { sources: ['video/VdA0.mp4']},
  { sources: ['video/VdA5.mp4'], data: {test_part: 'practice', blocktype: 'ADA', congruency: 'incongruent', level: 'VdA5'}},
  { sources: ['video/VdA0.mp4']},
  { sources: ['video/VdA5.mp4'], data: {test_part: 'practice', blocktype: 'ADA', congruency: 'incongruent', level: 'VdA5'}},
  { sources: ['video/VdA0.mp4']},
  { sources: ['video/VdA5.mp4'], data: {test_part: 'practice', blocktype: 'ADA', congruency: 'incongruent', level: 'VdA5'}},
]
}

In addition, we would like to see if their responses are affected by the video sequence they're shown, therefore Is it possible to combine data tags from the video and audio trials for easier data analysis?

Some clarification on our experiment structure, during one practice trial (same format as our test trials), participants are presented with one of 4 video procedures (prac_incon5ADA or prac_con5ABA, etc.) and then they are presented with a series of audio clips (prac_audio_stimuli5) immediately after, to which they have to respond to each. Would it be possible to tag the audio clips/response with the data tags "congruency: "congruent", "blocktype: "ABA", etc." from the video that they were just presented with?

I'm unsure how to go about finding a solution for this with my current experiment structure. Example of how the two practice trials are structured and the test trials are structured:

// practice trial condition 5 //
if(randomint > 175){
  timeline.push(
    // condition 5: practice playlist #1 (out of 8 possible playlists) // 
    say_ABA, prac_con5ABA, prac_audioDL5,
    say_ABA, prac_incon5ADA, prac_audioDL5
  );
} 

// actual test trials condition 5 //
if(randomint > 175){
  timeline.push(
    // condition 5: test playlist #1 (out of 32 possible playlists) // 
    say_ABA, con5ABA, audioDL5,
    say_ABA, con5ABA, audioDL5,
    say_ABA, con5ABA, audioDL5,
    say_ABA, con5ABA, audioDL5,
    say_ABA, incon5ADA, audioDL5,
    say_ABA, incon5ADA, audioDL5,
    say_ABA, incon5ADA, audioDL5,
    say_ABA, incon5ADA, audioDL5,
    say_ABA, incon5ADA, audioDL5,
    halfway,
// ... continues in same format: incon5ABA and con5ADA are both repeated 5 times //
  );
} 

If anyone has any insight, it would be much appreciated!
Thanks!

[jodeleeuw]

Can you share the text of the error message that you get in the console? The data parameter should work the same for any plugin, so it's likely that the problem you are getting is due to something else in the setup of the video trials.

[jodeleeuw]

I think the issue is that you don't have a data timeline variable for some of the trials in your video block. (jsPsych should do a better job with handling this error.) Try adding data: {} to each of the trials that doesn't have a data variable.

[jodeleeuw]

One way that you could accomplish this is by updating the data for the audio clips after they are presented with the most recent video file. If you give the video trials a unique tag in their data like data: {task: 'video'} then you can find the most recent video trial:

var video = jsPsych.data.get().filter({task: 'video'}).last(1).values()[0].stimulus

You could put this command inside the on_finish event for the audio trials that you need to tag.

on_finish: function(data){
	data.last_video = jsPsych.data.get().filter({task: 'video'}).last(1).values()[0].stimulus
}

The above will create a new entry in the data for the audio trial called last_video and it should be the value of the most recent video trial.