unable to load/use audio files dynamically during experiment

[bjornherrmann]

Hi,

short version:
How can I play an audio file for which I generated the file name during the experiment (based on a participant's input; different participants would thus be presented with different files)?

longer version:
I am building a multi-step experiment. First, participants listen to sounds and a threshold is determined. I would like to use this threshold value to select a subset of ~100 audiofiles from ~10000 pre-generated audio files (I would like to avoid loading all 10000 files = ~300 MB), which are then used in the next block of the experiment.

I tried a lot of things. All works when I do things statically, i.e., pre-define the threshold and the 100 audio files before everything starts. But the dynamic threshold and use of the 100 files does not work. Audio files are undefined then, although all variables are in place when I check using on_start. I assume that using the function with on_start gets run later, while stimulus assignment (not being a function) is run initially when audio files are still undefined.

Is there any solution to dynamic use/load of audio files based on the experiment?
I would be grateful for input,

Bjorn

[jodeleeuw]

Hi @bjornherrmann,

If you are using jsPsych 6.3.0+ then you could use the preload plugin. You can pass a function as the parameter for audio to select the subset of audio you want to load at that moment.

[bjornherrmann]

Thank you for your quick reply. I use jsPsych 6.3.0 and the preload plugin is indeed great. I tested whether a function could work as you recommend, but seem not able to.
I generate a list of the 100 audio files I need in a dummy trial before I preload the files (see below [simplified]). In the timeline, I first place PL_init then PL_preload_audio and then run my trials. I think the problem arises at the trial level and the assignment to "stimulus: ". In the code, below jsPsych.timelineVariable('fname', true) is undefined when I use the dynamic generation of audiolist (note, the json object related to 'fname' is also generated in PL_init), but not when I define audiolist (and the json object) statically and do not use PL_init. Do you have any recommendations where I could be off track? Thank you.

// dummy trial
var PL_init = {
	type: 'html-keyboard-response',
	stimulus: '<div style="font-size:60px;" tabindex="0"></div>',
	trial_duration: 100,
	choices: jsPsych.NO_KEYS,
	on_finish: function() {
           ...
           audiolist = .... // participant specific
	},
};

var PL_preload_audio = {
	type: 'preload',
	audio: audiolist,
};

/* same result as for using the previous 
var PL_preload_audio = {
	type: 'preload',
	audio: function() {
          return audiolist;
       }
};
*/

var PL_trial_play = {
	type: 'audio-keyboard-response',
	stimulus: function() {
		var stim = audio_PL_folder + jsPsych.timelineVariable('fname', true) + '.mp3';
		return stim
	},
 	choices: jsPsych.NO_KEYS,
	prompt: '+',
	trial_ends_after_audio: true
};

[jodeleeuw]

How are you specifying the timeline_variables argument for the timeline that contains PL_trial_play?

[bjornherrmann]

Thank you Josh for taking the time. I built a simplified version of my experiment to describe the problem and avoid code complexities:

// ---------- begin code ----------------------
var audioname_pl = [];
var obj = [{}];
//var audioname_pl = ["audExpMP3/exp_V01_B03_D180_UP09.mp3"];
//var obj = [{"fname": "exp_V01_B03_D180_UP09"}];

var PL_preload_audio = {
  type: 'preload',
  audio: function() {
    audioname_pl = ["audExpMP3/exp_V01_B03_D180_UP09.mp3"];
    obj = [{"fname": "exp_V01_B03_D180_UP09"}];
    return audioname_pl;
  },
};

var PL_instructions = {
  type: 'html-button-response',
  stimulus: 'Instructions',
  choices: ['BEGIN'],
  on_finish: function() {
    console.log(obj);
    console.log(audioname_pl);
  }
};

var PL_trial_play = {
  type: 'audio-keyboard-response',
  stimulus: function() {
    var stim = "audExpMP3/" + jsPsych.timelineVariable('fname', true) + ".mp3";
    return stim
  },
  choices: jsPsych.NO_KEYS,
  prompt: '+',
};

var PL_trial = {
  timeline: [PL_trial_play],
  timeline_variables: obj,
  on_start: function() {
    console.log(obj);
    console.log(audioname_pl);
  }
};

var PLtimeline = [];
PLtimeline.push(PL_preload_audio);
PLtimeline.push(PL_instructions);
PLtimeline.push(PL_trial);

var PROC_perclearn_test = {
	timeline: PLtimeline
};

// -------------- end code ---------------

This code produces the error: "Failed to load audio file "audExpMP3/undefined.mp3".". However, if I uncomment the 3rd and 4th line (i.e., audioname_pl and obj are allocated right at the beginning [static]), the code works. Note that at the end of PL_instructions and the begining of PL_trial, obj and audioname_pl are properly defined/filled in either case. Same result in Mozilla and Opera browsers. Am I missing something more general about javascript and how scripts are loaded? Do you have any suggestions? My intention for my experiment was to make the "D180" part of the file name dependent on the participant's responses.

Another odd issue arose. If I comment out the line "PLtimeline.push(PL_instructions);", that is, do not run the instruction trial first, I get the following error message instead of the sound being played: "Uncaught (in promise) DOMException: The play method is not allowed by the user agent or the platform in the current context, possibly because the user denied permission." If the instruction trial is run first, then the sound plays (assuming audioname_pl and obj are allocated outside of the preload trial). This maybe an unrelated issue.

I am grateful for your input.

[jodeleeuw]

The problem here is that PL_trial is defined when the page is loaded, before the experiment runs. This means that in the code snippet below, obj is still [{}].

var PL_trial = {
  timeline: [PL_trial_play],
  timeline_variables: obj,
  on_start: function() {
    console.log(obj);
    console.log(audioname_pl);
  }
};

Normally the fix for this would be to use dynamic parameters but unfortunately we don't yet support dynamic parameters for timeline variables.

One way to solve this problem is to use a looping timeline in conjunction with dynamic parameters to create a replacement for timeline variables with more flexibility. For example, you can assign a global variable (perhaps by selecting the next item in an array) in the loop_function of the timeline and then pull that value as the trial parameter.

[bjornherrmann]

Thank you. I figured it was something about being unable to dynamically input to timeline_variables. Thank you for your suggestion, I will check it out. I also had another idea how to work around, which seems to work. Basically, I keep everything that goes into the timeline_variables static (globally assigned at the beginning) by splitting up my files names and then fill in the missing piece with a string that I may update while I run through the script. The script below seems to work, with 'param' being the relevant parameter that gets changed dynamically.

var audioname_pl = [];
var obj = [{"fnameP1": "exp_V01_B03_", "fnameP2": "_UP09"}];

var param = "";
var PL_preload_audio = {
	type: 'preload',
	audio: function() {
		param = "D180";
		audioname_pl = ["audExpMP3/exp_V01_B03_D180_UP09.mp3"];
		return audioname_pl;
	},
};

var PL_instructions = {
	type: 'html-button-response',
	stimulus: 'Instructions',
	choices: ['BEGIN'],
	on_finish: function() {
		console.log(obj);
		console.log(audioname_pl);
	}
};

var PL_trial_play = {
	type: 'audio-keyboard-response',
	stimulus: function() {
		var stim = "audExpMP3/" + jsPsych.timelineVariable('fnameP1', true) + param + jsPsych.timelineVariable('fnameP2', true) + ".mp3";
		return stim
	},
 	choices: jsPsych.NO_KEYS,
	prompt: '+',
};

var PL_trial = {
	timeline: [PL_trial_play],
	timeline_variables: obj,
	on_start: function() {
		console.log(obj);
		console.log(audioname_pl);
	}
};

var PLtimeline = [];
PLtimeline.push(PL_preload_audio);
PLtimeline.push(PL_instructions);
PLtimeline.push(PL_trial);

var PROC_perclearn_test = {
	timeline: PLtimeline
};

Thank you for all your input Josh. I appreciate it.