Precise timing of stimuli presentation and accurate recording of RTs

[kimelit]

Hi,

I'm worried about timing of stimuli presentation and accurate recording of RTs (e.g., a browser has its own internal priorities, other things are running in the background in addition to the browser). Is there any solution for that? If not a solution, perhaps some information about the overhead?

Thanks!

[becky-gilbert]

Hi,

I'll try to answer your specific question first, then add more info about general timing issues and information/resources.

jsPsych currently uses a function called setTimeout to control stimulus timings, and a high-accuracy/precision timestamp called performance.now to get response times. The accuracy and precision with these methods are reported here:
de Leeuw 2015: https://doi-org.ezp.lib.cam.ac.uk/10.3758/s13428-014-0458-y
de Leeuw & Motz 2016: https://doi-org.ezp.lib.cam.ac.uk/10.3758/s13428-015-0567-2
(And in some other papers that I'll link to below)

If you want to be very sure that you have accurate stimulus display times, instead of setTimeout you can use a function called requestAnimationFrame, which is a higher-priority function that allows you to run code linked to every frame update. jsPsych will soon switch over to using requestAnimationFrame, but we're currently doing some testing to figure out the best way to implement this.

As far as I'm aware, there's no way to know whether the participant's computer is overloaded. However, modern computers are higher-spec than they used to be, and modern browsers are now pretty good at prioritizing the page that is open/active over any others in the background, so that helps. As part of our testing, we're also planning to look at how different timing methods perform under different loads.

[kimelit]

Thanks so much!
Is there an estimation of the error that might occur? I mean for example that if I get an RT of 300 ms, can I be sure that it was in fact 200-400 ms?

[jodeleeuw]

Hi @kimelit ,

RTs from jsPsych will almost always be slightly on the long side. In our 2016 paper that @becky-gilbert linked above we found that RTs were 10-40ms longer in JavaScript than in Psychtoolbox. Other folks have run similar kinds of timing tests. Here's a nice interactive table showing absolute lags of various options, including jsPsych. You'll notice that the lag varies by browser and OS, because ultimately the timing measurement is being performed by the particular browser that someone is using to run the experiment. But lags are definitely under 100ms.

[kimelit]

Thank you :)! And what about stimuli presentation? Their timing/duration may vary too, right?

[jodeleeuw]

You're welcome! Presentation timing/duration varies too, but not as much as RTs. There's information on timing of displays in the table I linked above. The summary is that for most trials the timing of the display will be accurate in terms of the number of frames. Most monitors refresh 60 times per second, so if your display timing is a multiple of ~16.67 then it should accurate most of the time. But occasionally there will be a dropped frame or an added frame and the display duration will be off by 16.67 ms or so. And sometimes there are two or more dropped/added frames, but that's pretty rare.

[kimelit]

Thanks! That actually happens with Psychtoolbox too...

[becky-gilbert]

And more about stimulus and response timing:

Since your question may not be about jsPsych specifically, but about stimulus presentations and RTs in online experiments more generally, it might also be worth asking this question in the online experiments google group forum as well: https://groups.google.com/forum/#!forum/online-experiments
For instance, this response provides a good summary of the current state of affairs: https://groups.google.com/g/online-experiments/c/3I0susbqQF8

Another thing to do is to look at the papers listed here: https://www.jspsych.org/about/about/#response-times Although the papers listed are now a bit out of date, so here are some additional ones:
Reimers & Stewart (2016): https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5003904/
Henninger et al. (2019): https://psyarxiv.com/fqr49
Garaizar & Reips (2019): https://doi-org.ezp.lib.cam.ac.uk/10.3758/s13428-018-1126-4
Bridges et al. (2020): https://psyarxiv.com/d6nu5/
Anwyl-Irvine et al. (2020): https://psyarxiv.com/jfeca/ (Gorilla COI)
Pronk et al. (2020): https://doi-org.ezp.lib.cam.ac.uk/10.3758/s13428-019-01321-2

Most of these papers test on the most common browsers, which is helpful for understanding how much variability there can be across browsers, and some of these also address the issue of hardware specs and CPU load. For instance, Reimers & Stewart (2016) tested on different systems and conclude that "within-system variability is generally small, even in low-powered machines under high load".

At the moment I don't think it's clear whether certain browsers are always better or worse than others, which is partly because different studies report different/conflicting results, and partly because browsers perform differently across different operating systems. Also, at this point the up-to-date versions of the most common browsers now support the necessary features, e.g.
https://caniuse.com/#search=performance.now
https://caniuse.com/#search=requestAnimationFrame
So I don't have any specific recommendations about which browsers to allow/exclude, except that internet explorer is generally a bad choice for timing!

Finally, there have been timing-related threads in the old jsPsych forum, so I'll post the links here in case you haven't seen them:
https://groups.google.com/d/msg/jspsych/RRVmG2GGr60/WfJmi4eXCAAJ
https://groups.google.com/d/msg/jspsych/FvRqZLvKzlA/seLVnOl3AQAJ
https://groups.google.com/d/msg/jspsych/1xZILtKwK5g/D6wTUcb6BgAJ
https://groups.google.com/d/msg/jspsych/sursrAKdcc0/FIWIBpQhBgAJ
(Disclaimer: some of the information in these links might now be out of date!)

I hope this helps, and that others will chime in with anything I've missed.

[kimelit]

@becky-gilbert thank you for all the resources and links!

[petyaracz]

Can I get jspsych to log the timestamp when the video finishes playing? on_finish won't work for this unless I end the trial when the video is complete. Logging the beginning and the end of the video would allow me to look for videos that got stuck.