Data saving incomplete with Cognition.run + Prolific

[fzenk]

Hello! I have an experiment that uses Cognition.run for hosting and Prolific for participant recruitment. My problem is that, at least some of the time, the results are not fully uploading to Cognition.run before the participant clicks the link to get redirected back to the Prolific website.

Here is what the code looks like at the end of my experiment:

    // blank trial that ends the experiment and takes the participant to the end message
    var end_task = {
        type: 'html-keyboard-response',
        stimulus: '',
        choices: jsPsych.NO_KEYS,
        trial_duration: 1,
        on_finish: function(){
            jsPsych.endExperiment(end_message);
        }
    };
    timeline.push(end_task);

    // end message
    var end_message = 
        '<p>You have completed the final task. Click on the link below to return to Prolific and finish the experiment.</p>'+
        '<a href="https://app.prolific.co/submissions/complete?cc=XXXXXXXX">Click here to return to Prolific</a>'

    // initiate experiment
    jsPsych.init({
        timeline: timeline,
        show_preload_progress_bar: true,
        show_progress_bar: true,
        auto_update_progress_bar: false,
    });

Sometimes it is just the data from the final task that has failed to upload the server, and in at least one case it was the data from the last two tasks that failed to upload. This struck me as strange because I thought that Cognition.run saves data to the server incrementally as participants are working through the experiment. Maybe that is still what was happening but the user in question had an especially slow internet connection?

When I was testing the experiment on myself, I had a warning message pop up once when I tried to click on the Prolific link before the data had finished uploading to the server. I guess that I could add a sentence to the end message asking participants to wait a bit and try again if their browser gives them a similar warning, but I don't see that as a permanent solution.

Is there something I can do to make sure that the data finishes uploading to Cognition.run before the participant sees the end message with the link to return to Prolific? I found a few other people who have already posted related questions, but fixing the problem was a bit different for them because they were all using their own servers.

Any help you could give me would be very much appreciated. Or is this a question I should be directing to the Cognition.run creators instead? Thank you for your time.

[becky-gilbert]

Hi @fzenk, I'm afraid I can't think of a way to improve your code to deal with this problem - hopefully others will chime in if they have any suggestions. Also tagging the cognition.run creator @javidalpe in case he can help!

[fzenk]

Thank you @becky-gilbert for your kind response and for tagging @javidalpe to see if he has any ideas! I'll make sure to post again to this thread if I find a good fix for the problem in case others are having similar issues with data loss. For now, I think I'll try adding a short exit survey and/or a debrief page at the end of the experiment to provide more of a buffer between the the final task and the end message.

[javidalpe]

Hello @fzenk and @becky-gilbert. I found the issue. endExperiment internally calls on_finish listener (configured in jsPsych.init) and then shows the message. Cognition adds a on_finish callback to consolidate data in the server. But because it doesn't wait (we are not returning a promise), the user is able to click on the link before the data arrives to the server.

I propose two solutions:
a) (Cognition driven) Monkey patching jsPsych.endExperiment. Override endExperiment with a custom code: perform data consolidation before calling the original endExperiment method. Pro jspsych: no work is needed. Cons cognition: new jspsych releases could introduce breaking changes and would require extra testing
b) (jspsych driven) Modify jsPsych.endExperiment. If on_finish callback returns a promise, wait for the promise before showing the message. Pro jspsych: allows any kind of async work at on_finish. Cons cognition: only available for the next version of jspsych.

What do you think @becky-gilbert . More than happy to work on any :)

[fzenk]

Thank you @javidalpe for finding the problem and proposing some solutions to it! I'm glad to hear that this is something that's fixable. I wish there was more I could do on my end to help. I also want to say thanks for all your work on Cognition--it's a great service and a real lifesaver for people like me who love building experiments with jsPsych but don't have experience with setting up and maintaining a server.

[becky-gilbert]

Thanks very much @javidalpe!
@jodeleeuw what do you think of the solutions that Javier has proposed?

[fzenk]

Hello @jodeleeuw! I'm wondering what you think about this issue. It seems to me that this is an important one to get resolved for anyone who wants to use jsPsych + Cognition for their research. @javidalpe has kindly offered some possible fixes, but I think he and @becky-gilbert are waiting for your input before going any further. Thank you :)

[jodeleeuw]

I've opened an issue to track this #2050. I think we should modify the behavior of on_finish to accept a promise.

[fzenk]

Thank you, @jodeleeuw! I'm excited to hear that a fix for this issue is going to be part of version 7.0 :)

[javidalpe]

Thank you @jodeleeuw!

On the meantime @fzenk , I was wondering if you tried another approach. Today I was thinking that you could use the global on_finish callback to redirect the user to prolific instead of manually ending the experiment.

This should work:

// Example trial, could be any trial
var hello_trial = {
    type: 'html-keyboard-response',
    stimulus: 'Hello world!'
}

var timeline = [hello_trial];

// Final trial before redirect
var end_message = 
    '<p>You have completed the final task. Press any button to return to Prolific and finish the experiment.</p>';
    
var end_task = {
    type: 'html-keyboard-response',
    stimulus: end_message,
};
timeline.push(end_task);

// initiate experiment
jsPsych.init({
    timeline: timeline,
    show_progress_bar: true,
    on_finish: () => window.location.href = "https://app.prolific.co/submissions/complete?cc=XXXXXXXX" 
});

[fzenk]

Thank you @javidalpe for the suggestion!

Here's what I'm currently doing in the code at the end of my experiment to manage the issue:

// Exit Survey
// Note: This survey comes after the last task with important data for my study and provides some buffer time
// at the end of the experiment for the data to finish uploading to the server.

    var exit_survey = {
        type: 'survey-html-form',
        data: {trial_name: 'exit_survey'},
        button_label: 'Continue',
        html: 
            '<style>p {text-align:left; spellcheck=false;} input[type="text"] {width:8ch;} fieldset {border:1px solid #999;box-shadow:2px 2px 5px #999;} legend {background:#fff;text-align:left;font-size:110%;}</style>'+
            '<img src="img_exit.jpg" height="150" title="image credit: vectorjuice / Freepik (www.freepik.com)"><br>'+
            '<span style="font-size:125%; font-weight:bold;">Exit Survey</span>'+
            '<p>What do you think the researcher is trying to find out from this experiment?</p>'+
            '<p style="text-align:left;"><textarea name="exit_purpose" rows="3" style="width:90%;" required></textarea></p>'+
            '<p>If you experienced any difficulties while taking part in the experiment, please describe them here.</p>'+
            '<p style="text-align:left;"><textarea name="exit_difficulties" rows="3" style="width:90%;"></textarea></p>'+
            '',
    };
    timeline.push(exit_survey);

// Blank Trial 
// Note: This blank trial completes the experiment and takes the participant to the end message.

    var end_task = {
        type: 'html-keyboard-response',
        stimulus: '',
        choices: jsPsych.NO_KEYS,
        trial_duration: 1,
        on_finish: function(){
            jsPsych.endExperiment(end_message);
        }
    };
    timeline.push(end_task);

// End Message
// Note: This page has a message asking participants not to force the tab to close if they get a pop-up warning
// when they try to exit it. Also, I set the target to "_blank" on the button redirecting participants to Prolific
// (so that Prolific opens is a separate tab on the browser), hoping that this would also give the data a bit longer to 
// upload to the server before the participant tries to exit the tab.
// Question: I'm operating under the assumption that data can still be uploading to the server when participants are on the 
// end message page. I think this is the case because of warning messages I have gotten in the past when trying to exit 
// the tab at the end of the experiment. Am I right about that?

    var end_message = 
        '<img src="img_success.jpg" height="150" title="image credit: syarifahbrit / Freepik (www.freepik.com)"><br>'+
        '<span style="font-size:125%; font-weight:bold;">Nice work!</span>'+
        '<p>You have completed the final task. Click on the link below to return to the Prolific website and finish the experiment.</p>'+
        '<p style="text-align:center;"><a href="https://app.prolific.co/submissions/complete?cc=XXXXXXXX" target="_blank" style="font-size:115%;">Click here to return to Prolific</a></p>'+
        '<p style="text-align:left; width:90%; margin:auto; border:2px; border-style:ridge; padding:10px;">IMPORTANT: If a warning pops up when you attempt to exit this window, it likely means that your results have not finished uploading to the server yet. If you see such a warning, please wait for a minute or two and then try again.</p>'

// Initiate Experiment

    jsPsych.init({
        timeline: timeline,
        show_preload_progress_bar: true,
        show_progress_bar: true,
        auto_update_progress_bar: false
    });

This strategy has been working reasonably well for me so far. I started a new round of piloting a couple of days ago, and I only had incomplete data loading problems with one out of the first 21 participants. If you think that your suggestion will work even better, would you mind explaining to me why that is? I'm not sure I understand how moving the code redirecting the participant to Prolific to the global on_finish callback will help to guard against data loss (because of my general lack of knowledge about how these things work).

BTW, if I'm having these data loss problems where other people are not, I think it might be because one of my tasks involves recording audio responses, and the resulting base64 strings probably take a long time to upload to the server, especially if the participant has a slow internet connection.

Thank you so much for your assistance with this :)

[javidalpe]

Sure!

Cognition.run saves the data using on_finish jspsych config callback (https://www.jspsych.org/core_library/jspsych-core/#jspsychinit). If participant leaves the experiment (closing the tab, navigating to another web, pressing back button, ...) before on_finish is called by the library, the data is not saved and the run is marked as unfinished.

In your current experiment, the user leaves the experiment because it is clicking on a link (https://app.prolific.co/submissions/complete?cc=XXXXXXXX). Even if you are using endExperiment function, in some cases the library it's unable to call on_finish before the browser start loading the new prolific page. You have to find a deterministic solution.

My suggestion is to redirect the user to prolific once the experiment is fully finished. To do so, you have to execute a redirection on the on_finish callback. Finally, let jspsych determine when the experiment is finished, do not call endExperiment.

In my previous example, I'm not calling endExperiment. Jspsych will call on_finish when the participant finishes end_task trial and the data is saved. By the time window.location.hef.... is executed, the data will be already saved.

[fzenk]

Thanks for explaining! That's very clear, and it sounds like an excellent solution. I'll give it a try :)

[fzenk]

Thanks again @javidalpe for suggesting this fix for my problem! It has been working great for me except that participants see a blank screen while they are waiting to be redirected to Prolific at the end of the experiment, which could make them think that something has gone wrong. Is there any way I could show them a loading message or a progress bar while they are waiting for their responses to finish uploading to the server instead of a blank screen?