More questions about how participants are assigned to conditions on Cognition.run

[fzenk]

The advanced configuration page for Cognition experiments says the following about how participants are assigned to different conditions:

“If you specify a number of conditions greater than 1, then Cognition will attempt to assign new participants to conditions to keep them all with equal N.”

I have a few questions about how this works that I don’t think have been answered in previous discussions.

1. Does Cognition evaluate the number of participants assigned to each condition based just on finished runs, or does it also include unfinished runs in those calculations?

2. If I delete runs from the server, will Cognition recalculate the number of participants assigned to each condition and take this into account when assigning future participants?

3. I can imagine situations where it might be desirable towards the end of data collection to manually set the condition for all incoming participants to one value or another. Is there any way to do this at present? I know that I can reset the condition in my code for the experiment, but if I do it that way, then the server will still think that the participant belongs to the condition that Cognition originally assigned it to, right?

And while I’m here, I might as well ask these two general questions about Cognition that I’ve been meaning to post for a while:

4. For some of my experiments, it would be very helpful if I could tell Cognition to pause data collection after a certain number of finished runs has been reached. Is there a way to do this now, and if not, is this a feature that might be added in the future?

5. At the end of my experiments, I redirect participants to an external page using the code shown below. This technique for ending the experiment has been working great for me except that participants see a blank white screen while they are waiting to be redirected, and sometimes this can take a while if the participant has a slow internet connection and there is a lot of data to be uploaded to the server. Is there any way that I can show participants a loading message or a progress bar while they are waiting to be redirected? I’m guessing that this is possible because the progress bar remains visible at the top of the page after the screen goes white.

    jsPsych.init({
        timeline: timeline,
        show_progress_bar: true,
        on_finish: () => window.location.href = "https://www.example-website.com"
    });

Thanks as always to @javidalpe and the rest of the Cognition team for their awesome work on this platform!

[jodeleeuw]

Is there any way that I can show participants a loading message or a progress bar while they are waiting to be redirected?

This is kind of a hacky solution, but I think it will work. On your final trial of the experiment, you can probably render content on the screen in the on_finish function.

var last_trial = {
	type: 'whatever',
	on_finish: () => {
		jsPsych.getDisplayElement().innerHTML = '<p>Thanks! Uploading data to server.</p>'
	}
}

[fzenk]

Thank you @jodeleeuw for sharing this fix with me! I just tried it out and it works perfectly :)

[javidalpe]

Great questions @fzenk !

1. Based on finished, unfinished and declined. Declined is something I'm implementing right now 😄
2. Yes, it will recalculate based on stored runs.
3. The only way you can force the run condition is by adding a query param condition to the task url. For example: https://blablabla.cognition.run/?condition=3 You will need a different task if you want to have a fixed condition.
4. You can pause data collection on the task's configuration. Just uncheck Accepting participants. There is no way of automate it. But this is a great feature request! Should we count both unfinished and finished, or just finished? What should cognition do if limit is set to 3 and 5 participants are finishing the task?
5. That's a good point. I really like @jodeleeuw suggestion -thank you!-, and I think this is something cognition should implement!

[fzenk]

Thank you @SimonStephan31 for your suggestion! The only problem I can see with that solution is that there would be a mismatch between the condition that the server thinks the participant is assigned to and the condition that the participant actually receives when they take the experiment. But perhaps that wouldn't matter if you are at the end of your data collection and not planning to return to having Cognition automatically assign participants to conditions.

[SimonStephan31]

@fzenk ah yes, that is correct. So it would be good (?) to have the "cond" variable written in your data files and treat that one instead of CONDITION as the condition identifier when analyzing the data.

I've now tried the following:

/////////////////////////////////////// Code ///////////////////////

var condition = CONDITION;
//var condition = 1;

jsPsych.data.addProperties({condition: +condition}); // append the variable "condition" to the data frame

/////////////////////////////////////// Code end ///////////////////////

This creates a new column in the data frames called "condition_1". As soon as condition != CONDITION, this new column codes the correct condition.

[fzenk]

Thanks for your response @SimonStephan31! Using "cond" or some other variable name in your code (I have been using "running_list" myself) solves the problem of "CONDITION" referring to two different things. However, it does not address the issue I was trying to describe earlier, i.e., that once you start reassigning participants to one condition or another in your code, the server's tally of how many participants have been assigned to each condition starts to get further and further divorced from the actual number of participants in each condition (but again, this is only problematic if you plan to go back to relying on Cognition to assign participants to conditions at some point in the future).

Here's another possible fix that occurred to me earlier today: On Prolific, you put a URL to a single-trial Cognition experiment (let's call it https://xxxxx.cognition.run) that only has one purpose: to forward the participant to the URL for the main experiment (let's call that one https://yyyyy.cognition.run). When you decide that you want to top off one of the conditions, you could just change the forwarding URL to, e.g., https://yyyyy.cognition.run/?condition=3. Of course, you would also need to capture the participant's URL parameters from Prolific (participant ID, study ID, etc.) and append them to the end of the forwarding URL, but that wouldn't be hard to do. Does that sound like something that might work?

[SimonStephan31]

@fzenk that's sounds like a feasible fix that I will try in the future, thanks! I usually need to top up some conditions only towards the end of a study, to have exactly equally many in each condition, not just approximately equally many. In this case, the problem you were describing does not occur with the "cond" solution, as the study is (almost) complete anyway. If you need to select conditions earlier during a study, howevber, another possibilty to avoid the problem with the tallying could be to (first save and then) delete the data collected so far before you let cognition.run take control again. Of course, this has the disadvantage that not all data of the study are stored online anymore.

[fzenk]

@SimonStephan31 Good point! That would work too.