JsPsych Data Structure

[dm0rg8n]

Hi Josh and GitHub community,

I was just wondering if anyone wouldn’t mind giving me some advice about accessing and storing the data collected using JsPsych?

Essentially I’m just wondering how I can look at the data structure which Jspsych uses so that I can set up my database in phpmyadmin accordingly? So for example does jspsych store participant responses in an array in a single key and value pair for the function used or multiple key value pairs for each trial? I think I am particularly confused because i'm unsure which column names the JsPsych package assigns to the data and in turn which column names I should use for my database so that the data can be inserted there.

Thanks,
David

[vijaymarupudi]

I would not recommend storing jsPsych data parsed into columns of a relational database if you're not going to be querying it heavily. By it's nature, the data is messy, and each plugins adds its own field to the object for each trial. What I'd recommend is

CREATE TABLE data {
 participantID INTEGER NOT NULL,
 jsPsychData TEXT NOT NULL,
 createdtime DATETIME NOT NULL
}

where jsPsychData is the JSON for the experiment.

If you have a use case where you really need to add the columns to the db, then let's talk.

[jodeleeuw]

I think @vijaymarupudi's solution is very reasonable. I do store my data in columns to make the export to CSV easier. I use the jsPsych.data.get().uniqueNames() function (docs page) to get a list of all of the columns in the data, and then I create the database structure from those columns. Note that you have to run the experiment to completion (or at least through all of the trials that generate different kinds of data) before calling the method.

[jodeleeuw]

Also, here's a helper library that I wrote that handles the jsPsych -> MySQL pipeline via PHP:

https://github.com/vassar-cogscilab/php-mysql-save-data

[jodeleeuw]

I think this is pretty unlikely. MySQL backends are designed to requests at a much higher rate than a typical online experiment generates. If this kind of data loss happened then it would be a major problem for web infrastructure that relies on MySQL backends. FWIW, I've never seen data loss in my experiments that is attributable to this kind of issue.

[dm0rg8n]

That is great to hear!

Thank you for getting back to me. :)

[dm0rg8n]

Hi Josh,

Thanks for all of your advice on this matter. I was wondering if you could give me some advice on saving data incrementally via the MySQL PHP pipeline. Currently the script is operating as you propose in your helper, but the data saves at the end of the experiment. I see from another post that it is possible to save data incrementally:

jsPsych.init({ ..., on_data_update: function(data){ var trial_json_result = JSON.stringify(data); jatos.appendResultData(trial_json_results); } });

I am wondering if this is also possible using the MySQL PHP pipeline you have provided and whether I could just run the function to send the data which you provide in the helper each time the data updates. I was also wondering would this send duplicates of the data from previous trials? For example, can I just save the data from the current trial on_data_update instead of repeatedly sending the data associated with preceding trials?

Thank you in advance for your help.

Best,
David

[jodeleeuw]

Hi David,

You can send data incrementally with the MySQL PHP pipeline. The on_data_update callback only includes the most recent data in the data parameter, so if you send just that object to the server it will just append the most recent data to the database.

[dm0rg8n]

Hi Josh,

Apologies for the slow response. I ended up adding this to the code and it appears to work just fine - data outputs in our server as we would expect:

on_data_update: function(data) { var trial_json_result = {0:data}; serverComm.save_data(trial_json_result); }

However, on one occasion (out of many test runs) one of our trials was missing from the data, do you think it could be attributable to the code above, specifically the second line? (I added the second line so that the php script could receive the data as a key:value pair) Or might it be attributable to the issue we discussed earlier regarding multiple inputs to the MySQL backend? Just wanted to double check what it might be, since on most occasion the data came in just fine, and I suspect something might have gone wrong on the users side but figured I'd pop this message in here anyway.

Thanks again,
David

[dm0rg8n]

Hi @jodeleeuw and @vijaymarupudi ,

Thanks for your responses I really appreciate it, I will try out @jodeleeuw's response as we will ideally need the data in separate columns.

Thanks again! :D