Bug saveDataLastPage

[BastienBlain]

Hi there,
I thank you very much for having developed this experience sampler - it really is easy to use the documentation to customise it. It's fantastic!

I have been struggling with the saving for a full week though.
I followed the template and the github instruction to use the google solution to save the data.
However, there are (at least) two problems in my case:
(1) a bug when calling the saveDataLastPage:

Uncaught TypeError: Illegal invocation
at e (jquery-2.1.3.min.js:4)
at Ac (jquery-2.1.3.min.js:4)
at Function.n.param (jquery-2.1.3.min.js:4)
at Function.ajax (jquery-2.1.3.min.js:4)
at Object.saveDataLastPage (index.js:571)
at Object.renderLastPage (index.js:400)
at Object.recordResponse (index.js:538)
at HTMLButtonElement. (index.js:286)
at HTMLButtonElement.dispatch (jquery-2.1.3.min.js:3)
at HTMLButtonElement.r.handle (jquery-2.1.3.min.js:3)

(2) no new sheet is created .

There is no bug when I run the google app script. I created a new google app script with variables I defined in it: it perfectly works to create a new sheet or to write some content in the database sheet.

If I create manually a sheet with the ID name, and if I write 'get' instead of 'post' as a type in the saveDataLastPage, there is no bug reported in the chrome console, and data are written in the google sheet. But I still need that a new ID sheet is automatically created.

I noticed that the functions to save the data are different in the example script (using 'get' and not 'post', which I think is not writing content, and using the JSON function on localStore), but I think that these are for another purpose.

Any idea of what the issue could be?

Thanks a lot!
Bastien

[BastienBlain]

EDIT: I noticed that a similar issue was pointed in the closed request, and trying "data: JSON.parse(JSON.stringify(localStore))," instead of just localStore helped in not having an error message but not to save the data.

No new sheet is created.

This happens for both the app on my phone and on the emulator.

Any idea?

[JamesNewey]

Can you post your saveDataLastPage method and any other pertinent code?

[BastienBlain]

Thanks for helping!

This is the code for saveDataLastPage (I tried a bunch of versions, but here is one):

 
saveDataLastPage:function() {
    $.ajax({
           type: 'post',
           url: 'https://script.google.com/macros/s/AKfycbzrzTylgE2xh_IawGLDhThJvFyXZj7rOIN0g7Hb2AKijFMKbtI/exec',
           data: JSON.parse(JSON.stringify(localStore)),
           crossDomain: true,
           timeout: 120000,
           success: function (result) {
           var pid = localStore.participant_id, snoozed = localStore.snoozed, uniqueKey = 
  localStore.uniqueKey;//, pause_time=localStore.pause_time
           localStore.clear();
           localStore.participant_id = pid;
           localStore.snoozed = snoozed;
	    localStore.uniqueKey = uniqueKey;
           localStore.pause_time=pause_time;
           $("#question").html("<h3>Your responses have been recorded. Thank you for completing this survey.</h3>");
           },

           error: function (request, error) {
			     console.log(error);
                $("#question").html("<h3>Please try resending data. If problems persist, please contact the researchers (uoft.dailylifestudy@gmail.com).</h3><br><button>Resend data</button>");
               $("#question button").click(function () {app.saveDataLastPage();});
				}
           });
},

Here is my google script (again, I tried different versions). I used doGet(e) as in the script example, but I read on the google script help that actually doPost(e) should be used instead when the type is post (instead of get), but both lead to the same mistake.

function doPost(e) {
  var lock = LockService.getPublicLock();
  lock.waitLock(30000);
  try {
    // the three lines below are opening the corresponding sheet on the seerver side
    var database = SpreadsheetApp.openById("1OcyRErdffeF_ZZI3uDjDxfpRGqRgpHwZnNbGXcHgJME"); //replace this string of numbers & letters with your database ID
    var splicedDataset = SpreadsheetApp.openById("1C8mwtO5d-lQUs_lv3ENgI7V0-pygsgJrdERkyY6Eg3E"); //replace this string of numbers & letters with your spliced database id
    var compliance = SpreadsheetApp.openById("1KF16fiF9c9Z-ILhr0GRFXtQisXR4tMKMAqw_pbbgfUw"); //replace this string of numbers & letters with your compliance database id
    // get the participant id from the app (i.e., from the js script)
    var PID = e.parameters['participant_id'];
    // this is "Returning a sheet with the given name": it checks whether a sheet exist for that id
    var sheet = database.getSheetByName(PID);
    var splicedSheet = splicedDataset.getSheetByName(PID);
    // if there is no sheet for that ID
    if (database.getSheetByName(PID) == null){
    // returns the number of sheets in this spreadsheet 
      var sheetIndex = database.getNumSheets()+1;
      // inserts sheet at the end with the relevant ID as a name
      sheet = database.insertSheet(PID, sheetIndex).setName(PID);
      splicedSheet = splicedDataset.insertSheet(PID, sheetIndex).setName(PID);
      var complianceSheet = compliance.getSheets()[0];
      var complianceNewRow = complianceSheet.getLastRow()+1;
      var today = new Date();
      complianceSheet.getRange(complianceNewRow,1).setValue(PID);
      complianceSheet.getRange(complianceNewRow,4).setValue(today);
      var endYear = today.getFullYear(), endMonth = today.getMonth();
      //the number of days you add is equal to the number of days you collect data. If you choose to use the compliance checker, add 2 to the number of 
      //data colsection days
      var endDate = today.getDate() + 9;
      var daysAfterDataCollection = new Date(endYear, endMonth, endDate);
      complianceSheet.getRange(complianceNewRow,5).setValue(daysAfterDataCollection);
      //this code will delete the extra columns in each sheet so can write more data to each spreadsheet
      //Each spreadsheet can only have a maximum of 2 million cells
      sheet.deleteColumns(3, 24);
      splicedSheet.deleteColumns(6, 21);
    } 
    // Now, it's time to write in the sheet.
    // first, get the last row with some content, add one for writing in the new raw. 
    var newRow = sheet.getLastRow()+1;
    // this is looping through the parameters in e, each parameter is written in a new row
    for (var key = 0; key < Object.keys(e.parameters).length; key++){ 
      // column 1
      sheet.getRange(newRow, 1).setValue(Object.keys(e.parameters)[key]);
      splicedSheet.getRange(newRow, 1).setValue(Object.keys(e.parameters)[key]);
      sheet.getRange(newRow, 2).setValue(e.parameters[Object.keys(e.parameters)[key]]);
      splicedSheet.getRange(newRow, 4).setValue(e.parameters[Object.keys(e.parameters)[key]]);
      newRow = newRow + 1;
    }
    //create a log of what is going on
    var doc = DocumentApp.openById("1yekZ77SklQaXE69YTq-5pOqpkEfm4PAZXTnSkJcYD6o");
    var body = doc.getBody();
    body.appendParagraph(Logger.getLog());
    return HtmlService.createHtmlOutput(Logger.getLog());
    return ContentService.createTextOutput(JSON.stringify({"result":"success", "row": nextRow})).setMimeType(ContentService.MimeType.JSON);
  } catch(e) {
    //if error return this
    return ContentService.createTextOutput(JSON.stringify({"result":"error", "error": e})).setMimeType(ContentService.MimeType.JSON);
  } finally {
    lock.releaseLock();
  }
}

[JamesNewey]

There's nothing obviously wrong with that code. From your description of how far you've gotten it sounds like the basic mechanics of posting data are working okay, but that something is going amiss with the data before it is posted, perhaps participant_id isn't getting set correctly.

I suggest you run the app in the browser (ie using cordova to emulate andriod in the browser) and use the browsers debugger to see what the code is doing. You need to put a breakpoint in the saveDataLastPage and see what is being posted.

[BastienBlain]

Thanks for the advice, I will try that.
When I inspect the code in chrome though, it seems that the data are stored properly in local store:
Storage {snoozed: "0", weekendSleepTime: "17:33", participant_id: "Bastien", 1590615162169_goodEvent1_2020_4_27_17_33_28_381: "Football", 1590615162169_snooze_2020_4_27_17_33_19_772: "1", …} 1590615162169_frequencyThermometer_2020_4_27_17_39_3_191: "88" 1590615162169_goodEvent1_2020_4_27_17_33_28_381: "Football" 1590615162169_snooze_2020_4_27_17_33_19_772: "1" 1590615162169_timeEvent1_2020_4_27_17_38_57_808: "01-06-2020" length: 10 participant_id: "Bastien" snoozed: "0" weekdaySleepTime: "17:33" weekdayWakeTime: "05:32" weekendSleepTime: "17:33" weekendWakeTime: "05:33" __proto__: Storage
I feel that something is wrong in the data format so that the google script cannot find the ID, but I did not uncover a reason of why this happens.

[BastienBlain]

Using breakpoints helped me finding the two following potential issues, but I don't think they are linked to the saving issue.

First potential issue: uniqueKey is not defined during the first run.

In renderLastPage (for the first run), uniqueKey is still defined but is not stored in localStore. Here is how localStore looks like:

1.	Storage {1590619551728_dreadThermometer_2020_4_27_18_50_53_245: "92", snoozed: "0", weekendSleepTime: "18:47", participant_id: "Bastien", 1590619551728_savouringThermometer_2020_4_27_18_50_50_61: "90", …}
1.	1590619551728.completed_completedSurvey_2020_4_27_18_50_53_246: "1"
2.	1590619551728_dreadThermometer_2020_4_27_18_50_53_245: "92"
3.	1590619551728_frequencyThermometer_2020_4_27_18_50_38_930: "50"
4.	1590619551728_goodEvent1_2020_4_27_18_48_56_590: "football"
5.	1590619551728_savouringThermometer_2020_4_27_18_50_50_61: "90"
6.	1590619551728_snooze_2020_4_27_18_48_29_948: "1"
7.	1590619551728_timeEvent1_2020_4_27_18_50_29_701: "01-01-2021"
8.	length: 13
9.	participant_id: "Bastien"
10.	snoozed: "0"
11.	weekdaySleepTime: "18:47"
12.	weekdayWakeTime: "06:47"
13.	weekendSleepTime: "18:47"
14.	weekendWakeTime: "06:47"
15.	__proto__: Storage

When app.saveDataLastPage() is run, and while being in saveDataLastPage, uniqueKey is no longuer defined and uniqueKey is missing. Any breakpoint wherever uniqueKey is defined did not pause the code.

Therefore, after being cleared, localStore looks like this:
Storage {snoozed: "0", participant_id: "Bastien", uniqueKey: "undefined", length: 3}

In the subsequent app launches however, uniqueKey is defined in localStore:

1.	Storage {uniqueKey: "1590620615893", 1590620615893_timeEvent1_2020_4_27_19_3_59_37: "02-04-2021", snoozed: "0", participant_id: "Bastien", 1590620615893_frequencyThermometer_2020_4_27_19_4_1_271: "87", …}
1.	1590620615893_frequencyThermometer_2020_4_27_19_4_1_271: "87"
2.	1590620615893_goodEvent1_2020_4_27_19_3_50_309: "shopping"
3.	1590620615893_savouringThermometer_2020_4_27_19_4_4_852: "26"
4.	1590620615893_snooze_2020_4_27_19_3_39_288: "1"
5.	1590620615893_startTime_2020_4_27_19_3_35_893: "1"
6.	1590620615893_timeEvent1_2020_4_27_19_3_59_37: "02-04-2021"
7.	length: 9
8.	participant_id: "Bastien"
9.	snoozed: "0"
10.	uniqueKey: "1590620615893"
11.	__proto__: Storage

In saveDataLastPage, after localStore is cleared:
Storage {snoozed: "0", participant_id: "Bastien", uniqueKey: "1590620615893", length: 3}

Firstly, I guess that this discrepancy between the first run and the second run (and the subsequent ones) is probably due to app.init in onDeviceReady that generates uniqueKey’s content. The break points were not effective and I suppose that this is because it’s not possible to inspect the code before the first question is asked (and after app.init was called). Secondly, I think that this has nothing to do with the saving issue, because even when uniqueKey is available, the data are not saved.

Second potential issue: In terms of error messages, I only saw the following:

Deprecation warning: moment.langData is deprecated. Use moment.localeData instead.
Error
    at Function.langData (file:///android_asset/www/js/plugin/moment-with-locales.min.js:1:7002)
    at Combodate.fillCommon (file:///android_asset/www/js/plugin/combodate.js:146:57)
    at Combodate.fillDay (file:///android_asset/www/js/plugin/combodate.js:162:30)
    at Combodate.fillCombo (file:///android_asset/www/js/plugin/combodate.js:126:32)
    at Combodate.initCombos (file:///android_asset/www/js/plugin/combodate.js:111:22)
    at Combodate.init (file:///android_asset/www/js/plugin/combodate.js:50:18)
    at new Combodate (file:///android_asset/www/js/plugin/combodate.js:31:14)
    at HTMLInputElement.<anonymous> (file:///android_asset/www/js/plugin/combodate.js:462:49)
    at Function.each (file:///android_asset/www/js/plugin/jquery-2.1.3.min.js:2:2882)
    at n.fn.init.each (file:///android_asset/www/js/plugin/jquery-2.1.3.min.js:2:847)

which I don’t think is the source of the problem, but at that stage I have no clue.

Any clues?

[JamesNewey]

It's good that you've made some progress, but strange about the breakpoints not being hit. I would suggest as a fallback using console.log() to monitor variable values, this will output to the console tab in the developer toolbar.

For example:

var testVariable = 1; console.log('value at start', testVariable); // more code.... console.log('value after code', testVariable);

Moment is the date helper library and it's just giving a warning about a function being called that is depreciated (but still working) so I wouldn't worry about it.

[BastienBlain]

Thanks. I will try that. Do you have a suggestion of what I would be looking for though?

[JamesNewey]

That's to try and find at what point uniqueKey is going missing, so put it in methods where you can't hit breakpoints to see whats going on.

I also notice that you've got participant_id twice in the list, I'm not sure if that's an artifact of copy-and-paste or it's really there twice, which would be an issue... I'd recommend checking what is being posted to google. In Chrome you can do this in the Dev toolbar, Network tab, select the network connection to script.google.com and then scroll down on the right until you see Form Data (if you're POSTing, otherwise parameters will be in the query string).

[BastienBlain]

For the first point, for the first run, uniqueKey is defined in renderLastPage until app.saveDataLastPage() is called, and not after and it is never stored in localStore an any stage.

I'm checking right now what is posted to google.

[BastienBlain]

Regarding the second point, this is what I got sent to the network:

Request URL: https://script.google.com/macros/s/AKfycbzrzTylgE2xh_IawGLDhThJvFyXZj7rOIN0g7Hb2AKijFMKbtI/exec
Request Method: POST
Status Code: 200 
Remote Address: 216.58.204.14:443
Referrer Policy: no-referrer-when-downgrade
alt-svc: h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
cache-control: no-cache, no-store, max-age=0, must-revalidate
content-encoding: gzip
content-security-policy: script-src 'report-sample' 'nonce-DVsJzXj6tUEV8+Aw8BfChg' 'unsafe-inline' 'unsafe-eval' 'strict-dynamic' https: http:;object-src 'none';base-uri 'self';report-uri /cspreport
content-type: text/html; charset=utf-8
date: Thu, 28 May 2020 14:29:02 GMT
expires: Mon, 01 Jan 1990 00:00:00 GMT
pragma: no-cache
server: GSE
status: 200
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
:authority: script.google.com
:method: POST
:path: /macros/s/AKfycbzrzTylgE2xh_IawGLDhThJvFyXZj7rOIN0g7Hb2AKijFMKbtI/exec
:scheme: https
accept: */*
accept-encoding: gzip, deflate
accept-language: en-US,en;q=0.9
content-length: 510
content-type: application/x-www-form-urlencoded; charset=UTF-8
origin: file://
user-agent: Mozilla/5.0 (Linux; Android 10; Samsung Galaxy S10 (preview) Build/QQ1D.200105.002; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36
x-requested-with: com.ExpectationsSamplerv0.name
1590675950889_goodEvent1_2020_4_28_10_26_3_317: football
uniqueKey: 1590675950889
1590675950889_savouringThermometer_2020_4_28_10_28_51_276: 91
1590675950889_timeEvent1_2020_4_28_10_28_45_560: 01-01-2021
snoozed: 0
participant_id: Bastien
1590675950889_startTime_2020_4_28_10_25_50_889: 1
1590675950889.completed_completedSurvey_2020_4_28_10_28_53_309: 1
1590675950889_dreadThermometer_2020_4_28_10_28_53_307: 18
1590675950889_snooze_2020_4_28_10_25_55_117: 1
1590675950889_frequencyThermometer_2020_4_28_10_28_48_621: 88

It looks good to me.

[JamesNewey]

What does it say in the Google Apps Script, My Executions page?

https://script.google.com/home/executions

[BastienBlain]

Last execution is the following:

DataAnticipationSamplerv3 | doPost | Éditeur | 26 mai 2020 à 16:46:25 | 0.261 s | Terminée

suggesting that the script was not executed.

[JamesNewey]

In English I get Completed (in black) or Failed (in red), what colour is it?

[BastienBlain]

Sorry: terminée meant completed. It is in black.
When it was executed, it was completed (but nor new sheet was created).
However today's run was not executed (I do not see it in the list).