SIM900 throws error after resetting and re-connecting.

[espruino-discuss3]

Posted at 2016-07-10 by sp33c

Excuse me this issue may not be SIM900 related, which performs quite good with the module... see my fix below:

Whenever I reset my SIM900 (switch it totally off) and would run a JSON-Request after reset it throws this kind of error, even I am online:

req start
Closed: undefined
Uncaught SyntaxError: Got ID:total_documents expected ':'
at line 3 col 3
{"total_documents":0,"result"l_documents":0,"result":""}:""}
^
in function called from system

However when I first doing a request with a non JSON result say the hello-request with a text result. The consecutive calls with JSON will NOT fail.

If someone has a hint, please tell me. kind regards, Alex

[espruino-discuss3]

Posted at 2016-07-10 by @gfwilliams

Sounds strange - The {"total_documents":0, ... stuff sounds like that's the JSON response to your request? If so, could it be that you're calling setTimeout, setInterval, or some function that expects a callback with that string as an argument?

Basically Espruino can often take a string or a function as an argument to callbacks - usually you'd just have something executable in that string so it's fine, but if you put something like JSON in it, it will fail. If you put "Hello World" in, that will work, because it's valid JavaScript, but {key:value} will fail as it's not valid (afaik surrounding it with brackets is valid: ({key:value}))

It'd be an idea to figure out what variable contains that data, and then go through your code and figure out where it is used.

---

Posted at 2016-07-11 by sp33c

Of course I do deserializing pass an object to the callback : - >

---

Posted at 2016-07-11 by sp33c

I try one guess... Could it be related to the fact, I applied a success callback on my own func within getIP (SIM900). This callback does not get executed in the SIM900 before the final statement?

---

Posted at 2016-07-12 by sp33c

@gfwilliams I reset the SIM900 with a relais. I really switch it off and back on. After that the uplink (refresh IP) works... but not the requests. they will fail with above errors. Should I initialize something I forgot about?
When is the gprs.reset() to be used? I could not bring my SIM900 online if I only use reset with callback. I really had to restart it by withdrawing power and put it back on.

---

Posted at 2016-07-12 by @gfwilliams

Is there a reset pin on your SIM900? Without it, gprs.reset() may not be as good at resetting (if SIM900 was in the middle of receiving block of data when Espruino reset).

But I don't think that is your problem... I don't think it's really SIM900 related at all - it looks to me like your code is passing non-executable code into a function that expects a callback. If you could post your code up here, we could take a look for you?

---

Posted at 2016-07-12 by sp33c

I have an SIM900 wich is Arduino compatible. As I know the Arduino has a RST-Pin I connected the appropriate SIM900 one to B4 on espruino.

---

Posted at 2016-07-12 by sp33c

All I do is run the below calls .... one after another it works as expected over 1.5 hour ... even chaining them via callback.

Then once I start resetting the SIM900 (my own reset) wich completely switches the SIM900 off and back on.

The first request (getJobs) will fail repeatedly.

// template for this device
var device1 = {
    protocol: "v1",
    checksum: "",
    device: "defaultDevice@xyz",
    at: "now",
    data: {}
};

function post(template, payload, fn, fne) {
  template.data = payload;
  content = JSON.stringify(template);
  var options = {
    host: 'api.carriots.com',
    path: '/status',
    port: '80',
    method:'POST',
    headers: {
      "carriots.apiKey": "aaaaaaaaaaaaaaaaa",
      "Content-Type": "application/json",
      "Content-Length": content.length
    }
  };
  var req = require("http").request(options, function(res)  {
    var d = "";
    res.on('data', function(data) { d+= data; });
    res.on('error', function(data) {
      console.log('error', data);
      if (fne){
        return fne();
      }
    });
    res.on('close', function(data) {
      var obj = JSON.parse(d);
        if (fn){
          return fn(obj);
        }
    });
  }).end(content);
}

the calls are done like:

function getJobs(fn, fne) {
  var options = {
    host: 'api.carriots.com',
    path: '/streams/?device=Commander@mydevice',
    port: '80',
    method:'GET',
    headers: {
      "carriots.apiKey": "key",
      "Content-Type": "application/json"
    }
  };
  var req = require("http").request(options, function(res)  {
    var d = "";
    res.on('data', function(data) { d+= data; });
    res.on('error', function(data) {
      console.log('error', data);
      if (fne){
        return fne();
      }
    });
    res.on('close', function() {
      var obj = JSON.parse(d);
      if (fn){
        return fn(obj);
      }
      console.log('req closed');
    });
  }).end();
}

---

Posted at 2016-07-12 by sp33c

ups... I forgot to post the requests...

1.post(device, {xyz: 1})
2.getJobs()

I chain it:
getJobs(function(){ post(.....) }, function(){ // error })

---

Posted at 2016-07-12 by @gfwilliams

And what exact messages do you get on the console? The messages you posted previously started like this:

req start
Closed: undefined

But you don't have those print statements in the code you posted up.

As it happens after Closed: undefined, it might be worth adding more print statements so you can get a better idea when the error is actually happening

---

Posted at 2016-07-13 by jonreid

Hi Gordon, as you know I have made a bunch of changes to my own SIM900 module. Most of these changes you probably wont want in the standard library but there are a couple of important bug fixes here and there for the original code. There is one bug in the receiveHandler which can cause corruption depending on the data framing.

Shall I send you my current file so you can see my fixes ?

---

Posted at 2016-07-18 by @gfwilliams

Sorry for the delay - yes, that'd be great! If you're ok with sharing it I'd love to be able to pull your changes back.

Potentially you could create a second module that you used with require(mymodule)(gsm) that actually added your extra functions onto the existing module - so you kept up to date if there were more changes in the future.

---

Posted at 2016-07-28 by sp33c

I rewritten things entirely from scratch.... also with use of promises.
However, the error still occurs...
My question is whether the SIM900 receives the response string as a whole?
In the JSON response I see a AT-Command-Response (+RECEIVE,0,940:) in middle of the string.
Can this be a problem? I do not see the end of the JSON-String, so my assumption is the missing characters were subtracted by the length of the AT-Response in middle.
Can I fix this myself?

$http(sampleGetReq).then(function(data){
: // success
: console.log('success', data);
:});
={ }

OK

0, CONN
ECT OK

0, S
END OK

+RECEIVE,0,
1380:
HTTP/1.1 200 OK
Date: Thu, 28 Jul 2016 22:25:00 GMT
Content-Type: application/vnd.carriots.v2+json; charset=utf-8
Connectio
n: close
Server: Carriots REST API
Cache-Control: no-cache
Allow: GET, POST
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: Date,Server,Allow,Connection,Content-Length,Content-Type,Cache-Control
{"total_documents"
:6,"result":[{"_id":"579842a25c5d7550425376e3","protocol":"v2","checksum":"","_t":"str","at":1469596322,"device":"[email protected]","data":{"time":1,"cycle":1},"id_developer":"185c0dde8b7c9df29d122b25d85fce1c26ad7645ed021f5c611fe4115132f4f8@afitterling.afitterling","created_at":1469596322,"owner":"afitterling"},{"_id":"579842ae5c5d752b45537679","protocol":"v2","checksum":"","_t":"str","at":1469596334,"device":"[email protected]","data":{"time":1,"cycle":1},"id_developer":"6c49c7f8cae5845cacd060ff93edb189517f691a47dbe4817a831869d3f
[email protected]","created_at":1469596334,"owner":"afitterling"},{"_id":"579842b85c5d75e1425376cc","protocol":"v2","checksum":"","_t":"str","at":1469596344,"device":"[email protected]","data":{"time":1,"cycle":1},"id_developer":"91df2481ae379b2c25fc485c6de961efea091aab3a4823853a17a22cd1feac52@afitterling.afitterling","created_at":1469596344,"owner":"afitterling"},{"_id":"579842d
+RECEIVE,0,940:
95c5d7599295379f1","protocol":"v2","checksum":"","_t":"str","at":1469596377,96387,"d
evice":"[email protected]","data":{"timex":1,"cylce":1},"id_developer":"20ed81b80c70b058cff3d1f339d8558cc00fc1dbf32fd193bde61b83c0185f72@afitterling.afitterling","created_at":1469596387,"owner":"afitterling"},{"_id":"579842f05c5d7550425376ea","protocol":"v2","checksum":"","t":"str","at":1469596400,"device":"[email protected]","data":{"timex":1,"cylce":1},"id_developer":"a281efa90907e5e8ff7f6a4fe1afadcd23e746a717a8b7610ab459f8ab538258@afitterling.afitterling","created

0, CLOS
ED

the next call then will get stuck then.

kind regards,
Alex

---

Posted at 2016-07-29 by sp33c

my carriots module is here:
https://github.com/afitterling/Espruino-Modules/blob/develop/carriots.js

---

Posted at 2016-07-29 by @gfwilliams

Strange - I haven't seen that before. After RECEIVE,0,940 there should be 940 characters - but I don't see that at all.

Can you check E.getErrorFlags() when this happens? It's possible it is running slowly enough that characters can't be processed and are getting lost. In fact if you have other code that takes a long time to execute then this could be the problem with the original code as well.

Can you try using gsm.at.debug() and dumping the output from that? I'm not quite sure how you're generating what I see in the dump above?

---

Posted at 2016-07-29 by sp33c

yes, I will try that.

---

Posted at 2016-07-29 by sp33c

Hi Gordon, thanks again for you support. I have only limited time today. Only, if you have time.
I am looking forward for your expert statement.

>requestsNew()
=undefined
>
 
] "\r\n" <--- "\r\n"
10.228.192.245
 
] "10.228.192.245\r\n" <--- "10.228.192.245\r\n"
{
  "host": "api.carriots.com",
  "path": "/streams/?_t=str&[email protected]",
  "port": "80",
  "method": "GET",
  "headers": {
    "Content-Type": "application/json",
    "carriots.apiKey": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "Content-Length": 0 }
 } null
["AT+CIPSTART=0,\"TCP\",\"api.carriots.com\",80\r\n"
>
OK
] "\r\nOK\r\n" <--- "\r\nOK\r\n"
>
0, CONNEC
] "\r\n0, CONNEC" <--- "\r\n0, CONNEC"
T OK
] "0, CONNECT OK\r\n" <--- "T OK\r\n"
>
>
] "\r\n> " <--- "\r\n> "
>
0, SEND O
] "\r\n0, SEND O" <--- "\r\n0, SEND O"
K
] "0, SEND OK\r\n" <--- "K\r\n"
>
+RECEIVE,
] "\r\n+RECEIVE," <--- "\r\n+RECEIVE,"
DEBUG: +RECEIVE,
0,1380:
HTTP/1.1 200 OK
Date: Fri, 29 Jul 2016 07:46:50 GMT
Content-Type: application/vnd.carriots.v2+json; charset=utf-8
Connection: cl
] "+RECEIVE,0,1380:\r\nHTTP/1.1 200 OK\r\nDate: Fri, 29 Jul 2016 07:46:50 GMT\r\nContent-Type: application/vnd.carriots.v2+json; charset=utf-8\r\nConnection: cl" <--- "0,1380:\r\nHTTP/1.1 200 OK\r\nDate: Fri, 29 Jul 2016 07:46:50 GMT\r\nContent-Type: application/vnd.carriots.v2+json; charset=utf-8\r\nConnection: cl"
DEBUG: +RECEIVE,0,1380:
HTTP/1.1 200 OK
Date: Fri, 29 Jul 2016 07:46:50 GMT
Content-Type: application/vnd.carriots.v2+json; charset=utf-8
Connection: cl
ose
Server: Carriots REST API
Cache-Control: no-cache
Allow: GET, POST
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: Date,Server,Allow,Connection,Content-Length,Content-Type,Cache-Control
{"total_documents":3,"result":[{"_id":"579b07af5c5d750a2e53769a","protocol":"v2","checksum":"","_
] "+D,0,1249:ose\r\nServer: Carriots REST API\r\nCache-Control: no-cache\r\nAllow: GET, POST\r\nAccess-Control-Allow-Origin: *\r\nAccess-Control-Allow-Credentials: true\r\nAccess-Control-Expose-Headers: Date,Server,Allow,Connection,Content-Length,Content-Type,Cache-Control\r\n\r\n{\"total_documents\":3,\"result\":[{\"_id\":\"579b07af5c5d750a2e53769a\",\"protocol\":\"v2\",\"checksum\":\"\",\"_" <--- "ose\r\nServer: Carriots REST API\r\nCache-Control: no-cache\r\nAllow: GET, POST\r\nAccess-Control-Allow-Origin: *\r\nAccess-Control-Allow-Credentials: true\r\nAccess-Control-Expose-Headers: Date,Server,Allow,Connection,Content-Length,Content-Type,Cache-Control\r\n\r\n{\"total_documents\":3,\"result\":[{\"_id\":\"579b07af5c5d750a2e53769a\",\"protocol\":\"v2\",\"checksum\":\"\",\"_"
t":"str","at":1469777839,"device":"[email protected]","data":{"This":"Is a JSON example","Carriots":{"text":"Hello World","number":123,"cool":true}},"id_developer":"20bab0871682e6f1f2724a2753a83c3b959af6088165e2abf88dfa045ebac8a0@afitterling.afitterling","created_at":1469777839,"owner":"afitterling"},{"_id":"579b07b85c5d75792d5376c2","protocol":"v2","checksum":"","_t":"str","at":1469777848,"device":"[email protected]","data":{"This":"Is a JSON example","Carriots":{"texfitterli
] "+D,0,900:t\":\"str\",\"at\":1469777839,\"device\":\"[email protected]\",\"data\":{\"This\":\"Is a JSON example\",\"Carriots\":{\"text\":\"Hello World\",\"number\":123,\"cool\":true}},\"id_developer\":\"20bab0871682e6f1f2724a2753a83c3b959af6088165e2abf88dfa045ebac8a0@afitterling.afitterling\",\"created_at\":1469777839,\"owner\":\"afitterling\"},{\"_id\":\"579b07b85c5d75792d5376c2\",\"protocol\":\"v2\",\"checksum\":\"\",\"_t\":\"str\",\"at\":1469777848,\"device\":\"[email protected]\",\"data\":{\"This\":\"Is a JSON example\",\"Carriots\":{\"texfitterli" <--- "t\":\"str\",\"at\":1469777839,\"device\":\"[email protected]\",\"data\":{\"This\":\"Is a JSON example\",\"Carriots\":{\"text\":\"Hello World\",\"number\":123,\"cool\":true}},\"id_developer\":\"20bab0871682e6f1f2724a2753a83c3b959af6088165e2abf88dfa045ebac8a0@afitterling.afitterling\",\"created_at\":1469777839,\"owner\":\"afitterling\"},{\"_id\":\"579b07b85c5d75792d5376c2\",\"protocol\":\"v2\",\"checksum\":\"\",\"_t\":\"str\",\"at\":1469777848,\"device\":\"[email protected]\",\"data\":{\"This\":\"Is a JSON example\",\"Carriots\":{\"texfitterli"
ng"},{"_id":"579b093a5c5d750a2e5376bc","protocol":"v2","checksum":"","_t":"str","at":1469778234,"device":"[email protected]","data":{"This":"Is a JSON example","Carriots":{"text":"Hello W
+RECEIVE,0,187:
orld","number":123,"cool":true}},"id_developer":"4aaed914017f667ecf933be52e147111d3b0b12460c7eff8d41e35b6164b31c6@afitterling.afitterling","created_at":1469778234,"owner":"afitterling"}]}
] "+D,0,387:ng\"},{\"_id\":\"579b093a5c5d750a2e5376bc\",\"protocol\":\"v2\",\"checksum\":\"\",\"_t\":\"str\",\"at\":1469778234,\"device\":\"[email protected]\",\"data\":{\"This\":\"Is a JSON example\",\"Carriots\":{\"text\":\"Hello W\r\n+RECEIVE,0,187:\r\norld\",\"number\":123,\"cool\":true}},\"id_developer\":\"4aaed914017f667ecf933be52e147111d3b0b12460c7eff8d41e35b6164b31c6@afitterling.afitterling\",\"created_at\":1469778234,\"owner\":\"afitterling\"}]}" <--- "ng\"},{\"_id\":\"579b093a5c5d750a2e5376bc\",\"protocol\":\"v2\",\"checksum\":\"\",\"_t\":\"str\",\"at\":1469778234,\"device\":\"[email protected]\",\"data\":{\"This\":\"Is a JSON example\",\"Carriots\":{\"text\":\"Hello W\r\n+RECEIVE,0,187:\r\norld\",\"number\":123,\"cool\":true}},\"id_developer\":\"4aaed914017f667ecf933be52e147111d3b0b12460c7eff8d41e35b6164b31c6@afitterling.afitterling\",\"created_at\":1469778234,\"owner\":\"afitterling\"}]}"
>
0, CL
] "wner\":\"afitterling\"}]}\r\n0, CL" <--- "\r\n0, CL"
OSED
] "0, CLOSED\r\n" <--- "OSED\r\n"
Uncaught Error: Field or method "total_documents" does not already exist, and can't create it on undefined
 at line 1 col 614
..."Carriots":{"texfitterling"},{"_id":"579b093a5c5d750a2e5376b...
                              ^
in function called from system
 at line 208 col 33
                total_docs: data.total_documents
                                ^
in function called from system
>E.getErrorFlags()
=[
  "FIFO_FULL"
 ]
> 

---

Posted at 2016-07-29 by sp33c

I accidentally also put my debug output from the request in their above? Should I take it out.
only the lines:

{
  "host": "api.carriots.com",
  "path": "/streams/?_t=str&device=Commander@afitt­erling.afitterling",
  "port": "80",
  "method": "GET",
  "headers": {
    "Content-Type": "application/json",
    "carriots.apiKey": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx­",
    "Content-Length": 0 }
 } null

---

Posted at 2016-07-29 by sp33c

It is wired once more carriots wants me to send content-length on get....

---

Posted at 2016-07-29 by sp33c

I believe I shouldn't bother about this behavior. I just seen, the carriots io supports the max parameter. So I will get only two at a time. And my code will be fixed. :-)

---

Posted at 2016-07-29 by @gfwilliams

I'm still confused about Uncaught Error: Field or method "total_documents" does not already exist, and can't create it on undefined

But the "FIFO_FULL" flag really is your problem - some part of your program is taking as long time to execute, and during that time the FIFO gets full and characters get dropped.

Which board are you using this on? And you're connecting over USB?

---

Posted at 2016-07-29 by sp33c

I am using the Pico board.
I run it currently over usb but additionally with an external power supply, as the usb connection is only for debugging purpoises. I sometimes pull off usb.

If the document gets parsed by one the total_document is there every time. If however the document getting JSON parsed is only truncated this is missing.
I could successfully reproduce that the error occurs only if a certain treshold of response length is tresspassed. And the response string from the hardware seems truncated.
I looks as if it receives only a limited amount of bytes? But there is only one callback? So my assumption was the callback gets called with an incomplete set of data, that itself will be passed in to the callbacks/promises.

A good thing. As I understood now at least the source of error, I could so far fix it with the max param that carriots provides. I pick only one job at a time. If this works for 7d/24h I'll be glad. My garden is in a remote area very far away. :-)

---

Posted at 2016-07-29 by sp33c

I use the maximum baud rate on the SIM900, can this be a cause?

---

Posted at 2016-07-29 by sp33c

in the above log from the example it looks like that the response string itself handed into JSON.parse is not valid json anymore. It has passages like:

e":"[email protected]­g","data":{"This":"Is a JSON example","Carriots":{"texfitterli
] "+D,0,900:t\":\"str\",\"at\"

So I came to the conclusion the SIM900 mixed it up(?)

---

Posted at 2016-07-29 by @gfwilliams

I use the maximum baud rate on the SIM900, can this be a cause?

What is the baud rate? It definitely won't help... The Pico can buffer around 500 bytes, so it won't take long to fill it at higher rates. I'd definitely try and lower the rate significantly - it should stop the FIFO_FULL messages anyway.

So I came to the conclusion the SIM900 mixed it up(?)

I think the SIM900 is probably fine - it's just that Espruino's buffers are getting full so it's losing the odd character.

---

Posted at 2016-07-29 by jonreid

This could be caused by a bug I found in the SIM900 driver. In the function 'receiverHandler' there is a line of code:
sockData[parms[0]] += line.substr(colon+1,parms[1]);

This needs to be changed to
sockData[parms[0]] += line.substr(colon+3,parms[1]);

I found if the whole response fitted within one handler transfer the orginal code would corrupt the response.

---

Posted at 2016-07-29 by sp33c

Amazinly doing so.. it seems to works! I just put the max=1 param out. As of today a collection length of >3 broke that.
@gfwilliams To the FIFO issue.... may it be a wrong canditate. I was noticing, the E.getErrorStats always returns FIFO_full on any request. But it is going to be erased later on.
Anyway the SIM900 driver is of really great help!

>  $http({
:      host: 'api.carriots.com',
:      path: '/streams/',
:      port: '80',
:      method: 'GET',
:      params: {
:        _t: 'str',
:        device: '[email protected]',
:        max: 10
:      }
:    }).then(function(res){ a=res})
{
  "host": "api.carriots.com",
  "path": "/streams/?_t=str&[email protected]&max=10",
  "port": "80",
  "method": "GET",
  "headers": {
    "Content-Type": "application/json",
    "carriots.apiKey": "xxxxxxxxx",
    "Content-Length": 0 }
 } null
={  }
>
>
OK
>
0, CONNECT OK
>
>
>
>
0, SEND OK
>
+
RECEIVE,0,1380:
HTTP/1.1 200 OK
Dat
e: Fri, 29 Jul 2016 20:55:58 GMT
Content-Type: applicatio
n/vnd.carriots.v2+json; charset=utf-8
Conne
ction: close
Server: Carriots REST API
Ca
che-Control: no-cache
Allow: GET, POST
Access-Con
trol-Allow-Origin: *
Access-Control-Allow-Credentia
ls: true
Access-Control-Expose-Headers: Dat
e,Server,Allow,Connection,Content-Length,Con
tent-Type,Cache-Control
{"total_do
cuments":6,"result":[{"_id":"579bc2385c5d75660953772e","protocol
":"v2","checksum":"","_t":"str","at":1
469825592,"device":"Commander@afitterl
ing.afitterling","data":{"This":"Is a JSO
N example","Carriots":{"text":"Hello World"
,"number":123,"cool":true}},"id_developer":"8c19
07cf2402f422b64069832d07aa47276a6414fd4fdcf558a96f6a9
[email protected]","created_at":1469825592,"o
wner":"afitterling"},{"_id":"579bc2435c5d75a007537738","protocol":"v2
","checksum":"","_t":"str","at":1469825603,"device":"[email protected]
erling","data":{"This":"Is a JSON example","Carriots":{"text":"Hello World","number":123,"cool":
true}},"id_developer":"d98688e13aae2861be6dd0bed33ba44f4d9171dfbe90d4ed64f5c091bb624cf4@afitterling.afitterling","crea
ted_at":1469825603,"owner":"afitterling"},{"_id":"579bc24c5c5d755102537814","protocol":"v2","checksum":"","_t":"str","at":1469825612,"device":"Commande
[email protected]","data":{"This":"Is a JSON example","Carriots":{"text":"Hello W
+RECEIVE,0,1339:
orld","number":123,"cool":true}},"id_developer":"6ddc5cab0331a2505419eb9d08afd74e2c31cc35c4
[email protected]","created_at":1469825612,"owner":"afitterling"},{"_id":"579bc2bc5c5d751b7e537889","protocol":"v2","checksum":"","_t":"str","at":1469825724,"device":"[email protected]","data":{"This":"Is a JSON example","Carriots":{"text":"Hello Wor
ld","number":123,"cool":true}},"id_developer":"c1564b155fa18858c1f608f4db64f359e6539d
[email protected]","created_at":1469825724,"owner":
"afitterling"},{"_id":"579bc2c75c5d75d908537723","protocol":"v2","checksum":"","_t":"str","at":1469
825735,"device":"[email protected]","data":{"This":"Is a JSON example","Carriots":{"text":"Hello World","n
umber":123,"cool":true}},"id_developer":"0a1e227e3914808240d4b29385fcc426f8dca235cce0295db059e41433a81a8a@afitterling.afitterling","created_at":1469825735,"o
wner":"afitterling"},{"_id":"579bc2d25c5d75f87f537843","protocol":"v2","checksum":"","_t":"str","at":1469825746,"device":"[email protected]","data":{"This":"Is a JSON example","Carriots":{"text
":"Hello World","number":123,"cool":true}},"id_developer":"e148352107e64d155ed2f32511afb197717f9d1f0e3e537773d0376ec1d0fec6@afitterling.afitterling","created_at":1469825746,"owner":"afitterling"}]}
>
0
, CLOSED
>a
={ "total_documents": 6,
  "result": [
    {
      "_id": "579bc2385c5d75660953772e",
      "protocol": "v2",
      "checksum": "",
      "_t": "str",
      "at": 1469825592,
      "device": "[email protected]",
      "data": {
"This": "Is a JSON example",
"Carriots": {
  "text": "Hello World",
  "number": 123, "cool": true }
       },
      "id_developer": "8c1907cf2402f422b" ... "rling.afitterling",
      "created_at": 1469825592,
      "owner": "afitterling"
     },
    {
      "_id": "579bc2435c5d75a007537738",
      "protocol": "v2",
      "checksum": "",
      "_t": "str",
      "at": 1469825603,
      "device": "[email protected]",
      "data": {
"This": "Is a JSON example",
"Carriots": {
  "text": "Hello World",
  "number": 123, "cool": true }
       },
      "id_developer": "d98688e13aae2861b" ... "rling.afitterling",
      "created_at": 1469825603,
      "owner": "afitterling"
     },
    {
      "_id": "579bc24c5c5d755102537814",
      "protocol": "v2",
      "checksum": "",
      "_t": "str",
      "at": 1469825612,
      "device": "[email protected]",
      "data": {
"This": "Is a JSON example",
"Carriots": {
  "text": "Hello World",
  "number": 123, "cool": true }
       },
      "id_developer": "6ddc5cab0331a2505" ... "rling.afitterling",
      "created_at": 1469825612,
      "owner": "afitterling"
     },
    {
      "_id": "579bc2bc5c5d751b7e537889",
      "protocol": "v2",
      "checksum": "",
      "_t": "str",
      "at": 1469825724,
      "device": "[email protected]",
      "data": {
"This": "Is a JSON example",
"Carriots": {
  "text": "Hello World",
  "number": 123, "cool": true }
       },
      "id_developer": "c1564b155fa18858c" ... "rling.afitterling",
      "created_at": 1469825724,
      "owner": "afitterling"
     },
    {
      "_id": "579bc2c75c5d75d908537723",
      "protocol": "v2",
      "checksum": "",
      "_t": "str",
      "at": 1469825735,
      "device": "[email protected]",
      "data": {
"This": "Is a JSON example",
"Carriots": {
  "text": "Hello World",
  "number": 123, "cool": true }
       },
      "id_developer": "0a1e227e391480824" ... "rling.afitterling",
      "created_at": 1469825735,
      "owner": "afitterling"
     },
    {
      "_id": "579bc2d25c5d75f87f537843",
      "protocol": "v2",
      "checksum": "",
      "_t": "str",
      "at": 1469825746,
      "device": "[email protected]",
      "data": {
"This": "Is a JSON example",
"Carriots": {
  "text": "Hello World",
  "number": 123, "cool": true }
       },
      "id_developer": "e148352107e64d155" ... "rling.afitterling",
      "created_at": 1469825746,
      "owner": "afitterling"
     }
   ]
 }
> 

---

Posted at 2016-07-29 by sp33c

I have to test this over a long period of time. Thanks @jonreid if this works!
The FIFO issue also sounded plausible to me. It may be an issue. As @gfwilliams knows his module.

---

Posted at 2016-07-29 by sp33c

I think @gfwilliams is right:

"FIFO_FULL",
"LOW_MEMORY"
       ]
     }
   }
 }
Uncaught SyntaxError: Got ID:total_documents expected ','
 at line 3 col 3
{"total_documents":3,"result":[{"_id":"579bc6ad5c5d751b7e537...
  ^
in function called from system

---

Posted at 2016-07-29 by jonreid

My fix is for responses that are small in size - so you still need that.

If you are getting low memory errors you need to process your data in smaller chunks ie either optimise your json or request it in smaller chunks.

You never want to get into a low memory scenario as things start breaking pretty fast. It is worthwhile monitoring 'process.memory().free' at various parts of your code to highlight issues.

In my project - if I ever get an 'out of memory' error (which you should code to prevent) I reboot otherwise the code operation will be in an unknown state, also use the new watchdog feature.

---

Posted at 2016-07-30 by @allObjects

@sp33c, I asume that the data you published in this forum is test data and therefore, my conclusion may not be the answer to the payload capacity issues... but I still want to mention it.

JSON is already much leaner than XML, but it is still 'fat', especially for repetitive data... and last but not least the lots of double quotes required. JSON is very easy and fast to process, but may be you can afford some CPU cycles to get over the bump. In case 'you know your data' / 'have control over the data', a CSV string is still the most effective to get there.... and furthermore, even having a catalog with ids for values that repeat.

The challenge is now to provide what xml does best and JSON about equally as good: provide the meta data (reference) about the individual data/information elements, and the structure...

The first one 'is easy' and is applicable if you have a number of defined data structures: first or last data item includes the id to the definition of the data structure.

The second one is when the values are sparse and variable in distribution across the overall (complete) structure: use the 1st approach - if useful - to identify the main structure, and a second - or second last - value for a bitmap that identifies which attribute (always same sequence) has value... (saves you from sending commas).

When values are repeated, a dictionary is built, passed as first thing in the payload, and then payload-local references describe the content (Some DBs do compression this way in order to not have to run uncompress algorithms).

Further option is to code the values if the values are known beforehand.

---

Posted at 2016-07-30 by sp33c

@jonreid yes, i knew about it and my status post requests are publishing the memory free. may i ask how do you reboot? what command?

hm... my code is not that big, actually at any point there shouldn't be a less memory state. i even taking care about intervals and timeout myself. it might due to memory leaking somewhere else.

---

Posted at 2016-07-30 by sp33c

@allObjects the blownup json is due mainly of infrastructure. carriots is a great iot platform. of course i can write the whole platform myself and being leaner. but i rely on such infrastructure for inter device communication. i like there way. i think the espruino device should be capable of handling such simple things. the productive response data is a bit leaner just carrying two values, cycle (watercycle) and time. also i use the max param to limit the get to one response entry at a time.

carriots doesnt represent a full schema but all you need to be able to handle different devices and logik. one optimization point can be to put the logic into carriots and have not a autonomous device anymore.

the overhead to deal with csv isnt really an advantage.

---

Posted at 2016-07-30 by sp33c

i am experienced with garbage collection issues, it should theoretically never be the case to run out of memory if constantly destroying references my self carefully. also not json is an isue as after processing the objects are nulled.

i may have to rething the overall model of that piece of hardware.

---

Posted at 2016-07-30 by sp33c

I updated my version to get rid of self caused memory leaks, I use the immediate return for timeouts. Thanks to the fix to the SIM900 module it is so far quite stable.
Additionally I set the SIM900 baud rate to 9600. Additionally I pull jobs out of resources with the max=1 parameter, means get only as less data as possible.
Memory seems stable so far.
I put the code online: https://github.com/afitterling/espruino-projects2/blob/master/RemoteWateringSystem.js
The cariots module can now handle wrong JSON and will reject the successful call:
https://github.com/afitterling/Espruino-Modules/blob/develop/carriots.js
this is the result of the status push at call 104.

    {
        "stats": {
            "process": {
                "stackEndAddress": 536959420,
                "flash_binary_end": 376280,
                "flash_code_start": 134234112,
                "free": 2875,
                "flash_length": 393216,
                "usage": 2225,
                "total": 5100,
                "flash_start": 134217728,
                "history": 1753
            },
            "errorFlags": [],
            "requests": 104,
            "errors": 0,
            "resets": 1
        }
    }

---

Posted at 2016-07-31 by sp33c

I just noticed, the SIM900 library is memory leaking, maybe du networkJS.create(netCallbacks).
Upon a repetive call on reconnect old references aren't getting collected with garbage collection, I assume old references are still being kept externally.
Could this issue be due to the networkJS library? Or is it due the SIM900 library only?

---

Posted at 2016-07-31 by sp33c

per reconnect my code is leaking 200 memory units (a 20bytes?)

---

Posted at 2016-07-31 by jonreid

I have given my SIM900 module to Gordon so he can sync in the bug fixes I have made. There are a couple of other basic things required for reliable operation ie in netcallbacks/create - there are more unregisterLine required for error cases and you need to remove the CIPSTART exception throw. Also if the SIM800 CIPSTART fails, the driver wont catch that so I added a timeout which then closes the socket and things are cleaned up properly.

At the app level, you need to ensure that the connect/send/receive process is completely finished before calling again otherwise memory will be consumed - this also means catching errors so the process can unwind properly. Use the request.on('error', function(e) ...

If you are in a low coverage area and the SIM800 can falloff the network there are a whole lot more considerations that come into play - you could just choose the approach of if things go wrong, reset and start all over, but I chose to handle all the states the SIM800 can get into and handle them in a more managed fashion.

Hopefully Gordon will have the time to sync in the basics of my changes to benefit everyone.

---

Posted at 2016-07-31 by sp33c

thanks for the feedback. could you send me your code as well? Do you have any? ([email protected])
I try to sync with gordon, but I am abroad a couple weeks, propably offline.

---

Posted at 2016-08-01 by @gfwilliams

Thanks - I'll try and update this, but maybe not this week - things are a bit busy!

For the memory leak, it's actually most likely that the sockets are still open? You should be able to check in gsm.debug() I think and see a list of active sockets.

---

Posted at 2016-08-02 by sp33c

thank you very much. eventhough i resetted the sockets, the leak was still there. i expect the netcallbacks to leave external references (to the sim900 code) in the networkjs. closures are prone for memory leaks but most websites do not run above 1h, so nobody notices it.

---

Posted at 2016-08-02 by sp33c

i have not closed the sockets. i declared them to null. this may not have been enough. however this may be part of networkjs. i cannot test further i am abroad, the next 4 weeks. thank you for your kind support so far.