Randomly Out of memory errors precedeed by JsVarOverflow

[espruino-discuss]

Posted at 2015-09-18 by asez73

So, I just have a problem which cause is very unclear to me.
There are no memory leakage for hours and it suddenly appears, or most of the time, never happens.
And that's without any change to the source code of course.

Whenever it happens, it is immediately precedeed by Jsvar length problems...
I do not see why it happens, except either some interpreter internal problem or some real time kind of problem.
I suspect a serial buffer overflow as the serial speed is set to 115200 bauds and the ublox NEO-6M sends data in chuncks which can be fairly long (115 chars for PUBX,00 nmea sentence for instance).

So, my code now reduces the types of nmea sentences that the ublox gps device can send. This is a ublox feature which allows to inhibate nmea sentences by sending some specific commands to the ublox. This greatly helps as the error did not showed again since this.

Anyway, following @gordon suggestions about debugging, I had the following uncoherent results.
Specifically, the gps and Serial4 objects should not be that big and even fit to RAM if I don't misunderstand the figures.

>for (var i in global) print(i+" : "+E.getSizeOf(global[i]))
Chrono : 108
ddmmyy : 37
E : 1
secn : 41
lln : 40
cks7 : 27
nmea_checksum : 54
start_stop_nmea : 84
fieldsOfInterest : 31
codeFieldOfInterest : 44
posFieldsOfInterest : 26
getFieldsPos : 8
parseHandlers : 124
estimate_dist : 25
tfTimeout : 1
tf : 66
connect : 183
gps : 1902
Serial : 6
Serial4 : 1903
console : 1
Number : 1
Date : 6
Error : 6
seconds : 1
i : 1
=undefined
>process.memory();
={ "free": 806, "usage": 1444, "total": 2250, "history": 479,
  "stackEndAddress": 536910664, "flash_start": 134217728, "flash_binary_end": 228576, "flash_code_start": 134449152, "flash_length": 262144 }
>

The source code file is joined.

>reset();
=undefined
 _____                 _
|   __|___ ___ ___ _ _|_|___ ___
|   __|_ -| . |  _| | | |   | . |
|_____|___|  _|_| |___|_|_|_|___|
          |_| http://espruino.com
 1v80.346 Copyright 2015 G.Williams
>echo(0);
Handler of GPGGA defined as function (d) {this['hhmmss.ss']=secn(d[1]);this['latitude']=lln(d[2],d[3],'S');this['longitude']=lln(d[4],d[5],'E');this['mslAlt']=Number(d[9])}
Handler of GPRMC defined as function (d) {this['hhmmss.ss']=secn(d[1]);this['status']=d[2];this['latitude']=lln(d[3],d[4],'S');this['longitude']=lln(d[5],d[6],'E');this['sogKn']=Number(d[7]);this['ddmmyy']=ddmmyy(d[9])}
Handler of GPZDA defined as function (d) {this['hhmmss.ss']=secn(d[1])}
Handler of GPGLL defined as function (d) {this['latitude']=lln(d[1],d[2],'S');this['longitude']=lln(d[3],d[4],'E');this['hhmmss.ss']=secn(d[5]);this['status']=d[6]}
Handler of GPVTG defined as function (d) {this['sogKn']=Number(d[5])}
Handler of GPGRS defined as function (d) {this['hhmmss.ss']=secn(d[1])}
Handler of GPGBS defined as function (d) {this['hhmmss.ss']=secn(d[1]);this['std_lat']=Number(d[2]);this['std_alt']=Number(d[4])}
Handler of GPGST defined as function (d) {this['hhmmss.ss']=secn(d[1]);this['std_lat']=Number(d[6]);this['std_long']=Number(d[7]);this['std_alt']=Number(d[8])}
Handler of PUBX00 defined as function (d) {this['hhmmss.ss']=secn(d[2]);this['latitude']=lln(d[3],d[4],'S');this['longitude']=lln(d[5],d[6],'E')}
Handler of PUBX04 defined as function (d) {this['hhmmss.ss']=secn(d[2]);this['ddmmyy']=ddmmyy(d[3])}
Now stopping the GPS from sending the following messages
Time fix from serial at 1442570315.18200206756
GPS position fix from serial at 1442570315.18380546569
Received a TimeFix event with parameter GPRMC,095836.00,A,4540.02704,N,00624.51971,E,0.625,,180915,,,A
Internal clock was only 348.32958984375 ms from GPS epoch Fri Sep 18 2015 09:58:35 GMT+0000 so no changes were made.
We have a GPS position fix!
Stopping GPGSV nmea sentence send by ublox gps with $PUBX,40,GSV,0,0,0,1,0,0*58
Stopping GPGSA nmea sentence send by ublox gps with $PUBX,40,GSA,0,0,0,1,0,0*4F
Stopping GPDTM nmea sentence send by ublox gps with $PUBX,40,DTM,0,0,0,1,0,0*47
Stopping PUBX03 nmea sentence send by ublox gps with $PUBX,40,03,0,0,0,1,0,0*19
Stopping PUBX40 nmea sentence send by ublox gps with $PUBX,40,40,0,0,0,1,0,0*1E
Stopping GPTXT nmea sentence send by ublox gps with $PUBX,40,TXT,0,0,0,1,0,0*42
Now stopping the GPS from sending the following messages
=undefined
Stopping GPZDA nmea sentence send by ublox gps with $PUBX,40,ZDA,0,0,0,1,0,0*45
Stopping GPGLL nmea sentence send by ublox gps with $PUBX,40,GLL,0,0,0,1,0,0*5D
Stopping GPVTG nmea sentence send by ublox gps with $PUBX,40,VTG,0,0,0,1,0,0*5F
Stopping GPGRS nmea sentence send by ublox gps with $PUBX,40,GRS,0,0,0,1,0,0*5C
Stopping GPGST nmea sentence send by ublox gps with $PUBX,40,GST,0,0,0,1,0,0*5A
Stopping PUBX04 nmea sentence send by ublox gps with $PUBX,40,04,0,0,0,1,0,0*1E
Stopping PUBX00 nmea sentence send by ublox gps with $PUBX,40,00,0,0,0,1,0,0*1A
>gps.chrono.info()
674 leaps (mean time 0.01794492031 s). Measured duration/Total duration 5.47%
=undefined
> 

---

Attachments:

• gps_ext12.js

[gfwilliams]

Posted at 2015-09-18 by @gfwilliams

Ahh, ok - so the reason GPS and Serial4 are so big is because they're linking to each other... So I guess Serial4 has a callback that references GPS, and vice versa.

What about trying:

function show(d) {
  for (var i in d) print(i+" : "+E.getSizeOf(d[i]))
}
console.log("--------- gps");
show(gps);
console.log("--------- Serial4");
show(Serial4);

... and maybe drill down with show(gps.whatever); to try and find out which part of them is taking all the memory?

[espruino-discuss]

Posted at 2015-09-18 by @allObjects

Very interesting. When I worked with a ublox 6M moudule, I had irregularities too: sometimes some values in a sentence were missing and lead to errors in conversion to a number. I used the standard Espruino GPS moudule after it was minimally modified to allow to have custom handlers. I did not think of stripping down the output scope in ublox moudule.

Glancing over the GPS_ext12.js code I wondered about the extra arguments in setTimeout() invocation in start_stop_nmea(), ~ line 110. Can you explain?

Btw, I like the function/method-creation approach by using the sentence attribute name as the signature and then have a content type addressed detail handling. The not so liked part is the eval()... but the heck: if code can create programmatic code based on declarations, why not go for it! Did you consider to run the function generation in a preprocess step and output the source, and then use that source explicitely?

[gfwilliams]

Posted at 2015-09-18 by @gfwilliams

I wondered about the extra arguments in setTimeout()

see this - you can now pass a function in, and have setTimeout pass arguments through.

[espruino-discuss]

Posted at 2015-09-18 by @allObjects

@gordon, thanks... I'm obviously damaged by working for too long for too much in lowest-common-(browser-)denominator settings... ;)

With these extra parameters, a plain, naked function reference can be used. If this is needed in it, a myFunction.bind(thisContext) eliminates the need for an extra anonymous functions ...

[espruino-discuss]

Posted at 2015-09-18 by asez73

Well, the eval part is doing programmaticaly what i found cumbersome. The real goal is to minimise any computation done in the serial.on loop and still provide any data of interest in a naturally javascript suitable for further computations and efficient data storage. It's still in progress.
The ublox device can be programmed from its usb interface directly with u-center and saved to its integrated flash memory. This would allow for a no time and no memory used on the espruino.

As a matter of fact, I prefer to pre-process on launch as much as possible and then delete any no more usefull functions or objects. This reduces the dependancy of this program to predefined gps board behaviour.
I found elsewhere something like 90 different nmea sentences...........

About incomplete sentences. from the device, it just drops any outdated sentence even if it is beeing send through the serial link, with no eol, just the $ before the next sentence. I ended with a systematic checksum verification, discarding any unvalid sentence before any further computation.

[espruino-discuss]

Posted at 2015-09-18 by @allObjects

I was thinking about preprocessing even before the launch - I assume you refere with launch to connect and what it does: I separate step, the source generation is run, and it's output is used as part of the .js file. The muting of unwanted sentences - or config which sentences - the gps modules has to send, can still happen when connecting. Doning so does not need the pre-config the device seperately. On the other hand, if memory get's tight, this will be a nice option. I have experienced memory shortages quit quickly, and resorted to similar aproaches: have the config info not in well structured objects, but as single string(s) with comma (or space) separation of the individual values. Processing with a *.split(",") uses only temporary memory. My example was the description of a pac-man board/maze, drawing it on a 320x240 ILI#### controled display , and boundary checking for the 'moving parts', etc... In the end I had even to come up with an encodeing to keep the code/data still readable but using up only one byte.

Finding out 'quickly' what to spend processing time on and what to drop is a nice approach. Do you have figures/stats how much - %-wise - is dropped?

Even with the good data, I had to optimize more: a) only changed data I did render and send to the display, and b) building up of the display is parts by parts to stay 'within the loop' of 1 update/second. Using your approach of 'quieting' the module would have been a great help. Enjoyed very much 'listening' to your code and its style - like a good piece of music!

[espruino-discuss]

Posted at 2015-09-18 by asez73

Yes, you are probably right and I have to understand something like 'this' in event context...

Here is the result of your proposition. Note on the last commands that I failed to list the event handlers defined with gps.on and Serial4.on...
Sorry for the curious display induced by sharp characters in event names.

>    show(gps);
serial : 1943
GPGGA : 1943
GPRMC : 1943
GPGBS : 1943
line : 1
enabled : 1
log : 1
debug : 1
chrono : 127
lagTime : 1
status : 1
hhmmss.ss : 1
 #onGPSFix : 1943
 #ongpsdata : 1943
latitude : 1
longitude : 1
sogKn : 1
ddmmyy : 1
mslAlt : 1
std_lat : 1
std_alt : 1
epoch : 1
 #onTimeFix : 66
=undefined

And

>    show(Serial4);
_baudrate : 1
_options : 5
 #ondata : 1951
=undefined
>show(Serial4.#ondata)
Uncaught SyntaxError: Got '#' expected ID
 at line 1 col 14
show(Serial4.#ondata)
             ^

Digging inside failed due to events handlers names...

>show(Serial4.'#ondata')
Uncaught SyntaxError: Got String:'#ondata'ID expected ID
 at line 1 col 14
show(Serial4.'#ondata')
             ^
>show(Serial4['#ondata'])
=undefined
>show(Serial4[#ondata])
Uncaught SyntaxError: Got '#' expected EOF
 at line 1 col 14
show(Serial4[#ondata])
             ^
>show(gps[#onGPSFix])
Uncaught SyntaxError: Got '#' expected EOF
 at line 1 col 10
show(gps[#onGPSFix])
         ^
>show(gps[onGPSFix])
Uncaught ReferenceError: "onGPSFix" is not defined
 at line 1 col 10
show(gps[onGPSFix])
         ^
>show(gps['onGPSFix'])
=undefined
>show(gps['#onGPSFix'])
=undefined
> 

[espruino-discuss]

Posted at 2015-09-18 by @allObjects

'this' represents the default runtime context from within a function is invoked - event or otherwise. If the function is bound though to an object before invocation with .bind(contextObject), this in the function will refere to the bound contextObject and - according to @gordon - it is even faster then an anonymous function. On the other hand, binding takes away the freedom to use the function on/in/for different contexts (objects). Im talking about simple invocation because .apply() and .call() can help to keep the flexibility and provide different ways to pass the arguments: as array or individually, respectively. All these language elements/constructs help with (meta) data driven code / logic / algorithms.

[espruino-discuss]

Posted at 2015-09-18 by asez73

Yes, I have memory shortages in sight and actually, the main problem seems to be in the events handlers Serial4.on and gps.on. cross referencing each other....
That said, it is fairly simple to execute the functions at launch time, wipe out any thing no more usefull, and save() a completely pre-processed gps object.
The thing is that I want to run the GPS at 10Hz during some circumstances, say a car in motion, and back to 1Hz or 1/3 Hz in others conditions, say a car parked during the last few minutes, and finally physically shut-it off by controlling some power regulator from the STM32, changing that dynamically and resetting the gps parameters on the fly at reboot.

Which is still more a hope than a fact.

About figures, this is the 12th iteration of code.
So now, there are some basic considerations:

1. The predefined setup of the NEO-6M sends 10 nmea sentences per second. There is no way to shorten any of those, you just can mute whichever you want. This is chipset dependant.
2. not using try { } catch(e) { } and using explicit tests everywhere takes 45 ms per nmea sentence without any control of the checksum. This resulted in a loop too slow to process all sentences. Then I discovered that the NEO-6M would drop transmission right in the middle of a sentence without any warning.
3. using try { } catch { } and going in appropriate treatments of exceptions and adding the checksum control of all nmea sentences, reduced to timing to 25 ms per nmea sentence. 50% of this time was spent in the checksum computation alone.
4. compiling basic conversion functions and furthermore the computation of checksum, it went done to 12 ms per nmea sentences.
5. shutting off unused sentences directly from the NEO-6M reduced the number of sentences from 10 to 3 usefull and increased to time of treatment to 18 ms. However, the previous figure implied to 7 sentences were consumming some time for nothing too. Net result is very positive (318<<1012)

Finally, I used, and adapted, my Chrono prototypes to get the processor charge in % of the time: that is rather than looking at individual timing of a nmea sentence, look at the global timing of the treatment of all of them.
Basically, now the processor charge is around 5.5% to process all 3 usefull sentences, while this was about 12% to process all sentences with some redundant fields.
I still have to simplify the nmea sentences handlers so as to not take in account more than once the fields I need (hhmmss.ss for instances is read and converted 3 times now while it is always the same figure as long as you have it from the first sentence in a given sequence of sentences).

[gfwilliams]

Posted at 2015-09-18 by @gfwilliams

Hmm, that's hard to narrow down. Could you try trace(Serial4)? It's not as easy to read, but it does output everything.

edit: actually forget that - it was the input buffer overflowing because of the time taken to execute the constructor.

[espruino-discuss]

Posted at 2015-09-18 by asez73

I haven't yet used it on the pico. However, except for the serial pins definitions it should run all the way...
Yes, I forgot, you have to send with throttle otherwise, it becomes totally unpredictable.

Ok, now the trace(Serial4) really prints everything. Let's hope we can understand what's going on from that.
It is too big to fit in a Reply so I joined it as a text file.

edit: Reading further this trace, I was wondering:

1. Is it normal that the values of the parameters of gps.connect are still keept?
2. Again, is it normal that the comments at the top level are keept too?

---

Attachments:

• trace_serial4.txt

[gfwilliams]

Posted at 2015-09-18 by @gfwilliams

Is that trace from when it has the memory leak? I don't see anything obvious in there (but it's hard to see!) - although E.getSizeOf is definitely getting pretty confused by the references back in the functions.

Perhaps outputting the data from trace() (without the Serial4) before and after the memory leak might help, as we could then diff them.

Also, do you get the memory leak with non-"compiled" code? It could be it's a memory leak in the interpreter caused by a bug in the compiler.

[gfwilliams]

Posted at 2015-09-18 by @gfwilliams

Is it normal that the values of the parameters of gps.connect are still keept?

Yes, because of the scope chain from functions that you have defined in gps.connect.

You could:

• Define the functions outside, and just add them to the listeners by name
• Set the parameters to undefined when you're done with them

Again, is it normal that the comments at the top level are keept too?

Yes, they're in history - the second Espruino runs low on memory it'll delete those.

[espruino-discuss]

Posted at 2015-09-18 by asez73

Yes, you are right.
After reading this tutorial, I still have to correctly understand the real closure/implicit or explicit definition(s) of this: This is a beginner's problem in Javascript any way.
The tutorial seems to be slightly out of date, on the other hand I nearly understood the difference between binding and late binding.
Still have to experiment that anyway.

[espruino-discuss]

Posted at 2015-09-18 by asez73

Ok, that's clear now: I have to review the connect function and make it much more memory and time savy.
The trace has no identified memory leak so far.
It has been running for a few hours and the free memory is just changing by a few 10s of JsVar back and forth: I guess this is induced by the length of the nmea sentences.

So far, the "compiled" functions have not been a problem since you solve the webide getting confused about the order of returned compiled function last week.

edit: Here is a memory status taken from just after the trace(Serial4) to now.
Apparently the size of gps and Serial4 are slowly changing, and getting smaller. I think this could be the result of some garbage collection...
edit2: Actually, it is because the GPS lost its position fix and thus some properties of gps went to NaN and so on...
edit3: It is worst than that: Serial4 is no more defined, I create a new psot about that

>process.memory();
={ "free": 791, "usage": 1459, "total": 2250, "history": 569,
  "stackEndAddress": 536910664, "flash_start": 134217728, "flash_binary_end": 228576, "flash_code_start": 134449152, "flash_length": 262144 }
>for (var i in global) print(i+" : "+E.getSizeOf(global[i]))
Chrono : 108
ddmmyy : 37
E : 1
secn : 41
lln : 40
cks7 : 27
nmea_checksum : 54
start_stop_nmea : 84
fieldsOfInterest : 31
codeFieldOfInterest : 44
posFieldsOfInterest : 26
getFieldsPos : 8
parseHandlers : 124
estimate_dist : 25
tfTimeout : 1
tf : 66
connect : 183
gps : 1997
Serial : 6
Serial4 : 1998
console : 1
Number : 1
Date : 6
Error : 6
seconds : 1
i : 1
process : 1
myInterval : 1
ReferenceError : 6
show : 10
SyntaxError : 6
=undefined
GPS position fix from serial at 1442595640.43765068054
We have a GPS position fix!
GPS position fix from serial at 1442595666.95421600341
We have a GPS position fix!
GPS position fix from serial at 1442595752.85040473937
We have a GPS position fix!
>process.memory();
={ "free": 811, "usage": 1439, "total": 2250, "history": 283,
  "stackEndAddress": 536910664, "flash_start": 134217728, "flash_binary_end": 228576, "flash_code_start": 134449152, "flash_length": 262144 }
>
Disconnected
>
Connected
>
=undefined
>
=undefined
>for (var i in global) print(i+" : "+E.getSizeOf(global[i]))
Chrono : 108
ddmmyy : 37
E : 1
secn : 41
lln : 40
cks7 : 27
nmea_checksum : 54
start_stop_nmea : 84
fieldsOfInterest : 31
codeFieldOfInterest : 44
posFieldsOfInterest : 26
getFieldsPos : 8
parseHandlers : 124
estimate_dist : 25
tfTimeout : 1
tf : 66
connect : 183
gps : 1719
Serial : 6
Serial4 : 1720
console : 1
Number : 1
Date : 6
Error : 6
seconds : 1
i : 1
process : 1
myInterval : 1
ReferenceError : 6
show : 10
SyntaxError : 6
JSON : 1
=undefined
>
Disconnected
> 

[espruino-discuss]

Posted at 2015-09-21 by cwilt

I have not explored the SD file writing routines to see how well handles JSON. Was just a passing thought.

[espruino-discuss]

Posted at 2015-09-21 by @allObjects

what is the reason to use the bitwise OR (|) for the length

It's actually a performance thing...

Encountering this 0 | ... I removed the 0 | and first noticed a performance decrease timing 1K invocations... but later I felt not seeing it aymore and that's why I asked. The explanation makes it obvious... may be for compiled one should be able to provide type information for variables... to help the compiler a bit more... may be in the form of jsDoc... For running uncompiled it will (for sure) slow down the process... except something has changed in the upload to remove comments within a stament / function / object

var /*{String}*/ s = "Abc";
var /*{int}*/ len = s./*{int}*/length; // (or s/*{int}*/.length;

Similar approach for function signatures incl. return type.

[gfwilliams]

Posted at 2015-09-22 by @gfwilliams

hat happens if I do something as setTimeout(f,0); That is a 0 timeout call. Is it queued at the end of the events waiting or inserted at the head of the events queue?

Yes, it's executed after the Serial events have been processed - that could change at some point in the future though.

However the 'real-time' handling of Serial events may do what you need anyway. When it calls your on('data' handler, it searches for all the Serial data it can find on the top of the event queue and calls the handler with it.

Suppose you have a stream of data ABCDABCDABCDABCD... and you have a very fast handler that can execute data as it comes in, it'll get called with:

handler("A")
handler("B")
handler("C")
// ...

But if you have a slow handler, it'll get called with more data:

handler("ABCDABCDABCDABCDABCDABCD");
handler("ABCDABCDABCDABCD");
// ...

I guess perhaps by splitting off your handling you could detect whether you had more than one GPS sentence waiting to be processed and could voluntarily drop one rather than filling up the input buffer.

[espruino-discuss]

Posted at 2015-09-22 by @allObjects

I think the batching is the better idea then try to have handlers that are actually fast enough to do the job. ...batching even to a custome defineable delimiter... or number of bytes received. An application (timeout) triggered getData(callbackData,callbackNoData) - w/ synchronous callbacks - could be an option to check for and get data...

For the SDI-12 protocol I was thinking of having a similar 'buffering middle-layer' that gets pulled data from the app in desired chunks... This allows failing patterns to drop before they even go into the buffer.

[espruino-discuss]

Posted at 2015-09-24 by asez73

Hello,
I must be blind: I had totally forgotten the Testing environment included in the web ide.
So, as you wrote, there are videos explaining how to use it.

[espruino-discuss]

Posted at 2015-09-24 by asez73

Well, gps is all about real time and position...

[espruino-discuss]

Posted at 2015-09-24 by @allObjects

Yep, it is a great help in monitoring your Espruino... to see what is going on... The challenge is that it 'interferes' timewise, because code is exectuted on an interval bases that consumes cycles to get the data and send it over the Serial/USB/Console to the testing tool.

[espruino-discuss]

Posted at 2015-09-26 by asez73

@gordon and all others interested:
My problem was indeed an out of memory error, happening due to the exessive size of the gps object created.
The object was very oversize in memory due to using an eval() to transform my dynamically computed strings into functions.
Eval uses the local context which was the local context of the initialising function. Thus eval linked all of its local vars and values, local functions and so on.
This discussion gave me the answer.

How to call eval in a given context? 3 words. Use a closure.

var result = function(str){
  return eval(str);
}.call(context,somestring);

The context object above is a global object created outside of the parseHandlers function.
It could be seen as a pre initialised gps object. Something like
{ serial: Serial4 } is sufficient if provided as a parameter of the function that computes the handling functions.

The joined file is actually the parsing function that creates the gps object with its nmea sentences as methods. You then can call gps.GPGGA(d) were d is the array, comma splitted, resulting of the nmea string received after checksum and dollar having been removed.

Once intialised, you just could delete parseHandlers function which will free the memory space it used: around 650 jsVars... While gps object is less than a 100 jsVars....

So, so...

Attachments:

• gps_gen13.js

[gfwilliams]

Posted at 2015-09-26 by @gfwilliams

That's great! Thanks for posting up!

Do you think there's anything that could be done to warn users when this is happening or help them detect it? I guess E.getSizeOf could have an option where it drills down into the object, showing hidden properties as well.

[espruino-discuss]

Posted at 2015-09-26 by asez73

Definitly yes,
While the case about "this" is a classic problem to javascript, it becomes crucial on very small platforms such as Espruino. This would have gone unnoticed in a web browser for instance.
Drilling down in objects is a need to diagnostic and i ended with a difficulty to read the trace output.

[espruino-discuss]

Posted at 2015-09-26 by @allObjects

@asez73, where do you draw the line 'real time'? Batching to simplify in the internal processing does not mean it is not real time for the user. Batching was also meant for just collecting all the received bytes until they make a reasonable gps sentence. This process should not be visible to the application. When look at @gordon's implementation of the GPS moudle, it has a continuous buffering of received data. When something is ready to put into the buffer it is just appended, and when a sentence is found, it removes that part from the buffer and calls the callback. In other words, the filling of the buffer happens in chunks not necessarily aligned with the calling of the callback. If the buffering happens in pieces smaller than a complete sentence callback is not called on recption/buffering event. If more than one sentence is received, then the code loops around invocation of the callback until no complete sentence is found anymore in the buffer.

[espruino-discuss]

Posted at 2015-09-26 by asez73

Real time, normally, means with a known delay between this instant of the event and the start of the routine in charge of handling the event.
In the particular case of the Espruino interpreter, things are handled by pushing a time stamp and the handler gets it with the pertinent data somehow later.
So, in such a very different context, I would call "realtime" anything that would end up by consumming events fast enough so as not to overflow the event fifo....
I guess this is how Espruino was designed.
Of course, in my particular gps goals, I need to end up all of the computations within a second since the gps gives its data every second, or 1/10 of a second if properly setup.
At this point, your own requirments are the rule.
Do forget that writing to aflash memory takes literally ages per comparison since they are blocking apis.
Even worst, the so called non blocking apis are never so:
Either you use a multi core / processor, which is not Espruino's case and you could delegate blocking tasks to one of them.
Or you have to buffer and delay blocking actions to the risk of not handling whatever happens at that instant.... So i would say that buffering is, per itself, a problem! That's why, I would that consider only buffering summaries, that can wait in a buffer for a very long time (seconds to hours). While i'd prefer to handle the full data stream in "realtime".
There is no universal answer to this!
It will, forever, be a matter of priorities: should I kill my batteries by over discharging them, or let my drone fall on someone's else head? A boat or a car could just safely stop in place and not destroy anything or anyone.

[espruino-discuss]

Posted at 2015-09-26 by @allObjects

There is no universal answer to this!

That's right, It all depends.

While i'd prefer to handle the full data stream in "realtime".

There are the layers... such as the comm layer byte by byte, bytes by bytes, sentence by sentence, processing sentence by sentence, etc. I think the solution is optimal to have each layer to be as efficient as possible. I see your point that you actually could even start to process a single value from a sentence as soon as available. For certain things that can make sense. On the other hand, to make a "final" decision, you nead 3 things in a 3D world.

I think your approach to tailor first which sentences and then do the same for which values in the remaining sentences, that gets you where you want to be: as realtime as possible, realtime as to the point: as soon as the information is available.

For storing the values? Do you already have a solution? If you can process the pertine data in time, how much time do you have left to store it?... for example: in sequences of deltas? reduced granularity? encoded to use minimal space? Any processing requires additional time before storing. 100ms is not much the get everything done... Your app may become the first multi-PICO solution... or you switch over to the Propeller.

[espruino-discuss]

Posted at 2015-09-27 by asez73

Well, you spotted it : To my opinion and so far identified needs, I would say that the "low level" realtime layer separates the gps data stream from the use of these data.
You could just use these data to display them or produce much more dynamic computations and finally store only the results of these...
Yet, all considered, those computations have to be real time too. As far as no other delaying hardware impose its own pace.
By dynamic computations I mean data values based complexity.
As it happens with the big data world it can just be impossible to do some post treatments either due to the data size (mb) or to the volume of data to get of the storage support per time unit (mb/s) .
Do we pretend that the espruino has to solve any real time need? I would n't

[gfwilliams]

Posted at 2015-10-01 by @gfwilliams

I've now made E.getSizeOf a bit more powerful. On latest git builds (and for 1v81), you can do:

>function a(b) {
:  return function() {
:  };
:}
=function (b) { ... }

>E.getSizeOf(a("Something really big"))
=12

>E.getSizeOf(a("Something really big"), 1)
=[
  {
    "name": "ÿcod",
    "size": 3 },
  {
    "name": "ÿsco",
    "size": 8 }
 ]

>E.getSizeOf(a("Something really big"), 2)
=[
  {
    "name": "ÿcod",
    "size": 3 },
  {
    "name": "ÿsco",
    "size": 8,
    "more": [
      {
"name": "b",
"size": 4 },
      {
"name": "return",
"size": 2 }
     ]
   }
 ]