precedence of setInterval, eval and print execution

[espruino-discuss3]

Posted at 2018-08-19 by Steffen

I've got a function wrapped in a setInterval that does analogRead every 400 ms (as a presence detector). Sometimes it seams to stall for a couple of 100 ms. Because the time periods involved are very short it is hard to get to the root cause by experimenting.
Now I'm wondering if setInterval execution is blocked while
a) console data is received (programmatically, ~200 bytes at once)
b) the received string is evaluated
c) the result string (~200 bytes) is printed out
Where exactly can I read up about the timing of code execution?

[espruino-discuss3]

Posted at 2018-08-20 by DrAzzy

While the interpreter is idle, it checks for intervals, timeouts, and other events, and runs the associated handlers. After such a callback returns, interpreter is back in idle state (until the next callback fires).

So, if you have something slow-ish, that will block callbacks from firing until it's done. The Espruino will queue up callbacks during this time, but that may not be good enough, depending on your application.

This really points to a fundamental difference between what Arduino and Espruino are good at. Espruino makes it so easy to interact with the world via http/etc, write civilized, modern code in a forgiving language, gives you a live interpreter, and generally incredible ease of coding. Anything where you are doing a lot with strings is way easier. But the price of that is that it's pretty slow - there are plenty of things I can do on an 8mhz AVR that the 72mhz stm32 or 160mhz ESP8266 running Espruino just isn't fast enough for, and similarly, things where sub-second timing is critical are often troublesome. I have a few applications where I use an Espruino for the stuff that handles strings, talks to the world, and so on, talking to an AVR dedicated to the timing critical stuff (in my case, dealing with 433mhz OOK RF) over serial.....

For more detailed assessment, post the code you're observing this with and what board (and espruino version) you're using it on, and an example of what it's receiving over serial (the content of the string could be relevant); I find it hard to belive that what you've described would end up messing the timing up by hundreds of ms - but who knows what you're doing when processing the received string, or how your serial handler is written, and so on...

---

Posted at 2018-08-20 by @gfwilliams

JavaScript execution on Espruino is only single-threaded, so it's only ever going to execute one bit of JS at a time. Everything has the same precedence.

When you print something, that goes into an output buffer, and assuming that what you print fits into that buffer the print statement will execute immediately. If not, it'll wait until enough has been sent to fit the remainder of the text in it, and it'll continue. The size of that output buffer varies depending on device, but at ~200 bytes of output it's possible you'll be filling it.

If you're printing over USB sending 200 bytes should be done almost instantly, but if it's 9600 baud serial then it'd be taking about 200ms, which might be your issue? That would delay your interval firing (although subsequent executions of the interval would be on time unless they got delayed too).

If you thought the sending was delaying your code from executing then you could always split the string in two and use setTimeout to send the second part after the first had completed.

As @drazzy says it'd help to be able to see your code though.

---

Posted at 2018-08-20 by Steffen

Thanks for the responses. I'll definitely look into the output buffer limitations. How can I find out about the size?
The device is an Espruino WiFi on 1.99.
Here is the part of the code that matters:

  // measures and counts, counter reset ff below threshold
  meas() {
    const v = Math.round(analogRead(this.pin) * this.rng - this.offs)
    if (this.vals.length === this.nmax) this.vals.shift()
    this.vals.push(v)
    if (this.both === true) v = Math.abs(v)  // zero based, both too low and too high considered
    if (v < this.thr) {
      return this.cnt = 0
    } else {
      if (++this.cnt > this.tcnt) this.cnt = this.tcnt
      return this.cnt
    }
  }

  // call meas, evaluate and call back
  cfrm() {
    const c = this.meas()
    if (c === 0) return this.cb(this.val = 0)
    if (c === this.tcnt) return this.cb(this.val = 1)
    if (this.tcfm === 0) {
      // this.cb(this.val = 0)  // not yet confirmed, do nothing (?)
    } else {
      setTimeout(this.meas.bind(this), this.tcfm)
    }
  }

  start(cb) {
    if (cb === undefined) cb = this.cb || print
    this.cb = cb
    this.iv = setInterval(this.cfrm.bind(this), this.tupd)
    this.cfrm()
  }

There are 2 sensor instances running, one with an interval (this.tupd) of 400 ms, the other one 2 s.
'this.cb' is a callback to little instance that keeps track of the 0/1 values sent and eventually switches a light through a MOSFET, along with a TPC notification to the controlling server:

  // called by switching (presence) sensor
  cbsnsr(v) {
    this.snsr = v  // just for information
    if (this.ovrd === 1 || this.toff !== undefined) return
    // act if all conditions are met
    if (v === 1 && this.enbl === 1) this.act()
  }

  act() {
    this.tend = Date.now() + this.drtn      // just for information
    if (this.toon) clearTimeout(this.toon)  // to re-set timeout while already running
    const self = this
    this.toon = setTimeout(function() {
      self.toon = undefined
      // if (self.toff) clearTimeout(self.toff)
      // self.toff = setTimeout(function() {self.toff = undefined}, self.dblk)
      self.actr(0)
    }, this.drtn)
    if (this.val !== 1) this.actr(1) // switch on
  }

  actr(v) {
    digitalWrite(this.pin, this.val = v)
    this.report('')  // report enbl, snsr, val in an object
  }

  report(d) {
    if (this.rurl === undefined) return
    var req = http.get(this.rurl + JSON.stringify(this.getValue(d)))
    // req.on('error', function(){})
    return req  // for debug
  }

  getValue(d) {
    if (d === undefined) return this.val
    return {
      ovrd: this.ovrd, trest: this.trest(), enbl: this.enbl,
      snsr: this.snsr, value: this.val
    }
  }

The getValue() method is the one that gets called every 5 minutes via USB or TCP. The call is

\necho(0)\nprint({r: JSON.stringify(swch.getValue()) }, '@@\n')\n

The TCP console is attached like this:

  wifi.connect('XXX',{password:'YYY'}, function(err) {
    // console.log(err);
    if (err) return;
    wifi.turbo(true, function(err) {
      net.createServer(function(conn) {
        global.conn = conn;
        conn.on('close', function() {USB.setConsole()});
        conn.pipe(LoopbackA);
        LoopbackA.pipe(conn);
        try {
          LoopbackB.setConsole();  // setDeviceClockCmd: Unknown Device 720
        } catch(e) {};
      }).listen(23);
    });
  });

---

Posted at 2018-08-20 by @gfwilliams

And you notice a delay over both USB and TCPIP? I'm out of the office at the moment but i believe the output buffer is 128 on the WiFi, and what you're sending seems a lot less than that?

One thing you could try is using setBusyIndicator so you get an idea of whether Espruino is executing code for a serious amount of time at any point?

---

Posted at 2018-08-20 by Steffen

Not sure about delays over USB anymore.
The 'production system' is out of reach, I can't watch the busy indicator.
Espruino WiFi's IO and TX BUFFERMASKS are both 127, which is enough for the response.

From my current understanding of what happens on a server call of getValue(), all these things happen at once (before anything else like setInterval's execution can happen):
parse the command string, execute the command, send the response
Is this correct?
If I created a mini-repl within the TCP connection to do the 'swch.getValue()' call and return the result upon receipt of a magic word, without the LoopBack pipes, the duration of the disturbation might be significantly shorter.

I'm wondering if there is a way to have a function interrupt currently running console code without changing the Espruino src code. I know about the implications of interrupts, still seeing this as a good option if it's possible at all.

---

Posted at 2018-08-20 by @gfwilliams

Does the server connection stay open, or is that opened for each request as well?

Definitely pushing data over the TCPIP to the console as you are is going to have some significant overhead - the data has to come in over serial from the ESP8266, get parsed, pushed back into the IO buffer for the console, handled for the console, executed, and then the result printed. However that shouldn't be anywhere near 100ms - and in fact a lot of that work should be spread out so wouldn't delay the execution of setInterval.

If I created a mini-repl within the TCP connection to do the 'swch.getValue()' call and return the result upon receipt of a magic word, without the LoopBack pipes, the duration of the disturbation might be significantly shorter.

That'd definitely help. I wonder if there isn't something else going on though... Could you maybe send me the full code in such a way that I could try it out here where I do have access to the device?

I'm wondering if there is a way to have a function interrupt currently running console code without changing the Espruino src code.

For analogRead one option is to use Waveforms: http://www.espruino.com/Waveform

The analog read will then be performed in an interrupt and saved to a buffer. I haven't tried this but you may be able to double-buffer with just a single-element 16 bit buffer, which would allow you to get accurate timings on the readings.

Interrupting some JS to execute other JS isn't really something that's possible right now, but it is possible to do it with Assembly, inline C, or compiled JS and setWatch(...irq:true). I really think that for this, that would be overkill though!

---

Posted at 2018-08-21 by @allObjects

Interrupting some JS to execute other JS isn't really something that's possible right now,...

...and it should not - at least not in a simple way - because it messes with the single threaded concept which makes programming as simple as it is...

As soon as this can happen, semaphoring is required throughout a lot of code to make sure the things are before as they are after (partial) execution of a statement. Even with the means as mentioned - Assembly, inline and compile JS - coding has to follow certain rules and respect constraints in order to not make Espruino's (execution) consistency fall apart... as obviously - as much as I connected the dots - has already happened in some way in some implementation for pixl.js... I may be mistaken, but certain chosen concepts will stand for ever. Yes, they may make certain new implementations a bit more challenging, but they have bin chosen in the first place to get even started and evolve fast and successfully to the current state.

---

Posted at 2018-08-21 by Steffen

The TCP connection is created and closed for each single request.

The main code is here, modules ar attached:

TSw=require("TSwitch");
tsw1=TSw.create(B9, {duration: 180000});
TSns=require("TSensor");
rdr1=TSns.create(A4, {iv: 400, iv2: 50, offset: 76, thr: 9, both: true});
rdr1.start(tsw1.getcbsnsr());
drk1=TSns.create(A0, {iv: 10000, thr: 72});
drk1.start(tsw1.getcbenbl());

B9 switches the MOSFET (digitalWrite),
A4 is the the presence sensor (analogRead of a radar module, cf. https://shop.weidmann-elektronik.de/index.php?page=product&info=8),
A0 is the 'darkness sensor' with a photo resistor.
TSensor is meant to be a universal threashold sensor, all the parameters basically control how the raw signal is prepared such that the threshold makes sense. Both sensors call into the tsw1 switch instance passing 1 or 0.
The system is self contained. The switch tsw1 additionally reports actions via HTTP, which seems to be of no relevance, I've checked it.

I'll look deeper into setWatch(...irq: true). It might well be overkill, but it would be fun to get it working!
After all, in the end, it might be not too hard, and it would be a nice 'backdoor' in case it's really needed (e.g. if I wanted to check the presence sensor every 50 ms).
@allObjects: Fully agree. Not breaking the really great Espruino concept while beeing able to unleash the close to infinite speed of the great processor - that's the goal. It feels wrong to know that my little light switch would run perfectly if implemented in C. And my VIP customer - my wife - needs to be content ;)

---

Attachments:

• TSensor.js
• TSwitch.js

---

Posted at 2018-08-22 by @gfwilliams

Is there a possibility of keeping the connection open? I think that could help as well - as there's a bit of extra work that has to be done by Espruino when the connection is created... It all adds a delay.

Definitely:

• Keeping the connection open
• Adding your own handler for received data on the socket rather than using the console

Should improve things a lot for you I imagine.

---

Posted at 2018-08-22 by Steffen

I've started to do 2 things:

1. use Waveforms to read the data, this seems to be exactly what I fancied
2. use a special handler to read and transfer data through TCP instead of attaching the console.

Thanks for all the suggestions and the valuable feedback. Espruino Forum is an invaluable source of inspiration and help!

---

Posted at 2018-08-23 by @allObjects

@steffen: ...happy wife... happy life... --- the complement of it: indescribable...