Architecture: net.connect() and connection callbacks ...

[espruino-discuss3]

Posted at 2015-09-02 by Kolban

In the architecture of the "net.connect()" function we can supply a callback function that will be invoked when a connection has been established. The parameter to the callback function is the Socket data structure that represents the connection. This is all sound.

However, studying the socketserver.c source (lines 700-712), I find that a socket creation request is sent to the board driver and then IMMEDIATELY, the connected callback is invoked. This is a problem for some board implementations (such as the ESP8266) which have their own callbacks to indicate when the connection has been established.

What this means is that today ... if I call "net.connect()", the connected callback is invoked before we have actually established a real connection through the board interface. What it seems we need is some form of integration with the board's concept of when a connection is ready.

[espruino-discuss3]

Posted at 2015-09-02 by @allObjects

I do not know the detail code, but some of the callbacks are called multiple times and are passed state, and the (app) software has to check for the expected state... this allows to work with just one callback versus many, one for each different states.

---

Posted at 2015-09-02 by DrAzzy

I find that a socket creation request is sent to the board driver and then IMMEDIATELY, the connected callback is invoked.

I don't see how this couldn't be a bug, unless the call is expected to block until the part returns a confirmation that it's finished - which would sort of defeat the point of passing a callback...

---

Posted at 2015-09-05 by Kolban

As I dig deeper, it is getting worse. It seems that the Espruino architecture assumes that the net, socket and HTTP libraries make TCP calls that are blocking until they have completed their tasks. However, the ESP8266 architecture is such that we make network requests and the results happen asynchronously some time later ... however, between the time that we make the request and get the result, we must give control back to the ESP8266 kernel ... which means that I can't just "busy wait" for the callback to happen.

---

Posted at 2015-09-06 by DrAzzy

Yes - the calls are blocking, despite the fact that we supply callbacks :-/

It would be much better if the calls didn't block, and the callback was then called asynchronously when the request was completed, and treated like any other espruino event (like timeout/etc)... it sounds like that's what the ESP8266 wants you to do, and it seems like it should be possible on Espruino, even if it doesn't work

I mean, if the calls are all blocking, we might as well not have a callback, and just have the http.get() return the response! But a callback, and non-blocking requests would be much more in keeping with the Espruino way...

---

Posted at 2015-09-07 by @gfwilliams

It'd be worth looking at how the existing ESP8266 library handles it because that has exactly the same issues and still works fine.

Did you see the inline documentation in network.h?

• for connect it's expected to block - but it doesn't matter too much if it doesn't since the actual HTTP callback won't be called until a header is received. The only case it'd really matter is in a standard (non-HTTP) connect call. If it turns out the connect has failed then send/recv can just return -1 when they're next called and everything will get cleaned up nicely.
• send/recv just return 0 if they can't return/take any data.

So it really shouldn't be very painful to make it work, with the minor issue that a simple TCP connect that fails will still call the connected callback. IMO that's something that you should look at doing after you've got working networking though.

---

Posted at 2015-09-21 by Kolban

Howdy Gordon. Thanks for the response. Am still catching up on a couple of weeks missed goodies. I had a look at how the ESP8266 library that already exists works. If I am understanding it correctly, the library is sending Serial AT commands and parsing their responses. This means that the ESP8266 as an external network device is "piggy-backed" onto the Espruino boards. The blocking connect works in this case because the library is sending a Serial request and can then happily block waiting for a serial response. Unfortunately, when we run on the ESP8266 itself, we can never afford to block. When C code running on the ESP8266 executes its equivalent of a sockets "connect" call, what really happens is that the a request to perform a connect is placed on a processing queue any is not executed until control is given back to the ESP8266. A callback function is registered that is invoked with the outcome of the connect. Would it be possible for us to chit chat on this story in more detail? I am a cross roads on which path ESP8266 board implementation should take and I'd love it to be as as seamless as possible with what already exists but some of that may need compromises and changes. I want to figure out a strategy with you that meets your desires while at the same time hoping to follow the ESP8266 architecture practices where ever they don't conflict.

---

Posted at 2015-09-21 by @gfwilliams

The blocking connect works in this case because the library is sending a Serial request and can then happily block waiting for a serial response

No, it doesn't block at all. Your implementation just needs to do:

bool isBusy = false;

function esp8266_connected() { isBusy = false; }
function esp8266_sent() { isBusy = false; }
function esp8266_received() { isBusy = false; }

function connect() {
  esp8266_connect(...);
 isBusy = true;
}

function send() {
  if (isBusy) return 0;
  isBusy = true;
  esp8266_send(...);
  return bytes_sent;
}

function recv() {
  if (isBusy) return 0;
  isBusy = true;
  esp8266_recv(...);
  return bytes_received;
}

---

Posted at 2015-09-21 by @gfwilliams

At some point in the future, connect can be changed such that it can return a i'm connecting value - see this

... but honestly right now it'll work fine if you do it as above. I don't think there's any need to do any massive changes - it's more important to just get something working.

---

Posted at 2015-09-22 by @allObjects

A callback does not automatically include a different thread (callstack wise)... It can be calling something that then does 'synchronously' call the callback after completion of the connect to continue in the app as the app defines. Looking at the call stack it would show 3(+) levels with the last 3 levels: the app running and calling the connect, connect running and calling the app (callback), the app (callback) running...

A non-blocking connect call would actually store the callback away, return to the app before the connect is completed. Completin of the connect then calls the stored away call back in a next js-activity 'thread'.

Since the connect on these small devices are 'blocking', interrupts become a challenge, especially when it takes some for the connect to come through.

I'm wondering if a software interrupt could be used... in the meantime with a setWatch and a pin... with that pin just being the (sacrificed) pawn to wake up a service routine resolver, latter fireing the execution of the service routine stored (instead of connect storing away the callback at invocation, the callback is put on the interrupt 'stack' / fix pickup location just before invoking the software interrup (yanking the pawn pin).

Instead of going down the HW / setWatch / pin-yanking lane, it could produce an interrupt in the regular Espruino interrupt queue... (which I think Espruino is already doing for all asynchronously handeled callbacks).

---

Posted at 2015-09-22 by Kolban

@gfwilliams,
I'm afraid I'm not following the meat in post #8. In JavaScript (pseudo code), I want to execute something like:

esp8266.connect("IP", function(handle) {
      handle.send("Some data", function() {
          handle.close();
      }
});

This would request a connection ... when the connection is made, it would send some data ... and when the data has been sent ... perform a close of the connection. However, since the current base code invokes the connection callback immediately after asking for a connection ... this is before the connection has been formed and we are not yet ready to perform a send operation. In the ESP8266 world, the connection operation (at the ESP8266 SDK level) is always non blocking and one supplies a callback function pointer (in C) that will be invoked when the connection has been completed. It is then that I would have imagined that the net.connect() callback would have been ready to be called.

I'm not seeing how post #8 changes that story....?

Neil

---

Posted at 2015-09-23 by @gfwilliams

The normal JS way of doing network connections is like this:

client = require("net").connect({host : IP, port: 80}, function() {
    console.log('client connected');
    client.write("Hello World");
    client.end();    
  });

That's all already implemented and would call into the network-related functions you implement. It actually works like this:

• JS: calls connect
• Espruino: call esp8266 connect
• esp8266: starts connection, stores that it's busy
• JS: connect callback is called
• JS: calls write
• Espruino: buffers written data
• JS: calls end
• Espruino: stores that it must close connection after data sent
• Espruino: next time around the idle loop....
• Espruino: calls esp8266 write with buffered data
• esp8266: write return 0 - saying it's busy
• Espruino: repeats above each time around idle loop
• esp8266: gets callback saying it's connected, sets state to not busy
• Espruino: calls esp8266 write with buffered data
• esp8266: write is ok, returns number of bytes written
• Espruino: calls esp8266 write with buffered data, if there is any
• esp8266: write is ok, returns number of bytes written
• Espruino: no more data, call close connection
• esp8266: closes connection
• We're done...

So the buffering is all built in, and it all 'just works' - including a 'drain' event that allows you to 'pipe' lots of data in without filling up buffers. All you have to do is implement the code from post 8.

So the 'connected' callback is called before the ESP8266 is completely connected (which is a shame, but not a huge problem). In the case of HTTP is all 'just works' correctly though, because the request is only called when an HTTP response is received.

Hopefully at some point in the future this could be changed, but there's absolutely no reason to re-write it just for this, when it's trivial to make it work as-is.

---

Posted at 2015-09-23 by Kolban

Gordon,
Aha!! The light bulb just went on for me after reading this post. Tonight after work I'll be studying this in GREAT detail but I think I see what you are saying. Basically, at the ESP8266 networking level that is SPECIFIC to the ESP8266 (i.e. the lowest level of the Espruino stack and the board specific code) ... THAT library should maintain state on socket connections and when the higher level functions ask ESP8266 to do something ... even though the connection has not been formed ... that socket data structure will have private STATE (and this is where I missed the key point) ... it will return 0 for sends and receives ... but when the ESP8266 (actual) connect callback returns, that will change the state so that subsequent sends and receives will now be honored.

Again... THANK YOU for your extreme patience. I'm going to try and write this story down for others who follow as well.

Neil

---

Posted at 2015-09-23 by @gfwilliams

Yep, that's it - thanks!

And if the connect fails, you just make sure that send/recv return -1 to signal that there was a problem.

---

Posted at 2015-09-24 by Kolban

I wanted to report back that your technique works well. What I think we want to do is put our heads together and capture and document the EXACT nature of the contract between the socketserver layer and the board layer. Specifically, each board must implement a set of expected functions. What we need to do is pin down exactly the nature of the contract including semantics. The code source code of the Espruino base and examining the examples describes the signatures of the functions ... but doesn't capture the execution flow nor semantics and I think (in hind sight) that is what gave me most of my grief. If we can learn from this and make notes on what each of the board supplied functions does and its expected contract ... that would help a lot.

I have found yet another blocking vs non-blocking issue which is going to be tricker to resolve ... and that is "gethostbyname" which takes as input a DNS name and returns the corresponding IP address. The ESP8266 has this ... it is called espconn_gethostbyname ... however ... it is asynchronous. What I mean by this is that its spec is that when one calls it, one supplies a callback function that will be called in the future when the DNS resolution is complete. In the network and serversocket code, we expect it to be blocking.

---

Posted at 2015-09-24 by @gfwilliams

Well, I'd be up for more documentation in network.h - although the descriptions there do actually cover what happens quite well. Maybe a comment showing the flow of function calls would help to clear things up.

For gethostbyname it's worth looking at what's been done already. That problem has already been solved for the existing ESP8266 stuff.

JSNetwork stores the string in gethostbyname, then returns 0xFFFFFFFF (an invalid IP) to signal that the 'connect' call should sort everything out.

So now your code looks like:

hostIp = "";
state = IDLE;
function gethostbyname(name) {
  hostName = name;
  esp8266_gethostaddress(name);
  return 0xFFFFFFFF;
}
function esp8266_gotHostAddress(ip) {
   hostIp =  ip;
   if (state==BUSY_HOSTNAME_AND_CONNECT)
    realconnect();
  else
   state = IDLE;
}
function realconnect() {
  state = BUSY_CONNECTING;
  esp8266_connect();
}
function esp8266_connected() {
  state = IDLE;
}
function connect(ip, ...) {
  if (ip!=0xFFFFFFFF) hostIp==ip;
 if (state==BUSY_HOSTNAME)
   state = BUSY_HOSTNAME_AND_CONNECT;
 else
  realconnect();
  return -1;
}

But please, if you have questions about this, can you look at the existing implementations of linux, jsnetwork, and wiznet to try and get some ideas first? At this rate I'm basically telling you step by step how to implement every function.

---

Posted at 2015-09-24 by Kolban

@gfwilliams ... awesome ... I understand your response and those were the designs I needed. As for telling me how to implement every function ... I don't disagree. You ARE the whole Espruino architect ... you OWN the grand design and vision. Our contribution is time, ESP8266 skills, ESP8266 coding, debugging, and testing. Right back in the original posts on the ESP8266 port, we recognized that our challenges on this port would not be ESP8266 knowledge or time ... but Espruino internals knowledge.

I'n my 9-5 job, I'm a grunt programmer and used to working with designers and architects. Those guys own the knowledge and vision and ... the important part ... architecture. Together ... we work on a project ("Port Espruino to ESP8266"). We identify work tasks ("Implement networking"), we assign responsibilities ("Kolban - go own 'implement networking') and then I buckle down to figure out how it can be done. However, there isn't a day that doesn't go by where I don't have a question or problem that I bring by the architect who points me in the right direction. He doesn't "implement" what my job is to build ... he guides me on the architecture and, just like you did in this post, he may help me with pseudo code and drawings. The grind work then comes back to me to take what he has given me and from there ... low level design, code, test, debug, document.

This is my way of saying how grateful I am that you take the time to respond patiently to these forum posts. Maybe that last post of yours took 10 minutes or 30 minutes to compose ... but I anticipate that my work now will take a few hours to get right and test and document ... and if I do it correctly, there will now be a paper trail for those who come next.

It is also vital that we not abuse your time and grace. If you ever feel that we are off track PLEASE let us know and we will throttle down questions ... or simply delay in responding to them till your time permits. If there are others in the Espruino community who have knowledge on the networking subsystem even close to Gordon, please make yourself known :-)

Neil

---

Posted at 2015-09-24 by @gfwilliams

Thanks! Yes, it's tricky - looks like the end result will be well worth it though :)

It's just a shame the forum search is so poor, as it's quite hard for these posts to get found - hell, I have trouble and I know what keywords I'm looking for!

---

Posted at 2015-09-24 by @allObjects

@gfwilliams, what is triggering invocation of any of the functions outlined in post 8? - Is it a) a real hardware interrupt? - or b) execution goes to idle? - or c) is it both a) and b)? - or d) an internal timeout to let other 'threads' become some cycle time? I guess it cannot be d), becuase that would lead to multi-threading in Javascript and create the 'contention/concurrency mess' of any 'regular' OS.

---

Posted at 2015-09-24 by @gfwilliams

I have no idea how esp8266_connected/etc are called - maybe @Kolban might be some help there?

The actual functions in the network wrapper are called via the socket client/server code that runs on idle.

---

Posted at 2015-09-24 by Kolban

From an ESP8266 perspective, EVERYTHING it does is asynchronous. When you write an app for an ESP8266 you are given one time control in an initialization function and from there you set up everything you need. Following that, everything is handled by callbacks. For example, if we wish to connect to an access point and then make a socket request ... the code in ESP8266 would make the API call to connect to the access point and then return. When the connection to the access point completes, our user code is called back to say it is done ... and there we would request a connection to a partner host ... we would AGAIN return control to the ESP8266 and when the connection is finally established, we are called back again ... where we can send data. Following the request to send data, we AGAIN return control to ESP8266 and when the data send is complete, we are called back once more.

So from an ESP8266 perspective, we make requests for ESP8266 to do something and then we pass control back to ESP8266 which tells us when done (or when an error occurred).

For the Espruino port, there is the concept of a "main loop" in Espruino (jsiLoop()). What we do there is:

MainLoop:

1. Call jsiLoop()
2. Schedule a callback to MainLoop when next ESP8266 idle
3. Return control to ESP8266

So basically what is happening is that we do one cycle of non-block Espruino JS work, register that we get called back when ESP8266 is idle and return control back to ESP8266. It does what ever it needs to do to keep itself alive and process network requests that are outstanding (which may be nothing in most cases) ... and then main loop is called again and we repeat.

It all hinges on Espruino being non-blocking. The rules of ESP8266 say that we can't starve ESP8266 execution for more than 50 msecs of elapsed time. So it is essential that any one pass through the Espruino processing loop be 50 msecs or less.

---

Posted at 2015-09-24 by @allObjects

50ms, that's not much... that means, if there is an ESP8266 event, you stick the information into the Espruino interrupt queue. When Espruione comes to an JS idle, it will then work off of that queue. I guess that is the reason to have this interrupt queue in the first place, because some JS stuff (thread started by an JS interrupt) is not completed in that time.

---

Posted at 2015-09-24 by Kolban

For an embedded system ... 50ms is an eternity .... (opinion).

---

Posted at 2015-09-25 by @allObjects

(opinion).

Granted.

But interesting comment here (Architecture: Core processing of jsiLoop) - and that was what I was thinking about when saying '50ms, that's not much...':

However Espruino must execute entire blocks of code at once. It's just the way it's written - it uses the main execution stack for parsing and to store program state, so it can't magically jump out of execution and resume where it left off.

Something that is only 'affordable' becuase by definition JS was thought to run single threaded...

I'm now asking myself: could there be places where an 'safe' interrupt in such a block could happen which would allow marking the conext in the stack and crating a new context for a 'pseudo multi-threading'? - of course with certain restrictions in what is possible and extra sync effort when accessing 'thread' shared 'things'?

---

Posted at 2015-09-25 by @gfwilliams

It's actually a big tradeoff Espruino makes rather than a JS problem. If it used bytecode it'd be relatively easy to break execution, but Espruino's parse+run behaviour means you have to have a separate stack for it.

It sucks, because in ES6 they're introducing yield in JS itself - which is going to be dead tricky to implement.

could there be places where an 'safe' interrupt in such a block could happen which would allow marking the conext in the stack and crating a new context for a 'pseudo multi-threading'?

Well, that's the thing. 'dead tricky' - not impossible. The interpreter could 'back out', writing 'I was in that for loop, that try block, and that function', and then try and recurse back to that exact position later on. It'd just be pretty inefficient.