string to hex string /hex string to string conversion

[espruino-discuss]

Posted at 2018-11-26 by Kartman

The Quectel BC95 module expects the data payload as hex strings. Are the following methods, the most memory efficient way of doing this?

//convert to a hex string
    for (var i = 0; i < data.length; i++) {
      var s = data.charCodeAt(i).toString(16);
      while (s.length < 2) {
        s = '0' + s;
      }
      msg += s;
    }

and conversely:

for (let ndx = 0; ndx < (recvData[1] * 2); ndx += 2) {
    let i = parseInt("0x" + recvData[2].substring(ndx, ndx + 2)); //ascii hex to integer
    sockData[connection] += String.fromCharCode(i);          //build a string
  }

From my limited understanding, this code would create a string, append and create a new string, rinse and repeat. This seems like a bit of overhead. I did consider typed arrays, but how do I convert these back to a string in one go?

[espruino-discuss]

Posted at 2018-11-26 by @gfwilliams

How much data are you looking at sending? If you don't mind a bit of (short-term) memory overhead you can do:

// to hex
data.split("").map(x=>(256+x.charCodeAt()).toString(16).substr(-2)).join("")
// from hex
E.toString("01020304102030".match(/../g).map(x=>parseInt(x,16)))

This should be reasonably fast, but it'll allocate an array so for anything over a few hundred bytes it could cause problems.

String append is actually pretty fast & efficient in Espruino, so using your idea with a few of the hacks from above you might find this is better:

// from hex
for (var x of data)msg+=(256+x.charCodeAt()).toString(16).substr(-2);
// to hex
var l = recvData[1]*2;
var s="";
for (var i=0;i<l;i+=2) s+=String.fromCharCode(data.substr(i,2));
sockData[connection]+=s;

It might be that you can configure the module to output data in binary though? That might be an easier way of working.

---

Posted at 2018-11-26 by Kartman

That code is like decoding a puzzle! I saw the 256 + bit and I'm thinking What!! Then I twigged that it is to ensure there's at least two hex chars. Thanks for the tips.

Unfortunately the Quectel BC95 uses hex ascii. Why? Who knows. I can't think of any other AT module that does it that way.

We were having some memory problems especially when receiving, so I had thought my code was not too efficient. I had it in my mind that there might be a simple way to use a byte array and cast it to a string. That way there's no copy and allocate.

---

Posted at 2018-11-27 by Nadnerb

I put together this implementation using a byte array without having to copy and allocate new strings.

var buff = new Uint8Array(1024);

//ascii string to byte array
for (let i = 0; i < data.length; i++) {
      let c = ('0' + data.charCodeAt(i).toString(16)).slice(-2);
      buff[i * 2] = c.charCodeAt(0);
      buff[i * 2 + 1] = c.charCodeAt(1);
}

//Ascii string from byte Array - Note 'apply' only works for 64 arguments
function fromByteArray(len) {
      if (len < 64) return String.fromCharCode.apply(String, buff.slice(0,len));
      for (var result = [], i = 0; i < len; i += 64) {
            result.push(String.fromCharCode.apply(String, buff.slice(i, i + 64)));
      }
      return result.join("").slice(0,len); 
}

//Hex string to byte array
function toByteArray(data) {
      for (let i = 0; i < data.length/2; i++) {
            buff[i] = parseInt((data.substring(i*2,i*2+2)),16)
      }
      buff.slice(0,data.length /2);
}

After running a process.memory() command before and after these changes, I can't see any significant memory savings. Should we expect any improvements in memory usage by using a byte array, or does the string append provide the best use of memory allocation?

What improvements could we make to this implementation to improve memory allocation?

---

Posted at 2018-11-27 by @gfwilliams

String append with += is definitely pretty efficient - as the string will be appended rather than copied to a new string each time.

That's a great use of String.fromCharCode.apply in fromByteArray - however there is actually E.toString which allows you to do that in one big chunk (because there didn't seem to be a nice way in JS itself).

@KarTmaN yes, the code is a bit cryptic! Unfortunately there's nothing built-in. Node.js has Buffer.toString(encoding) which would have been perfect, but we don't have Buffer in Espruino - just DataView which is depressingly similar, but without a .toString(encoding) option