chunks of an Uint8Array

[espruino-discuss]

Posted at 2016-11-23 by @MaBecker

this works excellent for Espruino

var C = {
 OLED_WIDTH                 : 128,
 OLED_CHAR                  : 0x40,
 OLED_CHUNK                 : 128
};
WIDTH = 128;
HEIGHT = 64;
var buffer = new Uint8Array((WIDTH * HEIGHT)/ 8);
bufferLen = buffer.length;
var chunk = new Uint8Array(C.OLED_CHUNK+1);
chunk[0] = C.OLED_CHAR;

for (var p=0; p < bufferLen; p+=C.OLED_CHUNK) {
  console.log(p +"-"+(p+C.OLED_CHUNK));
  chunk.set(new Uint8Array(buffer,p,C.OLED_CHUNK), 1);
}

but anywhere else it causes something like "RangeError: Offset/length out of range" for chunk.

Any suggestions how to rewrite this line

 chunk.set(new Uint8Array(buffer,p,C.OLED_CHUNK), 1);

to work with other javascript engines ?

[gfwilliams]

Posted at 2016-11-23 by @gfwilliams

Looks like it's that chunk.set(new Uint8Array(buffer,p,C.OLED_CHUNK), 1); should really be chunk.set(new Uint8Array(buffer.buffer,p,C.OLED_CHUNK), 1);

In the browser:

(new Uint8Array((new Uint8Array(1000)).buffer,0,8)).length
=8

(new Uint8Array((new Uint8Array(1000)),0,8)).length
=1000

It's one of those times where I can't help thinking that Espruino does the more sensible thing, but it's not compliant with the spec.

[gfwilliams]

Posted at 2016-11-23 by @gfwilliams

Ahh, and I just realised - the module itself that you copied that code from is perfectly JS compliant and fine.

The problem is that Graphics.buffer is an ArrayBuffer (so untyped). You replaced it with a Uint8Array :)

[espruino-discuss]

Posted at 2016-11-23 by @allObjects

How about... (corrected):

chunk.set(buffer.slice(p, p + C.OLED_CHUNK), 1);

[espruino-discuss]

Posted at 2016-11-23 by @MaBecker

thanks - I was so focused on chunk.set :-(

btw : The code of module SSD1306 if the fastest i have found so far - great work !

[espruino-discuss]

Posted at 2016-11-23 by @MaBecker

Oh yeah - that works too - thanks allObjects

[gfwilliams]

Posted at 2016-11-23 by @gfwilliams

@allObjects there is a slight problem with that:

• As per the spec, buffer.slice creates a brand new sparse array - it uses lots of memory and time to create
• Using new Uint8Array(buffer,p,C.OLED_CHUNK) only creates a 'view' of the data - nothing is copied until the set command.

Basically the .set(new Uint8Array(buffer,x,y), z); pattern is about as close to a memcpy in JavaScript as you can get

[espruino-discuss]

Posted at 2016-11-23 by @allObjects

edited... (and see post #10)
I expected the negative side effects... unfortunately (Chrome) browser just does not behave as (EDITED: by me misread) MDN constructor specs / doc ...

In test code new Uint8Array(buffer, begin, length).length return just the length of buffer... and on .set() it reports Uncaught RangeError: Source is too large(…).

In Chrom inspec/debug console:

> var b = new Uint8Array(10);
< undefined
> var t = new Uint8Array(5);
< undefined
> t.set(new Uint8Array(b,0,5))
X> VM380:1 Uncaught RangeError: Source is too large(…)(anonymous function) @ VM380:1
> (new Uint8Array(b,0,5)).length
< 10

Found no complaint on the Web about this violation...

Write a compiled copy function, could that help speeding up an element-wise copy?

[gfwilliams]

Posted at 2016-11-23 by @gfwilliams

As I'd said above, Chrome is spec compliant. See: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array

The constructor is:

new Uint8Array(length);
new Uint8Array(typedArray);
new Uint8Array(object);
new Uint8Array(buffer [, byteOffset [, length]]);

In this case, you're not passing in buffer, but typedArray - so the next 2 arguments are not being used. If you instead so (new Uint8Array(b.buffer,0,5)).length it'll work

[espruino-discuss]

Posted at 2016-11-23 by @allObjects

NICE! --- me: ddddoooohhhh (: ... LOL...

That's where implicit - wishful - typing by variable naming (trap) kicks in... the 'wrong' name has been chosen for the variable: buffer is not a buffer because it is called so... ;(

I made a rough performance test, and it shows that slice is still about 6 times slower (even with some optimization / moving the var s out of loop)... memory is for sure contributing to the time aspect... - you can directly run the attached .html file by just clicking on the link. Test on Espruino still pending.

<html><head><title>typedBuffedSliced</title></head>
<body><h3>typedBuffedSliced</h3>
<button onclick="exec()">exec()</button>
<script>
var C = {
 OLED_WIDTH                 : 128,
 OLED_CHAR                  : 0x40,
 OLED_CHUNK                 : 128
}, WIDTH = 128, HEIGHT = 64;
function exec() {
  var buf = new Uint8Array((WIDTH*HEIGHT) / 8), bLen = buf.length;
  var chunk = new Uint8Array(C.OLED_CHUNK+1), cLen = chunk.length;
  console.log("buf: " + bLen + " - chunk: " + cLen);
  chunk[0] = C.OLED_CHAR;
  var p, pe, c1, t10, t11, slice, c2, t20, t21, cL = C.OLED_CHUNK;
  for (p=0; p < bLen; p += C.OLED_CHUNK) { pe = p + C.OLED_CHUNK;
    t10 = new Date().getTime(); for (c1 = 0; c1 < 100000; c1++) {
      chunk.set(new Uint8Array(buf.buffer,p,cL), 1);
    } t11 = new Date().getTime();
    t20 = new Date().getTime(); for (c2 = 0; c2 < 100000; c2++) {
      chunk.set(buf.slice(p, pe), 1);
    } t21 = new Date().getTime();
    console.log(p+"-"+(pe)+": " + ((t21 - t20) / (t11 - t10)));
  }
}
</script>
<pre>
Console:
buf: 1024 - chunk: 129
0-128: 4.375
128-256: 5.529411764705882
256-384: 7.142857142857143
384-512: 7.142857142857143
512-640: 7.846153846153846
640-768: 5.875
768-896: 6.928571428571429
896-1024: 7.615384615384615
</pre>
</body>
</html>

PS: updated - (line 18 had buf.buf and passing undefined which did speed it up to factor 2)

---

Attachments:

• typedBuffedSliced.html
  
• typedBuffedSliced.png

[espruino-discuss]

Posted at 2016-11-23 by @allObjects

Espruino test (uploaded from editor pane and invoked with exec() in the console on Espruino Wifi):

var C = {
 OLED_WIDTH                 : 128,
 OLED_CHAR                  : 0x40,
 OLED_CHUNK                 : 128
}, WIDTH = 128, HEIGHT = 64;
function exec() {
  var buf = new Uint8Array((WIDTH*HEIGHT) / 8), bLen = buf.length;
  var chunk = new Uint8Array(C.OLED_CHUNK+1), cLen = chunk.length;
  console.log("buf: " + bLen + " - chunk: " + cLen);
  chunk[0] = C.OLED_CHAR;
  var p, pe, c1, t10, t11, slice, c2, t20, t21, cL = C.OLED_CHUNK;
  for (p=0; p < bLen; p += C.OLED_CHUNK) { pe = p + C.OLED_CHUNK;
t10 = new Date().getTime(); for (c1 = 0; c1 < 100; c1++) {
chunk.set(new Uint8Array(buf.buffer,p,cL), 1);
} t11 = new Date().getTime();
t20 = new Date().getTime(); for (c2 = 0; c2 < 100; c2++) {
chunk.set(buf.slice(p, pe), 1);
} t21 = new Date().getTime();
    console.log(p+"-"+(pe)+": " + ((t21 - t20) / (t11 - t10)));
  }
}

Console:

 1v87 Copyright 2016 G.Williams
>exec()
buf: 1024 - chunk: 129
0-128: 4.07852737698
128-256: 4.52856306801
256-384: 5.24940776218
384-512: 5.27483808673
512-640: 6.57535412290
640-768: 8.54367417763
768-896: 7.12319090028
896-1024: 7.51280680688

Interesting is the relatively great variation... so I run it with HEIGHT = 128 - double the original buffer... and variation is worse:

 1v87 Copyright 2016 G.Williams
>exec()
buf: 2048 - chunk: 129
0-128: 2.42377434903
128-256: 6.01698313765
256-384: 3.10307471306
384-512: 5.11592931261
512-640: 8.91618411518
640-768: 9.59016115959
768-896: 8.67426096522
896-1024: 8.14185429263
1024-1152: 13.84030717299
1152-1280: 13.78322895572
1280-1408: 8.70759501664
1408-1536: 10.94154115774
1536-1664: 22.54689730282
1664-1792: 12.00001317154
1792-1920: 34.27794204212
1920-2048: 16.76400105146

Have no answer why...

[gfwilliams]

Posted at 2016-11-24 by @gfwilliams

So you're saying that it gets worse the larger the offset in the ArrayBuffer?

I think it's due to the implementation of slice on Espruino. It's expecting that it'll run on a normal sparse array, which is a Linked List.

Because of that it's just iterating to get to the correct start index - even though with an ArrayBuffer it could just go straight there in most cases. I've just filed an issue for this.

[espruino-discuss]

Posted at 2016-11-24 by @MaBecker

wow - thanks to both for your effort and those very interesting explanations and samples !

[espruino-discuss]

Posted at 2016-11-24 by @MaBecker

So let me share the motivation behind this.

All npm modules out there are using Oled data updates byte by byte - so you can imagine how slow the screen updates are - takes more than 1sec for a full screen update.

The Espruino SSD1306.js does it different, updates are so quick - because it uses chunks to update.

So I decided to pimp the npm module I use for my beaglebone projects.

And this is the result: MaBecker/oled-i2c-bus@c099f61

[gfwilliams]

Posted at 2016-11-24 by @gfwilliams

Neat - thanks for the comment in the commit!