Understanding Uint8Array Memory Blocks Usage: 3 Bytes Data Consuming 3 Blocks (14 bytes per block)

[espruino-discuss2]

Posted at 2023-08-14 by user156224

I am working on a project where I generate data that initially occupies 17 bytes, and I compress it down to 3 bytes. These 3 bytes are then stored in a Uint8Array, as shown in the code snippet below:

var buffer = new Uint8Array(3);
buffer[0] = deltaTimestamp;
buffer[1] = (deltaElevation >> 8) & 0xff;
buffer[2] = deltaElevation & 0xff;

I've noticed that this 3-byte array is taking up 3 blocks in memory, with each block being 14 bytes on my system. When I write this array to a binary file, the size of the file is just 3 bytes, as expected.

I would like to understand why this 3-byte array is consuming 3 blocks instead of being represented within a single block, and if there's a way to optimize this. Any insights would be greatly appreciated. Thank you!

[espruino-discuss2]

Posted at 2023-08-14 by @fanoush

the name "buffer" is stored there too, try to name it 'b', try to set the three byte array to string variable 's' via E.toString() - that could maybe take less space(?)

---

Posted at 2023-08-14 by user156224

Thank you for your suggestions! I've tried both changing the variable name to 'b' and converting the three-byte array to a string using E.toString(). Unfortunately, neither approach seems to have reduced the number of blocks used.

---

Posted at 2023-08-14 by @fanoush

how do you determine the size? just tried

Found G5, 2v16.294
>
>var a=5
=5
>E.getSizeOf(a)
=1
>var a="123"
="123"
>E.getSizeOf(a)
=1
>var a=new Uint8Array([1,2,3])
=new Uint8Array([1, 2, 3])
>E.getSizeOf(a)
=3
>var a=E.toString(Uint8Array([1,2,3]))
="\1\2\3"
>
>E.getSizeOf(a)
=2

---

Posted at 2023-08-14 by user156224

Either by checking the free blocks before creating the array and after using "process.memory().free"
or by using "E.getSizeOf(buffer)" and the answer is 3.

---

Posted at 2023-08-14 by @fanoush

also

>var a=Uint8Array([1,2,3]).buffer
=new Uint8Array([1, 2, 3]).buffer
>E.getSizeOf(a)
=2

maybe that is what E.toString takes from the array too in the previous example. Still it is interesting that only the string takes 1 and the arraybuffer needs 2 (and typed array even 3)

---

Posted at 2023-08-15 by @gfwilliams

There's a small bit of info on http://www.espruino.com/Performance#array-buffers-are-the-most-efficient-way-to-store-data

But basically Uint8Array/etc store their data as a string, but they have to have one variable allocated which explains what type (Uint8/Int16/etc) they are, with size and offset. They also have to link to an ArrayBuffer which then references the string, which is mainly so you can do:

a = new Uint8Array(10)
b = new Uint16Array(a.buffer)
a.buffer==b.buffer
=> true

If I just automatically made an ArrayBuffer every time someone referenced .buffer then the two buffers wouldn't be equal.

You'll say why can't the ArrayBuffer store data directly and the reason is that a String isn't just one datatype - we have string_1,_2,etc depending on the amount of characters in that string block - so we'd have to do the same with ArrayBuffer which would then use up more space in our enum than we have.

It can help to use trace(..) sometimes to see what a variable is made of:

>a=new Uint8Array([1,2,3])
=new Uint8Array([1, 2, 3])
>E.getSizeOf(a)
=3
>trace(a)
#3677[r1,l1] Uint8Array (offs 0, len 3)  #3676[r1,l0] ArrayBuffer (offs 0, len 3)    #3413[r1,l0] String [1 blocks] "\1\2\3"

So it shows Uint8Array->ArrayBuffer->String

@fanoush I'm not sure what happened in your test, but a 3 char string should only be one var:

>E.getSizeOf(E.toString(new Uint8Array([1, 2, 3])))
=1

So actually the docs are misleading. A String is the most efficient form of storage, but Uint8Array/etc allows you to reference those in a useful way (because normally Strings cannot be modified once created)

---

Posted at 2023-08-15 by @fanoush

@fanoush I'm not sure what happened in your test, but a 3 char string should only be one var:

Don't have newest build but just retried on 52840 dongle with 2.17 and I still see 2, but in Bangle 2 Emulator I see 1 indeed.

>var a=E.toString(new Uint8Array([1, 2, 3]))
="\1\2\3"
>E.getSizeOf(a)
=2
>process.memory()
={ free: 13963, usage: 37, total: 14000, history: 10,
  gc: 0, gctime: 20.6298828125, blocksize: 14, stackEndAddress: 537092008, stackFree: 39728,
  flash_start: 0, flash_binary_end: 447188, flash_code_start: 516096, flash_length: 1048576 }
>process.version
="2v17.450"
>trace(a)
#24[r1,l1] FlatString [2 blocks] "\1\2\3"
=undefined
> 

EDIT: added trace

EDIT2: maybe that's older E.toString() which still only made flat strings?

---

Posted at 2023-08-15 by @gfwilliams

maybe that's older E.toString() which still only made flat strings?

Ahh! Yes, that'll be it :) The newer one uses whatever takes the least space

---

Posted at 2023-08-15 by user156224

Thank you for the detailed explanation; it all makes sense now! I've included my code below where I store data in the "compressedData = []" array. Is there a way to save this data with fewer storage blocks or space?

var lastTimestamp = 1692048524472;
var lastElevation = 0;
var compressedData = [];


function compressData(sample) {
  // Split sample into timestamp and elevation
  var parts = sample.split("/");
  var timestamp = parts[0];
  var elevation = Math.round(parseFloat(parts[1]) * 100);

  // Calculate deltas
  var deltaTimestamp = (timestamp - lastTimestamp) / 10; // Time difference in tenths of milliseconds
  var deltaElevation = elevation - lastElevation;

  // Mask values
  deltaTimestamp &= 0xff;
  deltaElevation &= 0xffff;

  // Create buffer with 1 byte for timestamp delta and 2 bytes for elevation delta
  var buffer = new Uint8Array(3);
  buffer[0] = deltaTimestamp;
  buffer[1] = (deltaElevation >> 8) & 0xff;
  buffer[2] = deltaElevation & 0xff;

	console.log("size = " + E.getSizeOf(buffer));
	console.log("remaining blocks = " +process.memory().free);
  // Update last values
  lastTimestamp = timestamp;
  lastElevation = elevation;
  compressedData.push.apply(compressedData, buffer);
}

setInterval(function(){
compressData("1692048525472/294.35")
},500);




//the output is this:

size = 3
remaining blocks = 11820
size = 3
remaining blocks = 11817
size = 3
remaining blocks = 11814
size = 3
remaining blocks = 11811
size = 3
remaining blocks = 11808
size = 3
remaining blocks = 11805
size = 3
remaining blocks = 11802

---

Posted at 2023-08-15 by @gfwilliams

Well, having the compressedData array isn't going to be very efficient as it's just a normal array by the look of it. I'd consider just having a big ArrayBuffer and then use a DataView to set the 3 bytes directly. It even has a setUint16 so you can avoid that shifting you were doing.

---

Posted at 2023-08-15 by @fanoush

I've included my code below where I store data in the "compressedData = []" array.

so this is not about Uint8Array size since in your code it is only temporary variable. The remaining blocks going down by three is about the ordinary array having three integers added there by push.apply

> var d=[]
>d.push.apply(d,new Uint8Array([1, 2, 3]))
=3
>d
=[ 1, 2, 3 ]
>trace(d)
#30[r1,l1] Array(3) [
  #66[r1,l2] Name Integer 0 = int 1
  #62[r1,l2] Name Integer 1 = int 2
  #70[r1,l2] Name Integer 2 = int 3
]
>E.getSizeOf(d)
=4
>d.push.apply(d,new Uint8Array([1, 2, 3]))
=6
>d
=[ 1, 2, 3, 1, 2, 3 ]
>E.getSizeOf(d)
=7
>d.push.apply(d,new Uint8Array([1, 2, 3]))
=9
>E.getSizeOf(d)
=10
>trace(d)
#30[r1,l1] Array(9) [
  #66[r1,l2] Name Integer 0 = int 1
  #62[r1,l2] Name Integer 1 = int 2
  #70[r1,l2] Name Integer 2 = int 3
  #77[r1,l2] Name Integer 3 = int 1
  #78[r1,l2] Name Integer 4 = int 2
  #79[r1,l2] Name Integer 5 = int 3
  #83[r1,l2] Name Integer 6 = int 1
  #84[r1,l2] Name Integer 7 = int 2
  #82[r1,l2] Name Integer 8 = int 3
]

---

Posted at 2023-08-15 by user156224

Thank you both...you really helped me a lot with your explanations.