Replace node Buffers with Uint8Arrays

[achingbrain]

In order to treat browsers as first-class citizens, we should not use modules from node core unless we can absolutely guarantee that the code we are writing will not run in a browser.

The next generation of multiformats & IPLD modules will also be Uint8Arrays all the way down so embracing more modern formats will mean the upgrade path easier to follow when they are ready for use.

Modern JavaScript runtimes have TypedArrays such as Uint8Array backed by ArrayBuffers, as well as DataViews that wrap an ArrayBuffer and allow you to do bit-twiddly operations like writing little endian 32 bit floats at arbitrary offsets, etc.

Node's Buffer module pre-dates all of this but since node v3 it extends the Uint8Array class, so we should be able to treat Buffers as Uint8Arrays internally in our stack, though some dependencies may require Buffers as input until they too can be refactored.

PRs to modules

IPFS

• chore: update deps js-ipfsd-ctl#531
• fix: replace node buffers with uint8arrays js-ipfs-block-service#105
• chore: update repo migrations dep js-ipfs-repo#253
• fix: replace node buffers with uint8arrays js-ipfs-repo-migrations#25
• fix: replace node buffers with uint8arrays js-ipfs-unixfs#69
• fix: replace node buffers with uint8arrays js-ipfs-bitswap#251
• fix: replace node buffers with uint8arrays js-ipns#67
• fix: replace node buffers with uint8arrays js-ipfs-http-response#55
• refactor: update to js-ipfs@0.50.0 with Uint8Arrays interop#107
• feat: store pins in datastore instead of a DAG #2771
• chore: libp2p 0.29 integration #3237

IPLD

• fix: convert output of multihash.decode to buffer ipld/js-ipld-git#71
• fix: create buffers from decoded multihash digests ipld/js-ipld#287
• fix: replace node buffers with uint8arrays ipld/ipld-in-memory#18
• fix: update all deps to versions that return uint8arrays ipld/js-ipld#286
• fix: replace node buffers with uint8arrays ipld/js-ipld-block#54
• fix: update to latest cids and uint8array compatible deps ipld/js-ipld-zcash#60
• fix: replace node buffers with uint8arrays ipld/js-ipld-git#70
• fix: replace node buffers with uint8arrays ipld/js-ipld-ethereum#69
• fix: replace node buffers with uint8array ipld/js-ipld-bitcoin#65
• docs: replace node buffers with uint8arrays ipld/interface-ipld-format#55
• fix: replace node buffers with uint8arrays ipld/js-ipld-dag-cbor#134

libp2p

• fix: replace node buffers with uint8arrays libp2p/js-libp2p-mplex#114
• chore: update deps libp2p/js-libp2p-tcp#135
• fix: replace node buffers with uint8arrays libp2p/js-libp2p-bootstrap#106
• fix: replace node buffers with uint8arrays libp2p/js-libp2p-secio#124
• chore: update deps libp2p/js-libp2p-record#24
• chore: update deps libp2p/js-libp2p-tcp#134
• chore: update deps libp2p/js-libp2p-utils#9
• chore: update deps libp2p/js-libp2p-interfaces#57
• fix: replace node buffers with uint8arrays libp2p/js-libp2p-kad-dht#202
• fix: replace node buffers with uint8arrays libp2p/js-peer-id#127
• fix: replace node buffers with uint8arrays libp2p/js-libp2p-crypto#180
• fix: replace node buffers with uint8arrays libp2p/js-libp2p-floodsub#109
• fix: replace node buffers with uint8arrays libp2p/js-libp2p-pubsub#70
• fix: replace node buffers with uint8arrays ChainSafe/js-libp2p-gossipsub#118
• chore: upgrade deps libp2p/js-libp2p-mdns#97
• fix: replace node buffers with uint8arrays libp2p/js-libp2p-webrtc-star#244
• fix: replace node buffers with uint8arrays libp2p/js-libp2p-websockets#115
• chore: update libp2p-tcp libp2p/js-libp2p-interfaces#59
• chore: update deps to latest versions ChainSafe/js-libp2p-noise#70
• chore: update deps libp2p/js-libp2p-secio#125
• chore: update deps libp2p/js-libp2p-bootstrap#107
• chore: update deps libp2p/js-libp2p-kad-dht#203
• chore: update deps libp2p/js-libp2p-tcp#136
• fix: replace node buffers with uint8arrays libp2p/js-libp2p-delegated-content-routing#42
• chore: update deps libp2p/js-libp2p-delegated-peer-routing#35
• fix: replace node buffers with uint8arrays libp2p/js-libp2p#730
• fix: replace node buffers with uint8arrays libp2p/js-libp2p-daemon#41
• fix: replace node buffers with uint8arrays libp2p/js-libp2p-daemon-client#42
• fix: replace node buffers with uint8arrays libp2p/interop#42

Multiformats

• fix: replace node buffers with uint8arrays multiformats/js-cid#117
• chore: update deps multiformats/js-multiaddr-to-uri#45
• fix: replace node buffers with uint8arrays multiformats/js-mafmt#59
• fix: replace node buffers with uint8arrays multiformats/js-multiaddr#140
• fix: replace node buffers with uint8arrays multiformats/js-multicodec#63
• fix: replace node buffers with uint8arrays multiformats/js-multihashing-async#78
• fix: replace node buffers with uint8arrays multiformats/js-multihash#92
• fix: replace node buffers with uint8arrays multiformats/js-multistream-select#90
• chore: update deps multiformats/js-uri-to-multiaddr#12
• fix: replace node buffers with uint8arrays multiformats/js-cid-tool#10

Other

• fix: replace node buffers with uint8arrays ipfs-shipyard/is-ipfs#34
• fix: unhandled promise rejection when async work done after handshake jacobheun/it-handshake#2
• chore: update deps mikeal/dag-cbor-links#61

[Weedshaker]

We could also adjust the examples. Eg.: https://github.com/ipfs/js-ipfs/tree/master/examples/ipfs-101

From
console.log('Added file contents:', Buffer.concat(chunks).toString())

To
console.log('Added file contents:', chunks.toString())

or at least make a note about the usage within Node/Browser.

[achingbrain]

@Weedshaker when this effort bubbles up to a PR for this repo, yes, the documentation will be updated.

[jacobheun]

Some notes on transitioning:

• If Buffer.allocUnsafe is being used it should be retained for Node.js. There is no alternative for Uint8Array and there is a significant performance drop switching to Uint8Array only. Hot paths in Node.js should continue to use Buffer.allocUnsafe.
• Buffer.slice(...) should become Uint8Array.subarray(...). Uint8Array's slice does a clone instead of referencing the underlying memory, which Buffer's slice and subarray does. subarray can be used for both Buffer and Uint8Array.

[achingbrain]

It's worth noting that .subarray cannot be used with BufferLists, which occasionally complicates matters. .slice can, and it'll give you a no-copy view of an underlying buffer, unless the requested range falls over two internal buffers, in which case it's a copy operation.

[mikeal]

BufferList has “shallowSlice” which is roughly what subarray does, just returning a new BufferList instead of Uint8array. But I would imagine that what you often want from BufferList is still .slice() because you probably need a complete binary value.

…

-Mikeal

________________________________ From: Alex Potsides <notifications@github.com> Sent: Tuesday, August 11, 2020 7:53:04 AM To: ipfs/js-ipfs <js-ipfs@noreply.github.com> Cc: Subscribed <subscribed@noreply.github.com> Subject: Re: [ipfs/js-ipfs] Replace node Buffers with Uint8Arrays (#3220) It's worth noting that .subarray cannot be used with BufferList<https://www.npmjs.com/package/bl>s, which occasionally complicates matters. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#3220 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAAAEQ6T3ERS552OHDBPVPLSAFLNBANCNFSM4P2DZHJQ>.

[wemeetagain]

Is BufferList usage to be replaced with Uint8ArrayList (I don't think it currently exists)?

[achingbrain]

The push here is really to have our ecosystem modules accept Uint8Arrays in place of Buffers and have them work without throwing errors.

For converting BufferList to Uint8ArrayList I think you'd have to do it for a module if it makes sense, but it may not - lots of our modules wrap other modules (crypto ones in particular) that only accept Buffers/BufferLists so from a practical point of view it may not be necessary or desirable until/if these wrapped modules can be refactored to accept Uint8Arrays.

[achingbrain]

This shipped in js-ipfs@0.50.0! We did it!