Skip to content

Add RLP library #5680

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 75 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 68 commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
9eb5f1c
Add memory utils
ernestognw Sep 4, 2024
2d397f4
Fix tests upgradeable
ernestognw Sep 4, 2024
2a0fb7e
Add docs
ernestognw Sep 5, 2024
a7e61c3
Make use of the library
ernestognw Sep 5, 2024
1aae8bb
Update docs/modules/ROOT/pages/utilities.adoc
ernestognw Oct 9, 2024
1b2679a
Merge branch 'master' into utils/memory
Amxx Mar 6, 2025
d514606
fix tests
Amxx Mar 6, 2025
14fa04e
Update contracts/utils/Memory.sol
ernestognw May 7, 2025
d0d55fc
Update contracts/utils/Memory.sol
arr00 May 7, 2025
7b3cb66
Add RLP library
ernestognw May 10, 2025
95149f8
Add TrieProof library
ernestognw May 11, 2025
ad5d4ac
up
ernestognw May 11, 2025
18540ef
Add docs
ernestognw May 11, 2025
163f27c
Workaround stack too deep
ernestognw May 24, 2025
c484289
Add Changesets
ernestognw May 29, 2025
e0d4790
Add more changesets
ernestognw Jun 7, 2025
a6f9053
Add FV and fuzz tests
ernestognw Jun 7, 2025
e2b5e4c
Merge branch 'master' into feature/rlp
ernestognw Jun 7, 2025
203d1a2
up
ernestognw Jun 7, 2025
48eabc1
docs
ernestognw Jun 7, 2025
63ced95
up pragma
ernestognw Jun 7, 2025
f342756
Add missing Bytes test
ernestognw Jun 7, 2025
23dba37
Add unit tests
ernestognw Jun 7, 2025
0cacca2
up pragma
ernestognw Jun 7, 2025
831e8ab
Move TrieProof
ernestognw Jun 7, 2025
5da111f
Fix countLeadingZeroes
ernestognw Jun 8, 2025
ba2293e
nits
ernestognw Jun 8, 2025
9409bc6
Improve
ernestognw Jun 8, 2025
e740dac
Fix
ernestognw Jun 8, 2025
0332ffe
Add Memory.sol library
ernestognw Jun 8, 2025
608e3cd
Merge branch 'master' into utils/memory
ernestognw Jun 8, 2025
ac92bb4
up
ernestognw Jun 8, 2025
6094bb7
Merge branch 'master' into utils/memory
ernestognw Jun 8, 2025
6bb96d5
WIP: Add more Memory functions
ernestognw Jun 8, 2025
860e5a8
up
ernestognw Jun 8, 2025
ecdb768
revert
ernestognw Jun 8, 2025
95907aa
Update docs
ernestognw Jun 8, 2025
124ccee
Nit
ernestognw Jun 8, 2025
c3237df
Finish fuzz tests and FV
ernestognw Jun 9, 2025
27f0a9b
up
ernestognw Jun 9, 2025
282ce39
up
ernestognw Jun 9, 2025
bdd2cf1
Add operations to Math.sol
ernestognw Jun 9, 2025
42c79f1
Add new equal, nibbles and countLeadingZeroes functions
ernestognw Jun 9, 2025
5754ab8
Rename countLeadingZeroes to clz
ernestognw Jun 9, 2025
44f0e14
up
ernestognw Jun 9, 2025
05c73bd
Pragma changes
ernestognw Jun 9, 2025
3a6fbf6
up
ernestognw Jun 9, 2025
e67e8b4
up
ernestognw Jul 4, 2025
3385718
Rename to in Math library and update corresponding tests for consis…
ernestognw Jul 9, 2025
40d7922
Update return types of reverseBits functions to match their respectiv…
ernestognw Jul 9, 2025
89860bc
Refactor reverseBits functions in to use fixed-size byte types
ernestognw Jul 9, 2025
9b58730
Test nits
ernestognw Jul 9, 2025
77ffa8c
Simplify
ernestognw Jul 9, 2025
ce91c80
up
ernestognw Jul 9, 2025
b3e3add
Move reverse functions to Bytes.sol
ernestognw Jul 9, 2025
2f3107c
Move Bytes.t.sol
ernestognw Jul 9, 2025
4383e01
Merge branch 'master' into feat/bytes-rlp
ernestognw Jul 9, 2025
5a44b11
up
ernestognw Jul 9, 2025
d6db2d7
Document
ernestognw Jul 9, 2025
3847050
Remove extra functions
ernestognw Jul 9, 2025
4fd1947
Update docs
ernestognw Jul 9, 2025
c4e0375
up
ernestognw Jul 9, 2025
acb14cb
Merge branch 'utils/memory' into feature/rlp
ernestognw Jul 9, 2025
2208006
Merge branch 'feat/math-reverse-bits' into feature/rlp
ernestognw Jul 9, 2025
13f4d8f
Merge branch 'feat/bytes-rlp' into feature/rlp
ernestognw Jul 9, 2025
502a520
Merge branch 'master' into feature/rlp
ernestognw Jul 12, 2025
aab9274
Merge branch 'master' into feature/rlp
Amxx Jul 15, 2025
aa26e48
up
Amxx Jul 15, 2025
948f0a1
Merge branch 'master' into feature/rlp
ernestognw Jul 31, 2025
d4bfb8b
Fix compilation
ernestognw Jul 31, 2025
138de7f
Remove dangling clz
ernestognw Jul 31, 2025
5efeb37
Make nibbles function private
ernestognw Jul 31, 2025
00ff228
Remove nibbles test
ernestognw Jul 31, 2025
c58c7fd
Remove TrieProof library
ernestognw Jul 31, 2025
d1aa944
Add some tests
ernestognw Aug 1, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/khaki-hats-leave.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'openzeppelin-solidity': minor
---

`Bytes`: Add a `nibbles` function to split each byte into two nibbles.
5 changes: 5 additions & 0 deletions .changeset/lovely-cooks-add.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'openzeppelin-solidity': minor
---

`RLP`: Add library for Ethereum's Recursive Length Prefix encoding/decoding.
5 changes: 5 additions & 0 deletions .changeset/shaky-phones-mix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'openzeppelin-solidity': minor
---

`TrieProof`: Add library for verifying Ethereum Merkle-Patricia trie inclusion proofs.
5 changes: 5 additions & 0 deletions .changeset/ten-steaks-try.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'openzeppelin-solidity': minor
---

`Bytes`: Add an `equal` function to compare byte buffers.
5 changes: 5 additions & 0 deletions .changeset/whole-cats-find.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'openzeppelin-solidity': minor
---

`Bytes`: Add a `clz` function to count the leading zero bytes in a `uint256` value.
22 changes: 22 additions & 0 deletions contracts/utils/Bytes.sol
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,28 @@ library Bytes {
return result;
}

/// @dev Split each byte in `value` into two nibbles (4 bits each).
function nibbles(bytes memory value) internal pure returns (bytes memory) {
uint256 length = value.length;
bytes memory nibbles_ = new bytes(length * 2);
for (uint256 i = 0; i < length; i++) {
(nibbles_[i * 2], nibbles_[i * 2 + 1]) = (value[i] & 0xf0, value[i] & 0x0f);
}
return nibbles_;
}
Copy link

@0xClandestine 0xClandestine Jul 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The toNibbles function converts a bytes array like 0xABCD into a nibble-expanded format like 0xA00BC00D. The existing implementation does this by iterating linearly over the input, which works correctly but is extremely inefficient.

My approach processes the input in parallel, reducing time complexity from O(n) to O(n / 32). This is inspired by bit-hacking techniques for interleaving bits, as described in the following resources:

https://stackoverflow.com/questions/38881877/bit-hack-expanding-bits
https://graphics.stanford.edu/~seander/bithacks.html#InterleaveBMN

    // 0xABCD -> 0xA0B0C0D0
    function expandNibbles(uint128 x) internal pure returns (uint256) {
        uint256 y = uint256(x);
        y = (y | (y << 64)) & 0x0000000000000000FFFFFFFFFFFFFFFF0000000000000000FFFFFFFFFFFFFFFF;
        y = (y | (y << 32)) & 0x00000000FFFFFFFF00000000FFFFFFFF00000000FFFFFFFF00000000FFFFFFFF;
        y = (y | (y << 16)) & 0x0000FFFF0000FFFF0000FFFF0000FFFF0000FFFF0000FFFF0000FFFF0000FFFF;
        y = (y | (y << 8)) & 0x00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF;
        y = (y | (y << 4)) & 0x0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F;
        return y;
    }
    
    // Puts nibbles in their respective height (high/low). 
    // 0xA0B0C0D0 -> 0xA00BC00D
    function toNibbles(uint128 x) internal pure returns (uint256) {
        uint256 y = expandNibbles(x);
        uint256 high = y & 0x00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF;
        uint256 low = y & 0xFF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00;
        high = high >> 4;
        return high | low;
    }

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bringing it all together:

function toNibbles(bytes memory input) internal pure returns (bytes memory output) {
    /// @solidity memory-safe-assembly
    assembly {
        output := mload(0x40)
        mstore(output, mul(mload(input), 2))
        mstore(0x40, add(output, add(0x20, mul(mload(input), 2))))
        
        for { let i := 0 } lt(i, mload(input)) { i := add(i, 16) } {
            let x := shr(128, mload(add(add(input, 0x20), i)))
            x := and(or(x, shl(64, x)), 0x0000000000000000FFFFFFFFFFFFFFFF0000000000000000FFFFFFFFFFFFFFFF)
            x := and(or(x, shl(32, x)), 0x00000000FFFFFFFF00000000FFFFFFFF00000000FFFFFFFF00000000FFFFFFFF)
            x := and(or(x, shl(16, x)), 0x0000FFFF0000FFFF0000FFFF0000FFFF0000FFFF0000FFFF0000FFFF0000FFFF)
            x := and(or(x, shl(8, x)),  0x00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF)
            let mask := 0x0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F
            x := and(or(x, shl(4, x)),  mask)
            x := or(shl(4, and(shr(4, x), mask)), and(x, mask))
            mstore(add(add(output, 0x20), mul(i, 2)), x)
        }
    }
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can probably be simplified further by adjusting the first 4 masks, working on that now.


/**
* @dev Returns true if the two byte buffers are equal.
*/
function equal(bytes memory a, bytes memory b) internal pure returns (bool) {
return a.length == b.length && keccak256(a) == keccak256(b);
}

/// @dev Counts the number of leading zeros in a uint256.
function clz(uint256 x) internal pure returns (uint256) {
return Math.ternary(x == 0, 32, 31 - Math.log256(x));
}

/**
* @dev Moves the content of `buffer`, from `start` (included) to the end of `buffer` to the start of that buffer.
*
Expand Down
3 changes: 3 additions & 0 deletions contracts/utils/README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ Miscellaneous contracts and libraries containing utility functions you can use t
* {InteroperableAddress}: Library for formatting and parsing ERC-7930 interoperable addresses.
* {Blockhash}: A library for accessing historical block hashes beyond the standard 256 block limit utilizing EIP-2935's historical blockhash functionality.
* {Time}: A library that provides helpers for manipulating time-related objects, including a `Delay` type.
* {RLP}: Library for encoding and decoding data in Ethereum's Recursive Length Prefix format.

[NOTE]
====
Expand Down Expand Up @@ -143,3 +144,5 @@ Ethereum contracts have no native concept of an interface, so applications must
{{Blockhash}}

{{Time}}

{{RLP}}
Loading
Loading