Add RLP library #5680
Add RLP library #5680
Conversation
Co-authored-by: Hadrien Croubois <[email protected]>
Co-authored-by: Arr00 <[email protected]>
🦋 Changeset detectedLatest commit: 8928039 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
| if (keyIndex == key.length) return _validateLastItem(node.decoded[radix], trieProof, i); | ||
|
|
||
| // Otherwise, continue down the branch specified by the next nibble in the key | ||
| uint8 branchKey = uint8(key[keyIndex]); |
There was a problem hiding this comment.
On my point before I don't think it's necessary to convert the key to nibbles beforehand, makes more sense to extract the Nth nibble here.
contracts/utils/Bytes.sol
Outdated
| function nibbles(bytes memory value) internal pure returns (bytes memory) { | ||
| uint256 length = value.length; | ||
| bytes memory nibbles_ = new bytes(length * 2); | ||
| for (uint256 i = 0; i < length; i++) { | ||
| (nibbles_[i * 2], nibbles_[i * 2 + 1]) = (value[i] & 0xf0, value[i] & 0x0f); | ||
| } | ||
| return nibbles_; | ||
| } |
There was a problem hiding this comment.
The toNibbles function converts a bytes array like 0xABCD into a nibble-expanded format like 0xA00BC00D. The existing implementation does this by iterating linearly over the input, which works correctly but is extremely inefficient.
My approach processes the input in parallel, reducing time complexity from O(n) to O(n / 32). This is inspired by bit-hacking techniques for interleaving bits, as described in the following resources:
https://stackoverflow.com/questions/38881877/bit-hack-expanding-bits
https://graphics.stanford.edu/~seander/bithacks.html#InterleaveBMN
// 0xABCD -> 0xA0B0C0D0
function expandNibbles(uint128 x) internal pure returns (uint256) {
uint256 y = uint256(x);
y = (y | (y << 64)) & 0x0000000000000000FFFFFFFFFFFFFFFF0000000000000000FFFFFFFFFFFFFFFF;
y = (y | (y << 32)) & 0x00000000FFFFFFFF00000000FFFFFFFF00000000FFFFFFFF00000000FFFFFFFF;
y = (y | (y << 16)) & 0x0000FFFF0000FFFF0000FFFF0000FFFF0000FFFF0000FFFF0000FFFF0000FFFF;
y = (y | (y << 8)) & 0x00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF;
y = (y | (y << 4)) & 0x0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F;
return y;
}
// Puts nibbles in their respective height (high/low).
// 0xA0B0C0D0 -> 0xA00BC00D
function toNibbles(uint128 x) internal pure returns (uint256) {
uint256 y = expandNibbles(x);
uint256 high = y & 0x00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF;
uint256 low = y & 0xFF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00;
high = high >> 4;
return high | low;
}There was a problem hiding this comment.
Bringing it all together:
function toNibbles(bytes memory input) internal pure returns (bytes memory output) {
/// @solidity memory-safe-assembly
assembly {
output := mload(0x40)
mstore(output, mul(mload(input), 2))
mstore(0x40, add(output, add(0x20, mul(mload(input), 2))))
for { let i := 0 } lt(i, mload(input)) { i := add(i, 16) } {
let x := shr(128, mload(add(add(input, 0x20), i)))
x := and(or(x, shl(64, x)), 0x0000000000000000FFFFFFFFFFFFFFFF0000000000000000FFFFFFFFFFFFFFFF)
x := and(or(x, shl(32, x)), 0x00000000FFFFFFFF00000000FFFFFFFF00000000FFFFFFFF00000000FFFFFFFF)
x := and(or(x, shl(16, x)), 0x0000FFFF0000FFFF0000FFFF0000FFFF0000FFFF0000FFFF0000FFFF0000FFFF)
x := and(or(x, shl(8, x)), 0x00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF)
let mask := 0x0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F
x := and(or(x, shl(4, x)), mask)
x := or(shl(4, and(shr(4, x), mask)), and(x, mask))
mstore(add(add(output, 0x20), mul(i, 2)), x)
}
}
}There was a problem hiding this comment.
Can probably be simplified further by adjusting the first 4 masks, working on that now.
| function _sharedNibbleLength(bytes memory _a, bytes memory _b) private pure returns (uint256 shared_) { | ||
| uint256 max = Math.max(_a.length, _b.length); | ||
| uint256 length; | ||
| while (length < max && _a[length] == _b[length]) { | ||
| length++; | ||
| } | ||
| return length; | ||
| } |
There was a problem hiding this comment.
This can also be much more efficient by comparing 32 bytes at a time. Once a mismatch is found, use xor and clz to count leading matching bytes within the word.
O(n) -> O(n/32)
There was a problem hiding this comment.
function _getSharedNibbleLength(bytes memory _a, bytes memory _b) private pure returns (uint256 shared) {
uint256 minLen = _a.length < _b.length ? _a.length : _b.length;
unchecked {
// Compare 32 bytes at a time
for (; shared < minLen; shared += 32) {
uint256 wordA;
uint256 wordB;
assembly {
wordA := mload(add(_a, add(32, shared)))
wordB := mload(add(_b, add(32, shared)))
}
if (wordA != wordB) {
uint256 diff = wordA ^ wordB;
uint256 leadingBits = clz(diff); // clz counts bits
return shared + (leadingBits / 8);
}
}
// Fallback to byte-by-byte for remaining bytes
while (shared < minLen && _a[shared] == _b[shared]) shared++;
}
}There was a problem hiding this comment.
With proofs in calldata, and the above the optimizations I'm hitting 175k gas for an account proof verification (down from over 400k).
There was a problem hiding this comment.
Mind you I'm working off of optimisms implementation, this version looks a bit more expensive.
There was a problem hiding this comment.
Tests. Then I optimize
There was a problem hiding this comment.
No worries, just wanted to write this out while I had a minute.
There was a problem hiding this comment.
Yeah, thanks for the review though, it's just I haven't put time into making sure the implementation fully works.
It's a priority for 5.5 regardless. Just, you know, the release will take a couple of months 😓
contracts/utils/RLP.sol
Outdated
| // Long list | ||
| (offset, length) = _decodeLong(prefix - SHORT_LIST_OFFSET, item); | ||
| return (offset, length, ItemType.LIST_ITEM); |
There was a problem hiding this comment.
Feel like we should use an if statement with a revert case after.
| /// @dev Convenience method to encode a uint256 as RLP. See {_binaryBuffer}. | ||
| function encode(uint256 value) internal pure returns (bytes memory) { | ||
| return encode(_binaryBuffer(value)); | ||
| } | ||
|
|
||
| /// @dev Same as {encode-uint256-}, but for bytes32. | ||
| function encode(bytes32 value) internal pure returns (bytes memory) { | ||
| return encode(uint256(value)); | ||
| } |
There was a problem hiding this comment.
This is going to remove leeding zeros. is that desirable ?
There was a problem hiding this comment.
Yeah, this is desirable and part of the spec: https://ethereum.org/en/developers/docs/data-structures-and-encoding/rlp/
Positive integers must be represented in big-endian binary form with no leading zeroes (thus making the integer value zero equivalent to the empty byte array). Deserialized positive integers with leading zeroes must be treated as invalid by any higher-order protocol using RLP.
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the ✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
Requires:
Memoryutility library #5189reverseBitsoperations to Bytes.sol #5724clz(bytes)andclz(uint256)functions #5725PR Checklist
npx changeset add)