Conversation
…tion TxHash optimization: - Write tx data directly to sha256.New() instead of serializing into an intermediate bytes.Buffer, eliminating a per-tx buffer allocation proportional to the transaction size. - Cache computed hash in MsgTx.cachedHash field, invalidated by AddTxIn/AddTxOut. Applied to both MsgTx and MsgExtendedTx. Block deserialization optimization: - Add bsvdecodeWithScratch method that reads all scripts into a shared scratch buffer (reused across transactions) instead of allocating a fresh buffer per large script via scriptFreeList.Borrow. - MsgBlock.Bsvdecode passes a shared scratch buffer to each tx, growing it only when a script larger than any previous one is encountered. After each tx, the buffer is reset (len=0) but capacity is preserved. - Pre-allocate MsgTx structs contiguously in MsgBlock.Bsvdecode (one slice instead of per-tx heap allocations). For a 3.64 GB testnet block (28,672 txs): - Block deserialization: 7,514 MB → 3,790 MB (-49.6%) - TxHash: 3,758 MB → 0 MB (eliminated from allocation profile)
Pre-declare scriptLen variable to avoid shadowing err from outer scope in the output script reading loop.
|
icellan
added a commit
to bsv-blockchain/teranode
that referenced
this pull request
Feb 27, 2026
Replace subtreeData.Serialize() + storer.Write(bytes) with subtreeData.WriteTransactionsToWriter(storer, 0, length), streaming transaction data directly to the blob store FileStorer pipe instead of serializing into a large intermediate byte slice. The FileStorer already implements io.Writer via a pipe connected to SetFromReader, and the file store guarantees atomic writes via temp file + rename. Add HandleBlockDirect memory profiling test for large testnet blocks. Depends on: - bsv-blockchain/go-bt#117 (SerializeTo) - bsv-blockchain/go-subtree (streaming WriteTransactionsToWriter) - bsv-blockchain/go-wire#106 (TxHash + block deserialization)
icellan
added a commit
to bsv-blockchain/teranode
that referenced
this pull request
Feb 27, 2026
Replace subtreeData.Serialize() + storer.Write(bytes) with subtreeData.WriteTransactionsToWriter(storer, 0, length), streaming transaction data directly to the blob store FileStorer pipe instead of serializing into a large intermediate byte slice. The FileStorer already implements io.Writer via a pipe connected to SetFromReader, and the file store guarantees atomic writes via temp file + rename. Add HandleBlockDirect memory profiling test for large testnet blocks. Depends on: - bsv-blockchain/go-bt#117 (SerializeTo) - bsv-blockchain/go-subtree (streaming WriteTransactionsToWriter) - bsv-blockchain/go-wire#106 (TxHash + block deserialization)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Summary
Two optimizations targeting block deserialization and transaction hashing, measured against a real 3.64 GB testnet block (28,672 txs, block 1681787).
1. Streaming TxHash (eliminates per-tx buffer allocation)
MsgTx.TxHash()previously serialized the entire transaction into abytes.Bufferand then hashed the buffer. For large transactions (100+ KB), this allocated a buffer proportional to the tx size just to compute a hash.Fix: Write transaction data directly to
sha256.New()(which implementsio.Writer) instead of an intermediate buffer. The double-SHA256 is computed assha256(sha256(tx))using a stack-allocated[32]bytefor the intermediate hash.Also adds a
cachedHash *chainhash.Hashfield toMsgTxthat lazily caches the computed hash, invalidated byAddTxIn()/AddTxOut(). Applied the same streaming approach toMsgExtendedTx.TxHash().Result:
MsgTx.TxHashdropped from 3,758 MB to 0 MB in the allocation profile.2. Block-level scratch buffer for deserialization (eliminates temporary script allocations)
During
MsgTx.Bsvdecode, each script is read into a temporary buffer viascriptFreeList.Borrow(), then copied into a per-tx contiguous buffer. Scripts larger than 512 bytes bypass the pool and get a freshmake([]byte, size)allocation. For a 3.64 GB block, this produced ~3.7 GB of temporary allocations that were immediately discarded after copying.Fix: Add
bsvdecodeWithScratch()method that reads all scripts into a shared scratch buffer. The buffer is passed fromMsgBlock.Bsvdecodeand reused across all transactions:len=0for each tx (capacity preserved)Also pre-allocates
MsgTxstructs contiguously inMsgBlock.Bsvdecode(make([]MsgTx, txCount)) instead of individual stack-escape allocations per tx.Single-tx
Bsvdecode(non-block path) remains unchanged, using the existing script pool.Result: Block deserialization allocations dropped from 7,514 MB to 3,790 MB (-49.6%). The remaining 3,716 MB is the irreducible per-tx contiguous script buffer (the actual script data that must live somewhere).
Test changes
msg_block_test.go: AddedclearBlockTxCacheshelper to clearcachedHashbeforereflect.DeepEqualcomparisonsmsg_tx_test.go: ClearcachedHashbeforereflect.DeepEqualcomparisonsmsg_extended_tx_test.go: No test changes needed (MsgExtendedTxhas no cached hash field)Test plan
go test -count=1 -short ./...)TestTxTxHashandTestExtendedTxTxHashtestsTestBlockSerialize*tests