Payment transaction nodeAccountID becomes misaligned during multi-node query retries #1614
Description
Description
When executing a query with multiple nodes configured via SetNodeAccountIDs(), the Hedera Go SDK generates payment transactions that become misaligned with the actual target node during retry attempts. This causes queries to fail with INVALID_NODE_ACCOUNT errors because nodes reject payment transactions that specify a different node ID than the one receiving the request.
The Problem
The SDK's query.go implementation has a synchronization bug between payment transaction generation and node selection during retries:
- generatePayments() creates payment transactions for all nodes in the node list
- Returns only the last payment transaction created
- When advanceRequest() moves to the next node during retry, the payment transaction still references the wrong node
- Result: Payment says "for node 0.0.4" but the query is sent to node 0.0.5
Impact
- Multi-node queries fail intermittently with INVALID_NODE_ACCOUNT errors
- Network resilience is compromised (can't effectively retry across nodes)
- Applications must implement workarounds (query one node at a time)
- Affects all query types: AccountInfoQuery, ContractInfoQuery, etc.
Root Cause Analysis
Location: vendor/github.com/hashgraph/hedera-sdk-go/v2/query.go
Line 248-254: makeRequest() regenerates all payment transactions but only uses the last one:
func (q *Query) makeRequest() interface{} {
if q.client != nil && q.isPaymentRequired {
tx, err := q.generatePayments(q.client, q.queryPayment) // ← Generates ALL payments
if err != nil {
return q.pb
}
q.pbHeader.Payment = tx // ← Uses only the LAST payment!
}
return q.pb
}
Line 232-242: generatePayments() loops through all nodes but returns only the last transaction:
func (q *Query) generatePayments(client *Client, cost Hbar) (*services.Transaction, error) {
var tx *services.Transaction
var err error
for _, nodeID := range q.nodeAccountIDs.slice { // ← Loops through ALL nodes
txnID := TransactionIDGenerate(client.operator.accountID)
tx, err = _QueryMakePaymentTransaction(
txnID,
nodeID.(AccountID), // ← Creates payment for this specific node
client.operator,
cost,
)
// ...
}
return tx, nil // ← Returns only the LAST transaction created!
}
The Bug: When advanceRequest() cycles to the next node during retry, makeRequest() regenerates payments but always uses the last one, causing a mismatch.
Steps to reproduce
Here is a sample code to explain, but it's not what caused me to encounter it:
package main
import (
"fmt"
"github.com/hashgraph/hedera-sdk-go/v2"
)
func main() {
// Setup client
client := hedera.ClientForTestnet()
client.SetOperator(
hedera.AccountID{Shard: 0, Realm: 0, Account: 2},
hedera.PrivateKeyFromString("your-operator-private-key"),
)
// Configure multiple nodes for retry
nodeAccountIDs := []hedera.AccountID{
{Shard: 0, Realm: 0, Account: 3},
{Shard: 0, Realm: 0, Account: 4},
{Shard: 0, Realm: 0, Account: 5},
}
// Create query with multiple target nodes
query := hedera.NewAccountInfoQuery().
SetAccountID(hedera.AccountID{Shard: 0, Realm: 0, Account: 2}).
SetNodeAccountIDs(nodeAccountIDs).
SetMaxRetry(5)
// Execute - SDK will retry across nodes, triggering the bug
_, err := query.Execute(client)
if err != nil {
fmt.Printf("Query failed: %v\n", err)
}
}
Expected Behavior
When the query retries to different nodes, each payment transaction's nodeAccountID should match the target node:
Attempt 1: Query → node 0.0.3, Payment nodeAccountID: 0.0.3 ✓
Attempt 2: Query → node 0.0.4, Payment nodeAccountID: 0.0.4 ✓
Attempt 3: Query → node 0.0.5, Payment nodeAccountID: 0.0.5 ✓
Actual Behavior
The payment transaction's nodeAccountID doesn't match the query target:
Attempt 1: Query → node 0.0.3, Payment nodeAccountID: 0.0.5 ✗ INVALID_NODE_ACCOUNT
Attempt 2: Query → node 0.0.4, Payment nodeAccountID: 0.0.5 ✗ INVALID_NODE_ACCOUNT
Attempt 3: Query → node 0.0.5, Payment nodeAccountID: 0.0.5 ✓ (works by luck)
Evidence from Logs
Decoding the protobuf request shows the mismatch (it was trying to query using account 0.0.5, but in the transaction it had 0.0.4:
Query {
cryptogetAccountInfo {
header {
payment {
signedTransactionBytes {
bodyBytes {
nodeAccountID: 0.0.4 // ← Payment is for node 0.0.4
// ...
}
}
}
}
accountID: 0.0.2
}
}
Additional context
Proposed Fixes
Option 1: Use Payment Transaction Array Index
Match payment to node using the current retry index:
func (q *Query) makeRequest() interface{} {
if q.client != nil && q.isPaymentRequired {
// Generate payment transactions once
if len(q.paymentTransactions) == 0 {
_, err := q.generatePayments(q.client, q.queryPayment)
if err != nil {
return q.pb
}
}
// Use the payment transaction matching the current node index
currentIndex := q.nodeAccountIDs.index
if currentIndex < len(q.paymentTransactions) {
q.pbHeader.Payment = q.paymentTransactions[currentIndex]
}
}
return q.pb
}
Option 2: Generate Payment Per Node On-Demand (no need for generatePayments method)
func (q *Query) makeRequest() interface{} {
if q.client != nil && q.isPaymentRequired {
// Get the current node being queried
currentNodeID := q.nodeAccountIDs.slice[q.nodeAccountIDs.index].(AccountID)
// Generate payment transaction for ONLY the current node
txnID := TransactionIDGenerate(q.client.operator.accountID)
tx, err := _QueryMakePaymentTransaction(
txnID,
currentNodeID, // ← Always matches the query target
q.client.operator,
q.queryPayment,
)
if err != nil {
return q.pb
}
q.pbHeader.Payment = tx
}
return q.pb
}
Advantages of Option 2:
- Simpler logic (no array management)
- Each payment is fresh for the current node
- More resilient to internal state changes
Workaround for Applications
Until the SDK is fixed, applications can work around this by querying one node at a time (this is what I am using atm):
// Manually implement node retry logic
for _, nodeID := range nodeAccountIDs {
query := hedera.NewAccountInfoQuery().
SetAccountID(accountID).
SetNodeAccountIDs([]hedera.AccountID{nodeID}). // ← ONE node at a time
SetMaxRetry(1) // Only retry once per node
result, err = query.Execute(client)
if err == nil {
break // Success!
}
// Log and try next node
}
Hedera network
other
Version
v2.50.0
Operating system
Linux