f_reward Inflated by Consolidated Amounts (Post-Electra Validators)

Summary

Validators that are targets of consolidations (post-Electra) have their f_reward values inflated by exactly the consolidated amount (~32 ETH) due to incorrect timing when subtracting ConsolidatedAmounts from the epoch reward calculation.

Problem Description

Symptoms

• Validators with effective_balance > 32 ETH show f_reward >> f_max_reward
• Typically f_reward is inflated by exactly 32 ETH (32,000,000,000 gwei)
• Only affects validators with withdrawal_prefix = 0x02 (compounding validators)
• Started appearing post-Electra (epoch >= 411389)

Root Cause

File: pkg/spec/metrics/standard.go:32
Function: EpochReward()

func (p StateMetricsBase) EpochReward(valIdx phase0.ValidatorIndex) int64 {
    consolidatedAmount, ok := p.CurrentState.ConsolidatedAmounts[valIdx]  // ← BUG
    if !ok {
        consolidatedAmount = 0
    }
    
    depositedAmount, ok := p.CurrentState.DepositedAmounts[valIdx]
    if !ok {
        depositedAmount = 0
    }
    depositedAmount += p.NextState.Deposits[valIdx]
    
    reward := int64(p.NextState.Balances[valIdx]) - int64(p.CurrentState.Balances[valIdx])
    reward += int64(p.NextState.Withdrawals[valIdx])
    reward -= int64(depositedAmount)
    reward -= int64(consolidatedAmount)  // ← Subtracts 0 when should subtract 32 ETH
    return reward
}

The bug: Uses CurrentState.ConsolidatedAmounts instead of NextState.ConsolidatedAmounts

Why This Happens

When validator B is the target of a consolidation from validator A:

Before consolidation (CurrentState - epoch N):

• Balances[B] = 32 ETH
• ConsolidatedAmounts[B] = 0 (doesn't exist in CurrentState yet)

After consolidation (NextState - epoch N+1):

• Balances[B] = 64 ETH (received 32 ETH from validator A)
• ConsolidatedAmounts[B] = 32 ETH (tracked in NextState)

Incorrect calculation (current code):

reward = NextState.Balances[B] - CurrentState.Balances[B] - CurrentState.ConsolidatedAmounts[B]
reward = 64e9 - 32e9 - 0
reward = 32,000,000,000 gwei  ← WRONG (32 ETH inflation)

Correct calculation (should be):

reward = NextState.Balances[B] - CurrentState.Balances[B] - NextState.ConsolidatedAmounts[B]
reward = 64e9 - 32e9 - 32e9
reward = 0 gwei  ← CORRECT

Impact

Severity: CRITICAL

Affected Data

• Table: t_validator_rewards_summary
• Column: f_reward
• Affected validators: All validators that are targets of consolidations
• Timeframe: Post-Electra upgrade (epoch >= 411389, ~December 2024)

Cascading Effects

Corrupted f_reward values propagate to:

1. t_pool_summary.aggregated_rewards (summed across pool)
2. Bindeth API APR calculations
3. Dashboard displays showing impossible APR values
4. Analytics and historical data

Example from Production

Validator receives consolidation at epoch X:

• Expected: f_max_reward = 108,000 gwei, f_reward ≈ 0-108,000 gwei
• Actual: f_max_reward = 108,000 gwei, f_reward = 32,108,000,000 gwei

This creates the impossible scenario where f_reward > f_max_reward * 297,000, breaking all downstream calculations.

Proposed Solution

Fix (1 line change)

Change line 32 in pkg/spec/metrics/standard.go from:

consolidatedAmount, ok := p.CurrentState.ConsolidatedAmounts[valIdx]

To:

consolidatedAmount, ok := p.NextState.ConsolidatedAmounts[valIdx]

Rationale

The consolidated amount is applied to the balance in NextState, so it should be tracked in NextState.ConsolidatedAmounts. This aligns with how Deposits are handled (line 42):

depositedAmount += p.NextState.Deposits[valIdx]  // Uses NextState, not CurrentState

Testing

Before Fix

Run goteth on a range containing consolidations (epoch >= 411389) and verify:

SELECT COUNT(*) FROM t_validator_rewards_summary 
WHERE f_reward > f_max_reward * 2 
  AND f_epoch >= 411389;
-- Should return > 0 (bug exists)

After Fix

Re-run same epoch range and verify:

SELECT COUNT(*) FROM t_validator_rewards_summary 
WHERE f_reward > f_max_reward * 2 
  AND f_epoch >= 411389 
  AND f_withdrawal_prefix = 2;
-- Should return 0 (bug fixed)

Edge Cases to Test

1. Validator receives multiple consolidations in sequence
2. Validator is both source and target of different consolidations
3. Validator with effective_balance > 64 ETH (multiple consolidations)

Migration Plan

Phase 1: Fix and Deploy

1. Apply fix to pkg/spec/metrics/standard.go
2. Test on development backfill (epochs 411389 - current)
3. Verify no f_reward > f_max_reward anomalies for consolidated validators

Phase 2: Data Cleanup

1. Identify all corrupted epochs:

   SELECT DISTINCT f_epoch 
   FROM t_validator_rewards_summary 
   WHERE f_reward > f_max_reward * 2 AND f_epoch >= 411389
   ORDER BY f_epoch;

2. Delete corrupted data:

   DELETE FROM t_validator_rewards_summary WHERE f_epoch IN (...);
   DELETE FROM t_pool_summary WHERE f_epoch IN (...);

3. Re-index those epochs with fixed goteth version

Phase 3: Validation

1. Verify APR calculations are reasonable
2. Check pool summaries for affected pools
3. Monitor dashboard displays

Related Issues

• Related to Electra upgrade and consolidation feature (EIP-7251)
• May interact with Issue UInt64 Underflow in t_pool_summary.aggregated_rewards #214 (UInt64 underflow) when pool aggregations include these inflated values

Related Code

• Consolidation processing: pkg/spec/metrics/state_electra.go:586-615
• State setup: pkg/spec/state.go:118
• Reward calculation: pkg/spec/metrics/state_altair.go:305 (assigns Reward field)

References

• Ethereum Consolidation Spec: https://github.com/ethereum/consensus-specs/blob/dev/specs/electra/beacon-chain.md#new-process_pending_consolidations
• EIP-7251 (Increase MAX_EFFECTIVE_BALANCE): https://eips.ethereum.org/EIPS/eip-7251

Additional Notes

This bug specifically affects target validators in consolidations. Source validators (the ones being consolidated away) are correctly handled as they are exited and their rewards stop being calculated.

The bug is deterministic and reproducible for any consolidation event, making it straightforward to test and verify the fix.