Stack corruption with go upgrade

[rkuris]

After upgrading to go1.25.7, we're seeing failures like this one:

https://github.com/ava-labs/avalanchego/actions/runs/21972259848/job/63476193719

goroutine 47097 gp=0xc00b2268c0 m=0 mp=0x102708300 [running]:
runtime.sigpanic()
	/Users/runner/hostedtoolcache/go/1.25.7/arm64/src/runtime/signal_unix.go:906 +0x35c fp=0x16fb3d950 sp=0x16fb3d950 pc=0x1003415bc
runtime: g 47097: unexpected return pc for runtime.sigpanic called from 0x10140af84
stack: frame={sp:0x16fb3d950, fp:0x16fb3d950} stack=[0xc0215f4000,0xc0215f5000)

created by github.com/ava-labs/firewood-go-ethhash/ffi.(*Database).Close in goroutine 47117
	/Users/runner/go/pkg/mod/github.com/ava-labs/firewood-go-ethhash/ffi@v0.1.0/firewood.go:436 +0xa4

goroutine 1 gp=0xc0000021c0 m=nil [chan receive]:

Also there are instances of stack overflow:

runtime: newstack sp=0x16fb3d950 stack=[0xc0215f4000, 0xc0215f5000]
	morebuf={pc:0x10140af84 sp:0x16fb3d950 lr:0x0}
	sched={pc:0x1003415bc sp:0x16fb3d950 lr:0x10140af84 ctxt:0x0}
runtime: gp=0xc00b2268c0, goid=47097, gp->status=0x2
 runtime: split stack overflow: 0x16fb3d950 < 0xc0215f4000
fatal error: runtime: split stack overflow

runtime stack:
runtime.throw({0x1015e6c93?, 0xc01477c000?})

Likely related to the slice changes which amplify the effect of pointer provenance issues. From the release notes:

Faster slices¶
The compiler can now allocate the backing store for slices on the stack in more situations, which improves performance. This change has the potential to amplify the effects of incorrect unsafe.Pointer usage, see for example issue 73199. In order to track down these problems, the bisect tool can be used to find the allocation causing trouble using the -compile=variablemake flag. All such new stack allocations can also be turned off using -gcflags=all=-d=variablemakehash=n.

[demosdemon]

I'm not sure the stack overflow is because of firewood. The linked error includes all of the goroutines executing at the time of the stack overflow panic and that indicates there are 189 ongoing goroutines spawned by github.com/ava-labs/avalanchego/graft/evm/core/state/snapshot.generateSnapshot. They all appear to be blocked on waiting for a chan recv.

The excessive amount of ongoing goroutines is the source of the stack overflow and the go runtime.GC() triggering the stack overflow is just a symptom. However, I don't know what's causing the snapshot goroutines to not deliver their aborting channel; so, I can't entirely guarantee that it's not related to firewood.

[demosdemon]

The stack traces also indicate that there was already an ongoing GC at the time we called go runtime.GC() and it was blocking on running finalizers. I can't find what called (*Database).Close() but I'm curious if the finalizer for Database is what triggered the close which may mean we were calling runtime.GC from inside an executing GC sweep.

The goroutine that spawned the go runtime.GC() had already moved beyond closing the firewood database:

goroutine 47117 gp=0xc010a1c700 m=nil [runnable]:
math/big.nat.cmp({0x0, 0x0, 0x0?}, {0xc015a9b858, 0x1, 0x1?})
	/Users/runner/hostedtoolcache/go/1.25.7/arm64/src/math/big/nat.go:151 +0x1cc fp=0xc00030c410 sp=0xc00030c410 pc=0x10058731c
math/big.(*Int).Cmp(0xc00012ac20, 0xc0144e1760)
	/Users/runner/hostedtoolcache/go/1.25.7/arm64/src/math/big/int.go:387 +0xc4 fp=0xc00030c470 sp=0xc00030c410 pc=0x10057e154
github.com/ava-labs/libevm/params.isBlockForked(...)
	/Users/runner/go/pkg/mod/github.com/ava-labs/libevm@v1.13.15-0.20260120173328-de5fd6fcd5df/params/config.go:788
github.com/ava-labs/libevm/params.(*ChainConfig).IsLondon(...)
	/Users/runner/go/pkg/mod/github.com/ava-labs/libevm@v1.13.15-0.20260120173328-de5fd6fcd5df/params/config.go:543
github.com/ava-labs/libevm/params.(*ChainConfig).IsShanghai(0xc000460340, 0xc0144e1760, 0xa)
	/Users/runner/go/pkg/mod/github.com/ava-labs/libevm@v1.13.15-0.20260120173328-de5fd6fcd5df/params/config.go:566 +0x58 fp=0xc00030c4a0 sp=0xc00030c470 pc=0x100ad87a8
github.com/ava-labs/libevm/params.(*ChainConfig).Rules(0xc000460340, 0xc0144e1760, 0x1, 0xa)
	/Users/runner/go/pkg/mod/github.com/ava-labs/libevm@v1.13.15-0.20260120173328-de5fd6fcd5df/params/config.go:941 +0x4cc fp=0xc00030c580 sp=0xc00030c4a0 pc=0x100adbbfc
github.com/ava-labs/avalanchego/graft/subnet-evm/core.hooks.OverrideNewEVMArgs({}, 0xc00d234c00)
	/Users/runner/work/avalanchego/avalanchego/graft/subnet-evm/core/evm.go:88 +0xcc fp=0xc00030c680 sp=0xc00030c580 pc=0x10122771c
github.com/ava-labs/avalanchego/graft/subnet-evm/core.(*hooks).OverrideNewEVMArgs(0x10367b860, 0xc00d234c00)
	<autogenerated>:1 +0x40 fp=0xc00030c6b0 sp=0xc00030c680 pc=0x1012a5e20
github.com/ava-labs/libevm/core/vm.overrideNewEVMArgs({0x101d250d8, 0x101d25190, 0xc01752f260, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...}, ...)
	/Users/runner/go/pkg/mod/github.com/ava-labs/libevm@v1.13.15-0.20260120173328-de5fd6fcd5df/core/vm/hooks.libevm.go:98 +0x260 fp=0xc00030c790 sp=0xc00030c6b0 pc=0x100c00d50
github.com/ava-labs/libevm/core/vm.NewEVM({0x101d250d8, 0x101d25190, 0xc01752f260, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...}, ...)
	/Users/runner/go/pkg/mod/github.com/ava-labs/libevm@v1.13.15-0.20260120173328-de5fd6fcd5df/core/vm/evm.go:150 +0x2b4 fp=0xc00030cb10 sp=0xc00030c790 pc=0x100bf5614
github.com/ava-labs/avalanchego/graft/subnet-evm/core.(*StateProcessor).Process(0xc0144e0f20, 0xc00b87a140, 0xc01183d688, 0xc0004b63c0, {{0x0, 0x0}, 0x0, 0x0, {0x0, 0x0, ...}})
	/Users/runner/work/avalanchego/avalanchego/graft/subnet-evm/core/state_processor.go:92 +0x760 fp=0xc00030d030 sp=0xc00030cb10 pc=0x101233100
github.com/ava-labs/avalanchego/graft/subnet-evm/core.(*BlockChain).insertBlock(0xc000331608, 0xc00b87a140, 0x1)
	/Users/runner/work/avalanchego/avalanchego/graft/subnet-evm/core/blockchain.go:1431 +0x1180 fp=0xc00030d6c0 sp=0xc00030d030 pc=0x10120a240
github.com/ava-labs/avalanchego/graft/subnet-evm/core.(*BlockChain).InsertChain(0xc000331608, {0xc0000ba080, 0xa, 0x0?})
	/Users/runner/work/avalanchego/avalanchego/graft/subnet-evm/core/blockchain.go:1345 +0xe4c fp=0xc00030da30 sp=0xc00030d6c0 pc=0x101208bec
github.com/ava-labs/avalanchego/graft/subnet-evm/core.testCanonicalHashMarker(0xc00b6d4e00, {0x1015cc184, 0x8})
	/Users/runner/work/avalanchego/avalanchego/graft/subnet-evm/core/blockchain_test.go:751 +0x8e8 fp=0xc00030de80 sp=0xc00030da30 pc=0x10126f538
github.com/ava-labs/avalanchego/graft/subnet-evm/core.TestCanonicalHashMarker.func1(0xc00b6d4e00)
	/Users/runner/work/avalanchego/avalanchego/graft/subnet-evm/core/blockchain_test.go:684 +0x40 fp=0xc00030dec0 sp=0xc00030de80 pc=0x10126ec20
testing.tRunner(0xc00b6d4e00, 0xc015fa4930)
	/Users/runner/hostedtoolcache/go/1.25.7/arm64/src/testing/testing.go:1934 +0x168 fp=0xc00030dfa0 sp=0xc00030dec0 pc=0x1004e8758
testing.(*T).Run.gowrap1()
	/Users/runner/hostedtoolcache/go/1.25.7/arm64/src/testing/testing.go:1997 +0x40 fp=0xc00030dfd0 sp=0xc00030dfa0 pc=0x1004e9cb0
runtime.goexit({})
	/Users/runner/hostedtoolcache/go/1.25.7/arm64/src/runtime/asm_arm64.s:1268 +0x4 fp=0xc00030dfd0 sp=0xc00030dfd0 pc=0x100347b84
created by testing.(*T).Run in goroutine 47050
	/Users/runner/hostedtoolcache/go/1.25.7/arm64/src/testing/testing.go:1997 +0x6e4

[demosdemon]

FFI unsafe Audit Results

Performed a systematic review of all unsafe usages across the Go FFI files (memory.go, proposal.go, revision.go, iterator.go, maybe.go, proofs.go). Below is the full analysis, cross-referenced against ffi/firewood.h.

---

Union Access via unsafe.Pointer(&result.anon0) — All Sound

CGo maps C anonymous unions to opaque byte arrays (anon0). The code casts &result.anon0 to the appropriate variant type. Every such access is:

1. Tag-guarded — always inside a switch result.tag that checks the discriminant first.
2. Within bounds — the target type is never larger than the union (sized to the largest variant).
3. Properly aligned — all tags are typedef size_t, ensuring the union body starts pointer-aligned (consistent with #[repr(C, usize)] on the Rust side).

Affected result types and their variants, all verified correct:

Result Type	Variants Read	Largest Variant
HashResult	HashKey some, OwnedBytes err	HashKey (32B)
VoidResult	OwnedBytes err	OwnedBytes (2 words)
ValueResult	OwnedBytes some, OwnedBytes err	HashKey revision_not_found (32B)
ProposalResult	ProposalResult_Ok_Body ok, OwnedBytes err	ProposalResult_Ok_Body (~40B)
RevisionResult	RevisionResult_Ok_Body ok, OwnedBytes err	RevisionResult_Ok_Body (~40B)
IteratorResult	IteratorResult_Ok_Body ok, OwnedBytes err	OwnedBytes (2 words)
HandleResult	*DatabaseHandle ok, OwnedBytes err	OwnedBytes (2 words)
KeyValueResult	OwnedKeyValuePair some, OwnedBytes err	OwnedKeyValuePair (4 words)
KeyValueBatchResult	OwnedKeyValueBatch some, OwnedBytes err	both 2 words
BatchOp (write)	Put_Body, Delete_Body, DeleteRange_Body	Put_Body (4 words)
Maybe_BorrowedBytes (write)	BorrowedBytes some	single variant
Maybe_OwnedBytes	OwnedBytes some	single variant
NextKeyRangeResult	NextKeyRange some, OwnedBytes err	NextKeyRange
CodeIteratorResult	*CodeIteratorHandle ok, OwnedBytes err	OwnedBytes (2 words)
RangeProofResult	*RangeProofContext ok, OwnedBytes err	HashKey (32B)
ChangeProofResult	*ChangeProofContext ok, OwnedBytes err	HashKey (32B)

---

unsafe.SliceData / unsafe.Slice — All Sound

• newBorrowedBytes (memory.go:61): unsafe.SliceData(slice) to get the base pointer. Nil-checked, pinned when non-empty. For empty-but-non-nil slices ([]byte{}), the pointer is not pinned but points to runtime.zerobase (static), length is 0, and the Rust side won't dereference it.

• newBorrowedBatchOps (memory.go:115): Same pattern for []C.BatchOp. Nil and length checked, pinned.

• ownedBytes.BorrowedBytes (memory.go:196): unsafe.Slice((*byte)(b.owned.ptr), b.owned.len) creates a Go slice header over Rust-owned memory. Nil-checked. Lifetime discipline documented — slice is only valid while the ownedBytes is alive.

• ownedKeyValueBatch.borrow (memory.go:340): Same pattern over OwnedKeyValuePair array. Only used transiently in copy() to create ownedKeyValue wrappers; batch is not freed until all loaded pairs are consumed by the iterator.

---

Type Punning (Hash ↔ HashKey) — Sound

• newCHashKey (memory.go:470): *(*C.HashKey)(unsafe.Pointer(&hash)) — Hash is [32]byte, C.HashKey is struct { uint8_t _0[32] }. Identical size, alignment, and layout.

• Root hash extraction (e.g. proposal.go:186, revision.go:146): *(*Hash)(unsafe.Pointer(&body.root_hash._0)) — same safe conversion in reverse.

---

Bug Found: Iterator.Drop() double-free

iterator.go:182-192 — Drop() claims "It is safe to call Drop multiple times; subsequent calls after the first are no-ops." but never sets it.handle = nil after freeing:

func (it *Iterator) Drop() error {
    err := it.freeCurrentAllocation()
    if it.handle != nil {
        return errors.Join(
            err,
            getErrorFromVoidResult(C.fwd_free_iterator(it.handle)))
        // BUG: it.handle is never set to nil
    }
    return err
}

On a second call, it.handle is still non-nil, so fwd_free_iterator is called on an already-freed pointer — use-after-free / double-free. Compare with Proposal.Drop() (p.handle = nil at line 156) and Revision.Drop() (r.handle = nil at line 110) which both nil their handles after freeing.

Fix: add it.handle = nil after the fwd_free_iterator call.

---

Latent Issue: C.int truncation in CopiedBytes

memory.go:212:

return C.GoBytes(unsafe.Pointer(b.owned.ptr), C.int(b.owned.len))

C.GoBytes takes C.int (32-bit), but b.owned.len is size_t (64-bit). Values exceeding ~2GB would be silently truncated. Low risk in practice for database key/values, but could be replaced with a manual copy using unsafe.Slice to eliminate the limitation.

---

Overall Assessment

The unsafe usage in the FFI Go bindings is generally sound. All union accesses are properly tag-guarded, type puns use identical layouts, and slice operations respect lifetime constraints. The Iterator.Drop() double-free bug is the most actionable finding and likely worth a fix.

[demosdemon]

The backgrounded goroutines is likely a red herring in and of itself and has been a known issue for a while.

I was researching upstream Go issues and came across golang/go#77317 which has similar symptoms but the situation is not exactly the same.

For our problem; on arm64, the panic is when a stack split yields an address less than the current stack pointer. And on amd64, the panic is when the bottom of a goroutine stack has a garbage pointer at the bottom. The amd64 panics actually indicate that the code was already unwinding and failed to unwind because of the garbage pointer. I suspect the original panic reason to be similar: stack split yielded an invalid pointer.

• arm64: https://github.com/ava-labs/avalanchego/actions/runs/21972259848/job/63476193719#step:4:220

runtime: newstack sp=0x16fb3d950 stack=[0xc0215f4000, 0xc0215f5000]
	morebuf={pc:0x10140af84 sp:0x16fb3d950 lr:0x0}
	sched={pc:0x1003415bc sp:0x16fb3d950 lr:0x10140af84 ctxt:0x0}
runtime: gp=0xc00b2268c0, goid=47097, gp->status=0x2
 runtime: split stack overflow: 0x16fb3d950 < 0xc0215f4000
fatal error: runtime: split stack overflow

runtime stack:
runtime.throw({0x1015e6c93?, 0xc01477c000?})
	/Users/runner/hostedtoolcache/go/1.25.7/arm64/src/runtime/panic.go:1094 +0x34 fp=0x16fb3d8c0 sp=0x16fb3d890 pc=0x10033e9c4
runtime.newstack()
	/Users/runner/hostedtoolcache/go/1.25.7/arm64/src/runtime/stack.go:1100 +0x590 fp=0x16fb3d9f0 sp=0x16fb3d8c0 pc=0x100323080
runtime.morestack()
	/Users/runner/hostedtoolcache/go/1.25.7/arm64/src/runtime/asm_arm64.s:392 +0x70 fp=0x16fb3d9f0 sp=0x16fb3d9f0 pc=0x1003457c0

goroutine 47097 gp=0xc00b2268c0 m=0 mp=0x102708300 [running]:
runtime.sigpanic()
	/Users/runner/hostedtoolcache/go/1.25.7/arm64/src/runtime/signal_unix.go:906 +0x35c fp=0x16fb3d950 sp=0x16fb3d950 pc=0x1003415bc
runtime: g 47097: unexpected return pc for runtime.sigpanic called from 0x10140af84
stack: frame={sp:0x16fb3d950, fp:0x16fb3d950} stack=[0xc0215f4000,0xc0215f5000)

created by github.com/ava-labs/firewood-go-ethhash/ffi.(*Database).Close in goroutine 47117
	/Users/runner/go/pkg/mod/github.com/ava-labs/firewood-go-ethhash/ffi@v0.1.0/firewood.go:436 +0xa4

• amd64: https://github.com/ava-labs/avalanchego/actions/runs/21976148273/job/63488199907#step:4:224

unexpected fault address 0x182053d
fatal error: fault
runtime: g13978: frame.sp=0x7fd0143fd0b0 top=0xc007b80fe0
	stack=[0xc007b80000-0xc007b81000
fatal error: traceback did not unwind completely

runtime stack:
runtime.throw({0x28885f6?, 0x0?})
	/opt/hostedtoolcache/go/1.25.7/x64/src/runtime/panic.go:1094 +0x48 fp=0x7fd0143fd0e0 sp=0x7fd0143fd0b0 pc=0x4a7e68
runtime.(*unwinder).finishInternal(0x4aaab9?)
	/opt/hostedtoolcache/go/1.25.7/x64/src/runtime/traceback.go:566 +0x12a fp=0x7fd0143fd120 sp=0x7fd0143fd0e0 pc=0x494cca
runtime.(*unwinder).next(0x7fd0143fd1c8?)
	/opt/hostedtoolcache/go/1.25.7/x64/src/runtime/traceback.go:447 +0x232 fp=0x7fd0143fd198 sp=0x7fd0143fd120 pc=0x494ad2
runtime.(*_panic).nextFrame.func1()
	/opt/hostedtoolcache/go/1.25.7/x64/src/runtime/panic.go:957 +0xa6 fp=0x7fd0143fd250 sp=0x7fd0143fd198 pc=0x46b946
runtime.systemstack(0x1000000)
	/opt/hostedtoolcache/go/1.25.7/x64/src/runtime/asm_amd64.s:513 +0x4a fp=0x7fd0143fd260 sp=0x7fd0143fd250 pc=0x4aed6a

goroutine 13978 gp=0xc00914c380 m=3 mp=0xc000043808 [running]:
runtime.systemstack_switch()
	/opt/hostedtoolcache/go/1.25.7/x64/src/runtime/asm_amd64.s:478 +0x8 fp=0x7fd0143fcfa0 sp=0x7fd0143fcf90 pc=0x4aed08
runtime: g 13978: unexpected return pc for runtime.systemstack_switch called from 0x7fd0143fd018
stack: frame={sp:0x7fd0143fcf90, fp:0x7fd0143fcfa0} stack=[0xc007b80000,0xc007b81000)

created by github.com/ava-labs/firewood-go-ethhash/ffi.(*Database).Close in goroutine 13871
	/home/runner/go/pkg/mod/github.com/ava-labs/firewood-go-ethhash/ffi@v0.1.0/firewood.go:436 +0xf3

• amd64: https://github.com/ava-labs/avalanchego/actions/runs/21970606498/job/63471066045#step:4:206

unexpected fault address 0x16974cf
fatal error: fault
runtime: g9919: frame.sp=0x7fdc4bffe120 top=0xc000888fe0
	stack=[0xc000888000-0xc000889000
fatal error: traceback did not unwind completely

runtime stack:
runtime.throw({0x1f25559?, 0x0?})
	/opt/hostedtoolcache/go/1.25.7/x64/src/runtime/panic.go:1094 +0x48 fp=0x7fdc4bffe150 sp=0x7fdc4bffe120 pc=0x4a7888
runtime.(*unwinder).finishInternal(0x4aa4d9?)
	/opt/hostedtoolcache/go/1.25.7/x64/src/runtime/traceback.go:566 +0x12a fp=0x7fdc4bffe190 sp=0x7fdc4bffe150 pc=0x49482a
runtime.(*unwinder).next(0x7fdc4bffe238?)
	/opt/hostedtoolcache/go/1.25.7/x64/src/runtime/traceback.go:447 +0x232 fp=0x7fdc4bffe208 sp=0x7fdc4bffe190 pc=0x494632
runtime.(*_panic).nextFrame.func1()
	/opt/hostedtoolcache/go/1.25.7/x64/src/runtime/panic.go:957 +0xa6 fp=0x7fdc4bffe2c0 sp=0x7fdc4bffe208 pc=0x46b4a6
runtime.systemstack(0x1000000)
	/opt/hostedtoolcache/go/1.25.7/x64/src/runtime/asm_amd64.s:513 +0x4a fp=0x7fdc4bffe2d0 sp=0x7fdc4bffe2c0 pc=0x4ae78a

goroutine 9919 gp=0xc00059efc0 m=7 mp=0xc00040a808 [running]:
runtime.systemstack_switch()
	/opt/hostedtoolcache/go/1.25.7/x64/src/runtime/asm_amd64.s:478 +0x8 fp=0x7fdc4bffe010 sp=0x7fdc4bffe000 pc=0x4ae728
runtime: g 9919: unexpected return pc for runtime.systemstack_switch called from 0x7fdc4bffe088
stack: frame={sp:0x7fdc4bffe000, fp:0x7fdc4bffe010} stack=[0xc000888000,0xc000889000)

created by github.com/ava-labs/firewood-go-ethhash/ffi.(*Database).Close in goroutine 9887
	/home/runner/go/pkg/mod/github.com/ava-labs/firewood-go-ethhash/ffi@v0.1.0/firewood.go:436 +0xf3

[demosdemon]

Something I didn't realize until now is that unexpected fault address is actually a SIGBUS and 0x182053d and 0x16974cf are the faulted address. This could be the misaligned tagged unions; however, the faulted address don't actually support that. If the misaligned tagged unions were responsible, I would expect a faulted address to be a factor or 4 (uint32).

I'm curious why on x86_64 this surfaces as a SIGBUS but on aarch64 it surfaces as morestack failure. This aarch64 error hints that the SP is somehow on the systemstack when morestack was invoked; however, none of the firewood cgo functions are re-entrant into go. Additionally, the builds of firewood where this is occurring are compiled with panic=abort so it cannot be a rust unwind. io-uring is currently disabled, which was firewood's only use of mmap (a common source of SIGBUS).

[demosdemon]

Actually, SIGBUS or SIGSEGV both print this "unexpected fault address" message. And, after re-reading the panic from aarch64, I see that sigpanic is very likely the go function that triggered morestack, as is indicated as possible in its documentation.

The machinery around injecting the sigpanic call whenever and wherever a signal is encountered is likely why and how the SP is not within the expected go arena.

IMO, it sounds like there is a race when closing a firewood database and the go runtime.GC() is ultimately triggering the race. I interpret the sequence of events like:

• (G1) runtime.GC() starts in a new goroutine (G2)
• (G1) (*Database).Close() proceeds to wait for the waitgroup
• (G2) finalizer invokes something that causes the waitgroup to reach zero
• (G1) fwd_close_db frees the memory owned by firewood
• (G2) another finalizer fires and somehow still holds a reference to memory previously allocated by firewood and invokes a cgo function that faults