Skip to content

[Provider Bug] Provider crashes with nil pointer panic in serialBroadcaster.syncSequence caused by ghost leases #434

@drape76

Description

@drape76

Describe the bug
Provider crashes with a nil pointer panic in serialBroadcaster.syncSequence (tx.go:541). The crash is triggered by spam tenants creating ghost leases — leases that are closed on-chain but remain in the provider's memory as active deployments. After repeated failed deployment health checks (~10+ attempts with "available replicas below target"), the provider panics and exits with code 1 or 2.

account address = akash1chnhnu50f6hv98xl0m7xm95vel457ysp32uwpj
deployment sequence = 25857450 (example, recurs with new ghost leases)
group sequence = 1

To Reproduce
Spam tenant creates a lease with the provider
Tenant closes deployment on-chain but provider retains it in memory
Provider deployment-monitor repeatedly checks service replicas, gets available=0 for 10+ attempts
Provider panics with nil pointer dereference in serialBroadcaster.syncSequence

Expected behavior
Provider should handle closed/invalid leases gracefully without panicking. Ghost leases should be detected and cleaned up without requiring a full provider restart.

Crash log
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x5e8fffd]

goroutine 56 [running]:
pkg.akt.dev/go/node/client/v1beta3.(*serialBroadcaster).syncSequence(...)
pkg.akt.dev/go@v0.1.9/node/client/v1beta3/tx.go:541 +0xfd
pkg.akt.dev/go/node/client/v1beta3.(*serialBroadcaster).broadcaster(...)
pkg.akt.dev/go@v0.1.9/node/client/v1beta3/tx.go:572 +0xa0c

Additional context
Provider version: 0.10.7 (latest)
Helm chart: 14.0.8
Kubernetes: K3s
Hardware: Dell PowerEdge R640, 80 vCPU, 384GB RAM
The crash recurs every few hours as new ghost leases appear

Workaround:
automated script that closes ghost leases on-chain hourly and restarts provider, but the underlying panic is not resolved

Metadata

Metadata

Labels

repo/providerAkash provider-services repo issues

Type

No type

Projects

Status

Released (in Prod)

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions