Skip to content

onos-consensus-store went into CrashLoopBackOff after GKE node restart. #1636

@praj527

Description

@praj527

Issue description:
We observed that after GKE node restart the onos-consensus-store went into CrashLoopBackOff, we have enabled the PersistentVolume for onos-consensus-store.

onos-consensus-store logs:

2023-02-28T09:06:40.986Z        INFO    github.com/atomix/runtime/sdk/pkg/protocol/statemachine v1/statemachine.go:119  Recovering Map from snapshot    {"Service": "atomix.runtime.map.v1.Map", "Primitive": 4, "Namespace": "aether-roc", "Profile": "", "Name": "configurations"}
2023-02-28T09:06:40.986Z        INFO    github.com/atomix/runtime/sdk/pkg/protocol/statemachine v1/statemachine.go:119  Recovering Map from snapshot    {"Service": "atomix.runtime.map.v1.Map", "Primitive": 11, "Namespace": "aether-roc", "Profile": "", "Name": "onos-topo-objects"}
2023-02-28T09:06:40.986Z        INFO    github.com/atomix/runtime/sdk/pkg/protocol/statemachine v1/statemachine.go:152  Recovering IndexedMap from snapshot     {"Service": "atomix.runtime.indexedmap.v1.IndexedMap", "Primitive": 2, "Namespace": "aether-roc", "Profile": "", "Name": "transactions"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xd813f2]

goroutine 366 [running]:
github.com/atomix/runtime/primitives/pkg/indexedmap/v1.(*IndexedMapEntry).Reset(0x10505e0?)
        github.com/atomix/runtime/primitives@v0.7.8/pkg/indexedmap/v1/indexedmap.pb.go:1277 +0x12
github.com/gogo/protobuf/proto.Unmarshal({0xc0012c0000, 0x559e8, 0x559e8}, {0x147fa60, 0xc000124bd0})
        github.com/gogo/protobuf@v1.3.2/proto/decode.go:335 +0x3d
github.com/atomix/runtime/sdk/pkg/protocol/statemachine.(*SnapshotReader).ReadMessage(0x0?, {0x147fa60, 0xc000124bd0})

image

[onos-consensus-store-0.log](https://github.com/onosproject/onos-config/files/10849855/onos-consensus-store-0.log) [onos-consensus-store-1.log](https://github.com/onosproject/onos-config/files/10849860/onos-consensus-store-1.log) [onos-consensus-store-2.log](https://github.com/onosproject/onos-config/files/10849861/onos-consensus-store-2.log)

Helm Chart Version:

  • Atomix-runtime-0.1.9
  • Onos-operator-0.5.6
  • Onos-topo-1.4.5
  • Onos-cli-1.3.13
  • Onos-config-1.8.5

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions