Monitor for stalled / forked off nodes

[josibake]

In a wargame, we need a process to check each namespace (team) and look for nodes that are no longer on the chain tip. If they are stalled, they get killed. This prevents teams from firewalling themselves from enemy nodes and just waiting out the war game.

Simple version of this would be a script that checks each namespace after a block has been mined and looks for nodes that are no longer at the chain tip or who have been potentially forked off the chain by an attacker. Once if finds these nodes, kill the pods.

[willcl-ark]

I wonder if we could use ForkMonitor from this @0xB10C ? Does it have any machine-parsable output?

[josibake]

From what I understand, fork monitor runs based on a single nodes' perspective, based on what it's heard from the network. This, in contrast, needs to be able to check in with random nodes and see what they believe the latest chain tip is, and then potentially take actions based on what the node responds with. Ideally, this can be something we can deploy as a commander? All we would need is to ensure it is deployed by a user that has permissions to delete pods from all namespaces.

[0xB10C]

Yes, it has a JSON API. Mainnet example: https://fork.observer/api/1/data.json

you are probably interested in something like this:

"nodes": [
    {
      "id": 3,
      "name": "btcd v0.24.2-beta.rc1",
      "description": "A btcd node.",
      "implementation": "btcd",
      "tips": [
        {
          "hash": "000000000000000000019066a8be0724175e18539db741579622738f1ce1f2ee",
          "status": "valid-fork",
          "height": 853120
        },
...
        {
          "hash": "00000000000000000002ad15f6680bf74febab1020128641fab0da4bdfa5b4bd",
          "status": "valid-fork",
          "height": 848477
        },
        {
          "hash": "00000000000000000000f7776ba0a94b5b4bd23e055989d08d2c63d44a450213",
          "status": "active",
          "height": 859540
        }
      ],
      "last_changed_timestamp": 1725283862,
      "version": "unknown",
      "reachable": true
    }
  ]

I also have RSS feeds:

• https://fork.observer/rss/1/forks.xml
• https://fork.observer/rss/1/invalid.xml (invalid blocks)
• https://fork.observer/rss/1/lagging.xml (reachable nodes lagging behind other nodes - i.e. stuck)
• https://fork.observer/rss/1/unreachable.xml (unreachable nodes)

[josibake]

@0xB10C

you are probably interested in something like this

Ah, nice! So fork observer is getting all of this just by having a p2p connection with each node it's monitoring, or some extra config? Could run this in an admin namespace then and poll it directly.

[0xB10C]

fwiw BitMex Research's/Sjors ForkMonitor and my fork-observer are different projects

At some point we had a running fork-observer in warnet already #90, though I'm not sure if this is in there or got removed during kubernetes migration

[willcl-ark]

Oh NO! Sorry, I of course meant fork-observer!

[willcl-ark]

I think I might set up fork-observer in the new setup, as:

i) It's awesome
ii) It can possibly do this thing we want here
iii) it is a great visual for tabconf in any case
iv) etc.

[0xB10C]

@0xB10C

you are probably interested in something like this

Ah, nice! So fork observer is getting all of this just by having a p2p connection with each node it's monitoring, or some extra config? Could run this in an admin namespace then and poll it directly.

It's RPC based. Mainly the getchaintips RPC. You can run it with something like rpcwhitelist=forkobserver:getchaintips,getblockheader,getblockhash,getblock (see https://github.com/0xB10C/fork-observer?tab=readme-ov-file#connecting-to-a-bitcoin-core-node) added to the Bitcoin Core config to limit the RPC permissions if that's an issue.

[josibake]

I think I might set up fork-observer in the new setup, as:

ACK, I think it solves all our requirements for being able to monitor nodes and kill them if they are too far behind. The RPC whitelist is nice, as well!

[willcl-ark]

I think I might set up fork-observer in the new setup, as:

ACK, I think it solves all our requirements for being able to monitor nodes and kill them if they are too far behind. The RPC whitelist is nice, as well!

Actually I may need help on this, as I don't know how to add a new service cleanly. Would this be a new chart or just a raw manifest? I guess strictly should be the former, as we add a chart for the service, and one for the config? Then update it based on the graph, then load it? (I don't really know how the charts work that well).

[willcl-ark]

nice typo

[josibake]

Would this be a new chart

This is how I would start. I'd expect fork-observer to be in the same namespace as warnet logging, or warnet-admin (for tabconf?), and as a config have the default usernames and passwords for each of the namespaces it will be monitoring (i.e., all nodes in a team namespace will have the same rpc / username password).

This way, networks (collections of tanks) can be deployed into their namespace (either default warnet, or team namespace) without needing to be aware of fork-observer. As the nodes come online, fork-observer can start to poll them. This assumes fork-observer doesn't crash if its deployed and tries to make an rpc call to a node that doesn't exist yet.

[willcl-ark]

WIP in #507

[willcl-ark]

This assumes fork-observer doesn't crash if its deployed and tries to make an rpc call to a node that doesn't exist yet.

I believe this is the case, but have not yet tested