WM tries to fetch non-finalized blocks overloading HC/node

[PHA-SYSOPS]

It looks like WM is trying to fetch non-finalized headers, in turn headers-cache logs many 404's attempting to collect non-finalized. headers. If you directly use a node you will see 100% CPU usage and the node struggling overtime getting stuck on finalizing blocks.

It feels like a off-by-one error, or some lazy attempt to fetch the next block without checking its actually there. The farm itself looks operational, but because of the node getting stuck from time to time, TX's are not send, causing workers to go offline. Many people reporting these symptoms an i feel that this might be in the area of the root cause.

log:

[2023-06-20T09:44:33Z INFO rocket::server::_] Matched: (get_headers) GET /headers/<start> [2023-06-20T09:44:33Z INFO rocket::server::_] Outcome: Success [2023-06-20T09:44:33Z INFO prpc_measuring] GET /headers/16048601 cost 51 microseconds, status: 404 [2023-06-20T09:44:33Z INFO rocket::server::_] Response succeeded. [2023-06-20T09:44:33Z INFO rocket::server] GET /headers/16048603: [2023-06-20T09:44:33Z INFO rocket::server::_] Matched: (get_headers) GET /headers/<start> [2023-06-20T09:44:33Z INFO rocket::server::_] Outcome: Success [2023-06-20T09:44:33Z INFO prpc_measuring] GET /headers/16048603 cost 33 microseconds, status: 404 [2023-06-20T09:44:33Z INFO rocket::server::_] Response succeeded. [2023-06-20T09:44:33Z INFO rocket::server] GET /headers/16048602: [2023-06-20T09:44:33Z INFO rocket::server::_] Matched: (get_headers) GET /headers/<start> [2023-06-20T09:44:33Z INFO rocket::server::_] Outcome: Success [2023-06-20T09:44:33Z INFO prpc_measuring] GET /headers/16048602 cost 33 microseconds, status: 404 [2023-06-20T09:44:33Z INFO rocket::server::_] Response succeeded. [2023-06-20T09:44:33Z INFO rocket::server] GET /headers/16048603: [2023-06-20T09:44:33Z INFO rocket::server::_] Matched: (get_headers) GET /headers/<start> [2023-06-20T09:44:33Z INFO rocket::server::_] Outcome: Success [2023-06-20T09:44:33Z INFO prpc_measuring] GET /headers/16048603 cost 233 microseconds, status: 404 [2023-06-20T09:44:33Z INFO rocket::server::_] Response succeeded. [2023-06-20T09:44:33Z INFO rocket::server] GET /headers/16048604: [2023-06-20T09:44:33Z INFO rocket::server::_] Matched: (get_headers) GET /headers/<start> [2023-06-20T09:44:33Z INFO rocket::server::_] Outcome: Success [2023-06-20T09:44:33Z INFO prpc_measuring] GET /headers/16048604 cost 176 microseconds, status: 404 [2023-06-20T09:44:33Z INFO rocket::server::_] Response succeeded. [2023-06-20T09:44:33Z INFO rocket::server] GET /headers/16048604: [2023-06-20T09:44:33Z INFO rocket::server::_] Matched: (get_headers) GET /headers/<start> [2023-06-20T09:44:33Z INFO rocket::server::_] Outcome: Success [2023-06-20T09:44:33Z INFO prpc_measuring] GET /headers/16048604 cost 116 microseconds, status: 404 [2023-06-20T09:44:33Z INFO rocket::server::_] Response succeeded. [2023-06-20T09:44:33Z INFO rocket::server] GET /headers/16048604: [2023-06-20T09:44:33Z INFO rocket::server::_] Matched: (get_headers) GET /headers/<start> [2023-06-20T09:44:33Z INFO rocket::server::_] Outcome: Success [2023-06-20T09:44:33Z INFO prpc_measuring] GET /headers/16048604 cost 106 microseconds, status: 404 [2023-06-20T09:44:33Z INFO rocket::server::_] Response succeeded.