Fort RTR sessions resets every 8 mins

[veerupra]

We are running two Fort 1.6.6 instances with 8 RTR sessions each and they suddenly stop fetching the ROAs from RIRs after 3 weeks running fine and resets the RTR sessions with the routers every 8 mins . No error logs, only log found - "INF: Client accepted [FD: 7]: :" with all the routers. Once the services is restarted it works fine for couple of weeks. It has happened twice so far, any issue with memory ??

[ydahhrk]

they suddenly stop fetching the ROAs from RIRs

Both instances stop at the same time?

Did the network flop?

No error logs, only log found - "INF: Client accepted [FD: 7]: :" with all the routers.

Well, I found a code path in which the client connection stops but Fort logs the event in debug level. (This is foolish. I just upgraded it to INFO. You might want to run it instead: https://github.com/NICMx/FORT-validator/tree/issue164.) However, it refers to a normal socket close from the client, which probably won't tell us much about this bug.

There's also the possibility that it's not printing anything because the connections are not actually dying...

Which would mean the file descriptors (FDs) could not be reused...

Is there a pattern to the FDs?

It has happened twice so far, any issue with memory ??

Not that I know of. Can you monitor the system resources?

RAM, disk space, disk inodes, open sockets, network usage...