Netopeer2-server unexpected crash

[yoousra]

Versions:

netopeer2-2.2.35
sysrepo-3.3.10
libnetconf2-3.5.5
libyang-3.7.8

I have a problem of unexpected netopeer2-server crash.

This kind of looks like the issue #1481

Here are some details and logs before the crash:
Our software stops and thus we stop the netopeer2-server (expected) and here we have a log Recovered a read-lock of CID 53 (sr_notif_sub_del)

netopeer2-server(PID:3446): [INF]: NP: Server terminated.
netopeer2-server(PID:3446): [INF]: LN: Session 1: Call Home client "NC-Client-1" removed, but an established session will not be terminated.
netopeer2-server(PID:3446): [INF]: LN: Session 1: Call Home client "NC-Client-1" thread exit.
netopeer2-server(PID:3446): [WRN]: SR: Connection with CID 53 is dead.
netopeer2-server(PID:3446): [WRN]: SR: Recovered a read-lock of CID 53 (sr_notif_sub_del).
netopeer2-server(PID:3446): [INF]: SR: EV ORIGIN: "ietf-netconf-notifications" "notif" ID 49 priority 0 for 4 subscribers published.
netopeer2-server(PID:3446): [INF]: NP: Generated new event (netconf-session-end).
netopeer2-server(PID:3446): [INF]: SR: Connection 55 destroyed.
INFO exited: netopeer2-server (exit status 0; expected)

Then we restart our software and at a certain moment the netconf server sends a "close-session" to the client so the session is terminated.
Meanwhile our software continues its treatement and at a certain moment we send a notif for a module "X", and we get the log
sysrepo: There are no subscribers for "X", I think because the session is closed and thus the subscriptions are removed, but after that we have an error log SR: Waiting on a conditional variable failed (sr_notif_sub_del: Connection timed out).:

netopeer2-server(PID:10239): [INF]: LN: Call Home client "NC-Client-1" timeout of 5 seconds expired, reconnecting.
netopeer2-server(PID:10239): [INF]: LN: Trying to connect the pending socket 55.
netopeer2-server(PID:10239): [INF]: LN: Timed out after 500 ms (Operation now in progress).
netopeer2-server(PID:10239): [ERR]: SR: Waiting on a conditional variable failed (sr_notif_sub_del: Connection timed out).
netopeer2-server(PID:10239): [INF]: LN: Call Home client "NC-Client-1" timeout of 5 seconds expired, reconnecting.
netopeer2-server(PID:10239): [INF]: LN: Trying to connect the pending socket 55.
netopeer2-server(PID:10239): [INF]: LN: Timed out after 500 ms (Operation now in progress).
netopeer2-server(PID:10239): [INF]: LN: Call Home client "NC-Client-1" timeout of 5 seconds expired, reconnecting.
netopeer2-server(PID:10239): [INF]: LN: Call Home client "NC-Client-1" endpoint "default-ssh" failed connection attempt limit 3 reached.
netopeer2-server(PID:10239): [INF]: LN: Call Home client "NC-Client-1" endpoint "default-ssh" connecting...
netopeer2-server(PID:10239): [INF]: LN: Timed out after 500 ms (Operation now in progress).
netopeer2-server(PID:10239): [INF]: LN: Call Home client "NC-Client-1" timeout of 5 seconds expired, reconnecting.
netopeer2-server(PID:10239): [INF]: LN: Trying to connect the pending socket 55.
netopeer2-server(PID:10239): [INF]: LN: Timed out after 500 ms (Operation now in progress).
netopeer2-server(PID:10239): [ERR]: SR: Waiting on a conditional variable failed (sr_notif_sub_del: Connection timed out).
netopeer2-server(PID:10239): [INF]: SR: EV ORIGIN: "ietf-netconf-notifications" "notif" ID 57 priority 0 for 7 subscribers published.
netopeer2-server(PID:10239): [INF]: NP: Generated new event (netconf-session-end).
netopeer2-server(PID:10239): [WRN]: LN: SSH: ssh_socket_exception_callback: Socket exception callback: 2 (110)
netopeer2-server(PID:10239): [WRN]: LN: SSH: ssh_socket_exception_callback: Socket error: Connection timed out

And we continue and have the logs of the crash:

WARN exited: netopeer2-server (terminated by SIGSEGV (core dumped); not expected)
[WRN] Connection with CID 55 is dead.
[WRN] EV ORIGIN: SHM event "notif" of CID 55 ID 2 recovered.
sysrepo-3.3.10/src/shm_sub.c:3055: sr_shmsub_notif_notify: Assertion `notif_sub_count' failed.

I appreciate your help, thank you.

[michalvasko]

I have made a change that should prevent the "crash" (just a failed assert) but the whole scenario is invalid and probably caused by some other error condition. Let us know if you learn more about the use-case, the synchronization seems to not work well.

[yoousra]

Thanks for your reply! I'll try the new code and see if it's better, but as you said the cause is not that, what do you think could cause the first warning we get: SR: Recovered a read-lock of CID 53 (sr_notif_sub_del). after stopping netopeer2-server? It seems there's a problem with the notifs subscription deletion?

[michalvasko]

I can answer only generally, a synchronization issue. But it seems you have quite a complex scenario with many subscribers and Call Home clients so without reproducing it it is quite impossible for me to debug it.