store: Exit with an error if the notification listener loses its connection

Events we listen for are important for the overall correctness of the
system; for example, if we miss the assignment event for a subgraph, we
will not index it, leaving the user to scratch their head why a newly
deployed subgraph is not making progress.

Because of this, the safest and most robust thing is to immediately exit
the process (with an error) if a notification listener ever loses its db
connection. A process supervisor should be used to restart the process as
is appropriate for the environment we are running in.

Fixes #1279

Author: lutter

store/postgres/src/notification_listener.rs

@@ -138,12 +138,14 @@ impl NotificationListener {
                     .filter_map(|item| match item {
                         Ok(msg) => Some(msg),
                         Err(e) => {
-                            warn!(
-                                logger,
-                                "Error receiving message";
-                                "error" => format!("{}", e)
+                            let msg = format!("{}", e);
+                            crit!(logger, "Error receiving message"; "error" => &msg);
+                            eprintln!(
+                                "Connection to Postgres lost while listening for events. \
+                                 Aborting to avoid inconsistent state. ({})",
+                                msg
                             );
-                            None
+                            std::process::exit(1);
                         }
                     })
                     .filter(|notification| notification.channel == channel_name.0)