cells: always try to re-establish dead tunnel, unless stopped

Motivation:
As long as cell tunnel is not explicitly stopped by calling
dmg.cells.network.LocationManagerConnector#stopped, other interrupts
should be ignored.

Modification:
Update retry logic to never give up, unless _isRunning flag set to
false.

Result:
More robust tulles in case of network issues.

Issue: #7707, #5326
Acked-by: Dmitry Litvintsev
Target: master, 10.2, 10.1, 10.0, 9.2
Require-book: no
Require-notes: yes
(cherry picked from commit 600ed1f)
Signed-off-by: Tigran Mkrtchyan <[email protected]>

Author: kofemann

modules/cells/src/main/java/dmg/cells/network/LocationManagerConnector.java

@@ -102,8 +102,8 @@ public void run() {
                     } finally {
                         getNucleus().kill(tunnel.getCellName());
                     }
-                } catch (InterruptedIOException | ClosedByInterruptException e) {
-                    throw e;
+                } catch (InterruptedIOException | InterruptedException | ClosedByInterruptException e) {
+                    _log.warn("Connection to {} ({}) interrupted. Reason: {}", _domain, _address, e.toString());
                 } catch (ExecutionException | IOException e) {
                     String error = Exceptions.meaningfulMessage(Throwables.getRootCause(e));
                     _log.warn(AlarmMarkerFactory.getMarker(PredefinedAlarm.LOCATION_MANAGER_FAILURE,
@@ -116,11 +116,16 @@ public void run() {
                 _status = "Sleeping";
                 long sleep = random.nextInt(16000) + 4000;
                 _log.warn("Sleeping {} seconds", sleep / 1000);
-                Thread.sleep(sleep);
+                try {
+                    Thread.sleep(sleep);
+                } catch (InterruptedException e) {
+                    // restore interrupted status
+                    Thread.currentThread().interrupt();
+                }
             }
-        } catch (InterruptedIOException | InterruptedException | ClosedByInterruptException ignored) {
         } finally {
             NDC.pop();
+            _thread = null;
             _status = "Terminated";
         }
     }