ci: improve handle-exit.sh timeout behavior (#1207)

For large validator folders, handle-exit.sh takes over 90 seconds to
archive it, and systemd kills it. This causes the disk space to fill up
because handle-exit.sh is unable to clean up resources.

To improve the situation, I've increased the time limit to 30 minutes.

I also clear out the validator folder when sig starts because sig is
unable to resume from old state. This was resulting in repeated crashing
and attempting to archive the huge validator folder and failing,
creating partial tarballs over and over that filled up the disk.

Instead now the service just tries to archive it to S3 once, and if that
fails, set aside the folder and restart.

Author: dnut

ci/run-and-update-service/run.sh

@@ -14,4 +14,20 @@ fi
 # Delete log files older than 1 month
 find "$BASE_DIR/logs" -name "sig.*.log" -type f -mtime +30 -delete
 
+# Sig is unable to resume from existing validator state, so clear it before starting.
+if [ -d "$BASE_DIR/validator" ]; then
+    # We make an effort to save the old state in case it is needed for
+    # debugging, but only save the latest instance to avoid running out of disk
+    # space. There is already handle-exit.sh which uploads crash artifacts to
+    # S3, so this is only a last resort to deal with cascading failures.
+    rm -rf "$BASE_DIR/validator-archive"
+    mv "$BASE_DIR/validator" "$BASE_DIR/validator-archive"
+fi
+
+# Clear out old archives to avoid running out of disk space. They should have
+# been uploaded to S3 and then deleted right away. If not, we must be flooding
+# the disk with archives on failed uploads, and we're going to run out of disk
+# space.
+rm -f "$BASE_DIR/validator-*.tar.zst"
+
 "$BASE_DIR/zig-out/bin/sig" $@ 2>>"$BASE_DIR/logs/sig.log" >>"$BASE_DIR/logs/sig.log"

ci/run-and-update-service/sig.service

@@ -13,6 +13,7 @@ LimitNOFILE=infinity
 ExecStopPost=/home/sig/sig/ci/run-and-update-service/handle-exit.sh
 MemoryAccounting=yes
 MemoryMax=85%
+TimeoutStopSec=30m
 
 [Install]
 WantedBy=multi-user.target