Recover from panic in cron task (#28409) (#28425)

Backport #28409 by @earl-warren

- Currently there's code to recover gracefully from panics that happen
within the execution of cron tasks. However this recover code wasn't
being run, because `RunWithShutdownContext` also contains code to
recover from any panic and then gracefully shutdown Forgejo. Because
`RunWithShutdownContext` registers that code as last, that would get run
first which in this case is not behavior that we want.
- Move the recover code to inside the function, so that is run first
before `RunWithShutdownContext`'s recover code (which is now a noop).

Fixes: https://codeberg.org/forgejo/forgejo/issues/1910

Co-authored-by: Earl Warren <[email protected]>
Co-authored-by: Gusted <[email protected]>

Author: 3 people

services/cron/tasks.go

@@ -84,13 +84,15 @@ func (t *Task) RunWithUser(doer *user_model.User, config Config) {
 	t.lock.Unlock()
 	defer func() {
 		taskStatusTable.Stop(t.Name)
-		if err := recover(); err != nil {
-			// Recover a panic within the
-			combinedErr := fmt.Errorf("%s\n%s", err, log.Stack(2))
-			log.Error("PANIC whilst running task: %s Value: %v", t.Name, combinedErr)
-		}
 	}()
 	graceful.GetManager().RunWithShutdownContext(func(baseCtx context.Context) {
+		defer func() {
+			if err := recover(); err != nil {
+				// Recover a panic within the execution of the task.
+				combinedErr := fmt.Errorf("%s\n%s", err, log.Stack(2))
+				log.Error("PANIC whilst running task: %s Value: %v", t.Name, combinedErr)
+			}
+		}()
 		// Store the time of this run, before the function is executed, so it
 		// matches the behavior of what the cron library does.
 		t.lock.Lock()