Db retries (#146)

Address #93

This PR adds a `retry` method that can take in another function for
retry until a specific condition is met. `retry` can be configured with
functional options and has a `retryWithResult` generic counterpart for
functions returning a value.

All system database methods are now called within `retry` or
`retryWithResult`.

For example

```go
dequeuedWorkflows, err := retryWithResult(ctx, func() ([]dequeuedWorkflow, error) {
return ctx.systemDB.dequeueWorkflows(ctx, dequeueWorkflowsInput{
	queue:              queue,
	executorID:         ctx.executorID,
	applicationVersion: ctx.applicationVersion,
})
}, withRetrierLogger(qr.logger))
```

retry performs exponential backoff with jitter. It defaults to infinite
retries.


The default condition for retrying is `isRetryablePGError`, which
matches the function returned error with a set of connection errors
(postgres codes, pgx connection errors, `net.Error`)

Note: in the particular case of RunWorkflow, which does manage
transactions, the entire function has to be retried. This is because the
transaction object becomes invalid as soon as Commit/Rollback have been
called, regardless of whether the operation was successful. See
https://github.com/jackc/pgx/blob/61d3c965ad442cc14d6b0e39e0ab3821f3684c03/tx.go#L180


This PR also improves `DBOSError` with the ability to unwrap the
underlying error, if any.

Author: maxdml

.github/workflows/chaos-tests.yml

@@ -0,0 +1,44 @@
+name: Run Chaos Tests
+on:
+    schedule:
+        # Runs every hour on the hour
+        - cron: '0 * * * *'
+    push:
+      branches:
+        - main
+    pull_request:
+      branches:
+      types:
+        - ready_for_review
+        - opened
+        - reopened
+        - synchronize
+    workflow_dispatch:
+
+jobs:
+  chaos-test:
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    steps:
+    - uses: actions/checkout@v5
+      with:
+        fetch-depth: 0
+        fetch-tags: true
+
+    - name: Setup Go
+      uses: actions/setup-go@v5
+      with:
+        go-version: '1.25.x'
+
+    - name: Download dependencies
+      run: go mod download
+
+    - name: Install gotestsum
+      run: go install gotest.tools/gotestsum@latest
+
+    - name: Run chaos tests
+      run: go vet ./... && go test -v -race -timeout 60m -count=1 ./...
+      working-directory: ./chaos_tests
+      env:
+        PGPASSWORD: a!b@c$d()e*_,/:;=?@ff[]22
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/tests.yml

@@ -44,16 +44,6 @@ jobs:
       with:
         go-version: '1.25.x'
 
-    - name: Cache Go modules
-      uses: actions/cache@v4
-      with:
-        path: |
-          ~/go/pkg/mod
-          ~/.cache/go-build
-        key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
-        restore-keys: |
-          ${{ runner.os }}-go-
-
     - name: Download dependencies
       run: go mod download