Memory leak in workflow-controller when workflows are run in many unique namespaces #15194
Description
Pre-requisites
- I have double-checked my configuration
- I have tested with the
:latestimage tag (i.e.quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on:latest. If not, I have explained why, in detail, in my description below. - I have searched existing issues and could not find a match for this bug
- I'd like to contribute the fix myself (see contributing guide)
What happened? What did you expect to happen?
The eventRecorderManager in workflow/events/event_recorder_manager.go has a memory leak where EventBroadcaster instances are created per namespace and stored in m.eventRecorders[namespace], but are never removed or shut down.
- workflow controller processes item and calls
newWorkflowOperationCtxhere - It calls
wfc.eventRecorderManager.Gethere - When a new namespace is occurred, it creates and stores a new broadcaster in
m.eventRecorders[namespace]here
However, there is no mechanism to:
- Remove entries when namespaces are deleted
- Shutdown the
EventBroadcasterinstances - Limit the size of the map
This causes unbounded memory growth in long-running workflow-controller instances, especially if there are workflows running in new unique namespaces each time.
To reproduce
# Create a new cluster and install argo 3.7.6
kind create cluster --name argo-leak-repro
kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.7.6/install.yaml
# Enable pprof endpoints to capture heap dump
kubectl -n argo set env deployment/workflow-controller ARGO_PPROF=true
kubectl port-forward -n argo deployment/workflow-controller 6060:6060
# Capture baseline heap dump from a few workflows in 1 namespace
kubectl create namespace baseline-ns
for i in {1..4}; do
argo submit -n baseline-ns https://raw.githubusercontent.com/argoproj/argo-workflows/master/examples/hello-world.yaml
done
go tool pprof -top http://localhost:6060/debug/pprof/heap
# Create workflows in unique namespaces
ITERATIONS=100
for i in $(seq 1 $ITERATIONS); do
NS="leak-ns-$i"
echo "Processing $NS..."
kubectl create ns $NS
cat <<EOF | kubectl create -n $NS -f -
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: leak-test-
spec:
entrypoint: main
templates:
- name: main
container:
image: alpine
command: ["/bin/sh", "-c"]
args: ["echo hello"]
EOF
sleep 2
kubectl delete ns $NS --wait=false
done
# Capture heap dump
go tool pprof -top http://localhost:6060/debug/pprof/heap
# Compare heap dump
go tool pprof -top -base before.pb.gz after.pb.gz
File: workflow-controller
Build ID: 9effbac1baba83a1b1d2a2e161c69c6c4717f0a9
Type: inuse_space
Time: 2025-12-22 17:27:20 +08
Showing nodes accounting for 6308.82kB, 75.49% of 8357.01kB total
Dropped 11 nodes (cum <= 41.78kB)
flat flat% sum% cum cum%
2640.83kB 31.60% 31.60% 2640.83kB 31.60% k8s.io/apimachinery/pkg/watch.(*Broadcaster).Watch.func1
2640.83kB 31.60% 63.20% 2640.83kB 31.60% k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
1538.63kB 18.41% 81.61% 1538.63kB 18.41% go.opentelemetry.io/otel/sdk/metric/exemplar.newStorage (inline)
512.62kB 6.13% 87.75% 512.62kB 6.13% fmt.Sprintf
-512.05kB 6.13% 81.62% -512.05kB 6.13% golang.org/x/net/http2.(*ClientConn).roundTrip
-512.04kB 6.13% 75.49% -512.04kB 6.13% k8s.io/apimachinery/pkg/runtime/serializer/streaming.NewDecoder (inline)
0 0% 75.49% -512.05kB 6.13% github.com/argoproj/argo-workflows/v3/util/logs.k8sLogRoundTripper.RoundTrip
0 0% 75.49% 1538.63kB 18.41% github.com/argoproj/argo-workflows/v3/util/telemetry.(*Instrument).AddInt
0 0% 75.49% 1538.63kB 18.41% github.com/argoproj/argo-workflows/v3/util/telemetry.(*Metrics).AddInt
0 0% 75.49% 4179.46kB 50.01% github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).processNextItem
0 0% 75.49% 512.62kB 6.13% github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).runConfigMapWatcher.func1
0 0% 75.49% -512.05kB 6.13% github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).runConfigMapWatcher.func2
0 0% 75.49% 4179.46kB 50.01% github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).runWorker
0 0% 75.49% 1025.75kB 12.27% github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).markWorkflowError
0 0% 75.49% 1538.63kB 18.41% github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).markWorkflowPhase
0 0% 75.49% 512.88kB 6.14% github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).markWorkflowRunning (inline)
0 0% 75.49% 1538.63kB 18.41% github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate
0 0% 75.49% 1538.63kB 18.41% github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).recordWorkflowPhaseChange
0 0% 75.49% 2640.83kB 31.60% github.com/argoproj/argo-workflows/v3/workflow/controller.newWorkflowOperationCtx
0 0% 75.49% 2640.83kB 31.60% github.com/argoproj/argo-workflows/v3/workflow/events.(*eventRecorderManager).Get
0 0% 75.49% 1538.63kB 18.41% github.com/argoproj/argo-workflows/v3/workflow/metrics.(*Metrics).ChangeWorkflowPhase
0 0% 75.49% -512.05kB 6.13% github.com/argoproj/argo-workflows/v3/workflow/metrics.metricsRoundTripper.RoundTrip
0 0% 75.49% 1538.63kB 18.41% go.opentelemetry.io/otel/sdk/metric.(*inserter[go.shape.int64]).cachedAggregator.func1.reservoirFunc[go.shape.int64].1
0 0% 75.49% 1538.63kB 18.41% go.opentelemetry.io/otel/sdk/metric.(*int64Inst).Add
0 0% 75.49% 1538.63kB 18.41% go.opentelemetry.io/otel/sdk/metric.(*int64Inst).aggregate (inline)
0 0% 75.49% 1538.63kB 18.41% go.opentelemetry.io/otel/sdk/metric.DefaultExemplarReservoirProviderSelector.FixedSizeReservoirProvider.func1
0 0% 75.49% 1538.63kB 18.41% go.opentelemetry.io/otel/sdk/metric/exemplar.NewFixedSizeReservoir
0 0% 75.49% 1538.63kB 18.41% go.opentelemetry.io/otel/sdk/metric/internal/aggregate.(*valueMap[go.shape.int64]).measure
0 0% 75.49% 1538.63kB 18.41% go.opentelemetry.io/otel/sdk/metric/internal/aggregate.Builder[go.shape.int64].Sum.Builder[go.shape.int64].filter.func9
0 0% 75.49% -512.05kB 6.13% golang.org/x/net/http2.(*ClientConn).RoundTrip (inline)
0 0% 75.49% -512.05kB 6.13% golang.org/x/net/http2.(*Transport).RoundTrip (inline)
0 0% 75.49% -512.05kB 6.13% golang.org/x/net/http2.(*Transport).RoundTripOpt
0 0% 75.49% -512.05kB 6.13% golang.org/x/net/http2.noDialH2RoundTripper.RoundTrip
0 0% 75.49% -512.04kB 6.13% k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
0 0% 75.49% -512.04kB 6.13% k8s.io/apimachinery/pkg/util/wait.BackoffUntil
0 0% 75.49% -512.04kB 6.13% k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
0 0% 75.49% 3667.99kB 43.89% k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext
0 0% 75.49% 3667.99kB 43.89% k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1
0 0% 75.49% 4180.03kB 50.02% k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
0 0% 75.49% 4179.46kB 50.01% k8s.io/apimachinery/pkg/util/wait.UntilWithContext
0 0% 75.49% 2640.83kB 31.60% k8s.io/apimachinery/pkg/watch.(*Broadcaster).blockQueue.func1
0 0% 75.49% 2640.83kB 31.60% k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop
0 0% 75.49% -512.04kB 6.13% k8s.io/client-go/dynamic.(*dynamicResourceClient).Watch
0 0% 75.49% -512.04kB 6.13% k8s.io/client-go/dynamic/dynamicinformer.NewFilteredDynamicInformer.func4
0 0% 75.49% -511.47kB 6.12% k8s.io/client-go/rest.(*Request).Watch (inline)
0 0% 75.49% -512.04kB 6.13% k8s.io/client-go/rest.(*Request).newStreamWatcher
0 0% 75.49% -511.47kB 6.12% k8s.io/client-go/rest.(*Request).watchInternal
0 0% 75.49% -511.47kB 6.12% k8s.io/client-go/tools/cache.(*ListWatch).WatchWithContext
0 0% 75.49% -512.04kB 6.13% k8s.io/client-go/tools/cache.(*Reflector).ListAndWatchWithContext
0 0% 75.49% -512.04kB 6.13% k8s.io/client-go/tools/cache.(*Reflector).RunWithContext
0 0% 75.49% -512.04kB 6.13% k8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1
0 0% 75.49% -512.04kB 6.13% k8s.io/client-go/tools/cache.(*Reflector).watch
0 0% 75.49% -512.04kB 6.13% k8s.io/client-go/tools/cache.(*Reflector).watchWithResync
0 0% 75.49% -512.04kB 6.13% k8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3
0 0% 75.49% 2640.83kB 31.60% k8s.io/client-go/tools/record.NewBroadcaster
0 0% 75.49% 2640.83kB 31.60% k8s.io/client-go/tools/record.NewBroadcasterWithCorrelatorOptions
0 0% 75.49% -512.05kB 6.13% net/http.(*Transport).RoundTrip
0 0% 75.49% -512.05kB 6.13% net/http.(*Transport).roundTrip
Version(s)
v3.7.6
Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: leak-test-
spec:
entrypoint: main
templates:
- name: main
container:
image: alpine
command: ["/bin/sh", "-c"]
args: ["echo hello"]Logs from the workflow controller
kubectl logs -n argo deploy/workflow-controller | grep leak-test
time="2025-12-22T09:04:25.408Z" level=info msg="Processing workflow" Phase= ResourceVersion=1784 namespace=leak-ns-1 workflow=leak-test-pq55j
time="2025-12-22T09:04:25.410Z" level=info msg="Task-result reconciliation" namespace=leak-ns-1 numObjs=0 workflow=leak-test-pq55j
time="2025-12-22T09:04:25.410Z" level=info msg="Updated phase -> Running" namespace=leak-ns-1 workflow=leak-test-pq55j
time="2025-12-22T09:04:25.410Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=leak-ns-1 workflow=leak-test-pq55j
time="2025-12-22T09:04:25.410Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=leak-ns-1 workflow=leak-test-pq55j
time="2025-12-22T09:04:25.410Z" level=info msg="Pod node leak-test-pq55j initialized Pending" namespace=leak-ns-1 workflow=leak-test-pq55j
time="2025-12-22T09:04:25.410Z" level=warning msg="couldn't get boundaryTemplate through nodeName leak-test-pq55j" namespace=leak-ns-1 workflow=leak-test-pq55j
time="2025-12-22T09:04:25.415Z" level=info msg="add pod event" pod=leak-test-pq55j
time="2025-12-22T09:04:25.415Z" level=info msg="Created pod: leak-test-pq55j (leak-test-pq55j)" namespace=leak-ns-1 workflow=leak-test-pq55j
time="2025-12-22T09:04:25.415Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-1 workflow=leak-test-pq55j
time="2025-12-22T09:04:25.415Z" level=info msg=reconcileAgentPod namespace=leak-ns-1 workflow=leak-test-pq55j
time="2025-12-22T09:04:25.418Z" level=info msg="Workflow update successful" namespace=leak-ns-1 phase=Running resourceVersion=1787 workflow=leak-test-pq55j
time="2025-12-22T09:04:25.419Z" level=info msg="update pod event" pod=leak-test-pq55j
time="2025-12-22T09:04:25.419Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1787 namespace=leak-ns-1 workflow=leak-test-pq55j
time="2025-12-22T09:04:25.419Z" level=info msg="Task-result reconciliation" namespace=leak-ns-1 numObjs=0 workflow=leak-test-pq55j
time="2025-12-22T09:04:25.419Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-1 workflow=leak-test-pq55j
time="2025-12-22T09:04:25.419Z" level=info msg=reconcileAgentPod namespace=leak-ns-1 workflow=leak-test-pq55j
time="2025-12-22T09:04:25.422Z" level=info msg="Workflow update successful" namespace=leak-ns-1 phase=Running resourceVersion=1790 workflow=leak-test-pq55j
time="2025-12-22T09:04:25.426Z" level=info msg="update pod event" pod=leak-test-pq55j
time="2025-12-22T09:04:26.381Z" level=info msg="update pod event" pod=leak-test-pq55j
time="2025-12-22T09:04:27.620Z" level=info msg="Processing workflow" Phase= ResourceVersion=1810 namespace=leak-ns-2 workflow=leak-test-nclrq
time="2025-12-22T09:04:27.621Z" level=info msg="Task-result reconciliation" namespace=leak-ns-2 numObjs=0 workflow=leak-test-nclrq
time="2025-12-22T09:04:27.621Z" level=info msg="Updated phase -> Running" namespace=leak-ns-2 workflow=leak-test-nclrq
time="2025-12-22T09:04:27.622Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=leak-ns-2 workflow=leak-test-nclrq
time="2025-12-22T09:04:27.622Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=leak-ns-2 workflow=leak-test-nclrq
time="2025-12-22T09:04:27.622Z" level=info msg="Pod node leak-test-nclrq initialized Pending" namespace=leak-ns-2 workflow=leak-test-nclrq
time="2025-12-22T09:04:27.622Z" level=warning msg="couldn't get boundaryTemplate through nodeName leak-test-nclrq" namespace=leak-ns-2 workflow=leak-test-nclrq
time="2025-12-22T09:04:27.627Z" level=info msg="Created pod: leak-test-nclrq (leak-test-nclrq)" namespace=leak-ns-2 workflow=leak-test-nclrq
time="2025-12-22T09:04:27.627Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-2 workflow=leak-test-nclrq
time="2025-12-22T09:04:27.627Z" level=info msg=reconcileAgentPod namespace=leak-ns-2 workflow=leak-test-nclrq
time="2025-12-22T09:04:27.627Z" level=info msg="add pod event" pod=leak-test-nclrq
time="2025-12-22T09:04:27.629Z" level=info msg="Workflow update successful" namespace=leak-ns-2 phase=Running resourceVersion=1813 workflow=leak-test-nclrq
time="2025-12-22T09:04:27.630Z" level=info msg="update pod event" pod=leak-test-nclrq
time="2025-12-22T09:04:27.638Z" level=info msg="update pod event" pod=leak-test-nclrq
time="2025-12-22T09:04:28.398Z" level=info msg="update pod event" pod=leak-test-nclrq
time="2025-12-22T09:04:29.850Z" level=info msg="Processing workflow" Phase= ResourceVersion=1832 namespace=leak-ns-3 workflow=leak-test-gljq9
time="2025-12-22T09:04:29.853Z" level=info msg="Task-result reconciliation" namespace=leak-ns-3 numObjs=0 workflow=leak-test-gljq9
time="2025-12-22T09:04:29.853Z" level=info msg="Updated phase -> Running" namespace=leak-ns-3 workflow=leak-test-gljq9
time="2025-12-22T09:04:29.854Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=leak-ns-3 workflow=leak-test-gljq9
time="2025-12-22T09:04:29.854Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=leak-ns-3 workflow=leak-test-gljq9
time="2025-12-22T09:04:29.854Z" level=info msg="Pod node leak-test-gljq9 initialized Pending" namespace=leak-ns-3 workflow=leak-test-gljq9
time="2025-12-22T09:04:29.854Z" level=warning msg="couldn't get boundaryTemplate through nodeName leak-test-gljq9" namespace=leak-ns-3 workflow=leak-test-gljq9
time="2025-12-22T09:04:29.860Z" level=info msg="Created pod: leak-test-gljq9 (leak-test-gljq9)" namespace=leak-ns-3 workflow=leak-test-gljq9
time="2025-12-22T09:04:29.860Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-3 workflow=leak-test-gljq9
time="2025-12-22T09:04:29.860Z" level=info msg=reconcileAgentPod namespace=leak-ns-3 workflow=leak-test-gljq9
time="2025-12-22T09:04:29.860Z" level=info msg="add pod event" pod=leak-test-gljq9
time="2025-12-22T09:04:29.864Z" level=info msg="update pod event" pod=leak-test-gljq9
time="2025-12-22T09:04:29.864Z" level=info msg="Workflow update successful" namespace=leak-ns-3 phase=Running resourceVersion=1836 workflow=leak-test-gljq9
time="2025-12-22T09:04:29.864Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1836 namespace=leak-ns-3 workflow=leak-test-gljq9
time="2025-12-22T09:04:29.864Z" level=info msg="Task-result reconciliation" namespace=leak-ns-3 numObjs=0 workflow=leak-test-gljq9
time="2025-12-22T09:04:29.864Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-3 workflow=leak-test-gljq9
time="2025-12-22T09:04:29.864Z" level=info msg=reconcileAgentPod namespace=leak-ns-3 workflow=leak-test-gljq9
time="2025-12-22T09:04:29.867Z" level=info msg="Workflow update successful" namespace=leak-ns-3 phase=Running resourceVersion=1838 workflow=leak-test-gljq9
time="2025-12-22T09:04:29.868Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1838 namespace=leak-ns-3 workflow=leak-test-gljq9
time="2025-12-22T09:04:29.868Z" level=info msg="Task-result reconciliation" namespace=leak-ns-3 numObjs=0 workflow=leak-test-gljq9
time="2025-12-22T09:04:29.868Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-3 workflow=leak-test-gljq9
time="2025-12-22T09:04:29.868Z" level=info msg=reconcileAgentPod namespace=leak-ns-3 workflow=leak-test-gljq9
time="2025-12-22T09:04:29.872Z" level=info msg="update pod event" pod=leak-test-gljq9
time="2025-12-22T09:04:30.448Z" level=info msg="update pod event" pod=leak-test-gljq9
time="2025-12-22T09:04:31.434Z" level=info msg="update pod event" pod=leak-test-gljq9
time="2025-12-22T09:04:32.065Z" level=info msg="Processing workflow" Phase= ResourceVersion=1855 namespace=leak-ns-4 workflow=leak-test-2pwsb
time="2025-12-22T09:04:32.066Z" level=info msg="Task-result reconciliation" namespace=leak-ns-4 numObjs=0 workflow=leak-test-2pwsb
time="2025-12-22T09:04:32.066Z" level=info msg="Updated phase -> Running" namespace=leak-ns-4 workflow=leak-test-2pwsb
time="2025-12-22T09:04:32.066Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=leak-ns-4 workflow=leak-test-2pwsb
time="2025-12-22T09:04:32.066Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=leak-ns-4 workflow=leak-test-2pwsb
time="2025-12-22T09:04:32.066Z" level=info msg="Pod node leak-test-2pwsb initialized Pending" namespace=leak-ns-4 workflow=leak-test-2pwsb
time="2025-12-22T09:04:32.066Z" level=warning msg="couldn't get boundaryTemplate through nodeName leak-test-2pwsb" namespace=leak-ns-4 workflow=leak-test-2pwsb
time="2025-12-22T09:04:32.070Z" level=info msg="Created pod: leak-test-2pwsb (leak-test-2pwsb)" namespace=leak-ns-4 workflow=leak-test-2pwsb
time="2025-12-22T09:04:32.070Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-4 workflow=leak-test-2pwsb
time="2025-12-22T09:04:32.070Z" level=info msg=reconcileAgentPod namespace=leak-ns-4 workflow=leak-test-2pwsb
time="2025-12-22T09:04:32.070Z" level=info msg="add pod event" pod=leak-test-2pwsb
time="2025-12-22T09:04:32.072Z" level=info msg="Workflow update successful" namespace=leak-ns-4 phase=Running resourceVersion=1858 workflow=leak-test-2pwsb
time="2025-12-22T09:04:32.072Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1858 namespace=leak-ns-4 workflow=leak-test-2pwsb
time="2025-12-22T09:04:32.072Z" level=info msg="Task-result reconciliation" namespace=leak-ns-4 numObjs=0 workflow=leak-test-2pwsb
time="2025-12-22T09:04:32.073Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-4 workflow=leak-test-2pwsb
time="2025-12-22T09:04:32.073Z" level=info msg=reconcileAgentPod namespace=leak-ns-4 workflow=leak-test-2pwsb
time="2025-12-22T09:04:32.073Z" level=info msg="update pod event" pod=leak-test-2pwsb
time="2025-12-22T09:04:32.082Z" level=info msg="update pod event" pod=leak-test-2pwsb
time="2025-12-22T09:04:32.511Z" level=info msg="update pod event" pod=leak-test-pq55j
time="2025-12-22T09:04:33.448Z" level=info msg="update pod event" pod=leak-test-2pwsb
time="2025-12-22T09:04:34.283Z" level=info msg="Processing workflow" Phase= ResourceVersion=1880 namespace=leak-ns-5 workflow=leak-test-nv22s
time="2025-12-22T09:04:34.284Z" level=info msg="Task-result reconciliation" namespace=leak-ns-5 numObjs=0 workflow=leak-test-nv22s
time="2025-12-22T09:04:34.284Z" level=info msg="Updated phase -> Running" namespace=leak-ns-5 workflow=leak-test-nv22s
time="2025-12-22T09:04:34.284Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=leak-ns-5 workflow=leak-test-nv22s
time="2025-12-22T09:04:34.284Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=leak-ns-5 workflow=leak-test-nv22s
time="2025-12-22T09:04:34.284Z" level=info msg="Pod node leak-test-nv22s initialized Pending" namespace=leak-ns-5 workflow=leak-test-nv22s
time="2025-12-22T09:04:34.284Z" level=warning msg="couldn't get boundaryTemplate through nodeName leak-test-nv22s" namespace=leak-ns-5 workflow=leak-test-nv22s
time="2025-12-22T09:04:34.290Z" level=info msg="Created pod: leak-test-nv22s (leak-test-nv22s)" namespace=leak-ns-5 workflow=leak-test-nv22s
time="2025-12-22T09:04:34.290Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-5 workflow=leak-test-nv22s
time="2025-12-22T09:04:34.290Z" level=info msg=reconcileAgentPod namespace=leak-ns-5 workflow=leak-test-nv22s
time="2025-12-22T09:04:34.290Z" level=info msg="add pod event" pod=leak-test-nv22s
time="2025-12-22T09:04:34.293Z" level=info msg="Workflow update successful" namespace=leak-ns-5 phase=Running resourceVersion=1883 workflow=leak-test-nv22s
time="2025-12-22T09:04:34.293Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1883 namespace=leak-ns-5 workflow=leak-test-nv22s
time="2025-12-22T09:04:34.293Z" level=info msg="Task-result reconciliation" namespace=leak-ns-5 numObjs=0 workflow=leak-test-nv22s
time="2025-12-22T09:04:34.293Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-5 workflow=leak-test-nv22s
time="2025-12-22T09:04:34.293Z" level=info msg=reconcileAgentPod namespace=leak-ns-5 workflow=leak-test-nv22s
time="2025-12-22T09:04:34.294Z" level=info msg="update pod event" pod=leak-test-nv22s
time="2025-12-22T09:04:34.302Z" level=info msg="update pod event" pod=leak-test-nv22s
time="2025-12-22T09:04:34.733Z" level=info msg="update pod event" pod=leak-test-nclrq
time="2025-12-22T09:04:35.416Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1790 namespace=leak-ns-1 workflow=leak-test-pq55j
time="2025-12-22T09:04:35.417Z" level=info msg="Task-result reconciliation" namespace=leak-ns-1 numObjs=0 workflow=leak-test-pq55j
time="2025-12-22T09:04:35.417Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-1 workflow=leak-test-pq55j
time="2025-12-22T09:04:35.417Z" level=info msg=reconcileAgentPod namespace=leak-ns-1 workflow=leak-test-pq55j
time="2025-12-22T09:04:35.428Z" level=info msg="Workflow update successful" namespace=leak-ns-1 phase=Running resourceVersion=1892 workflow=leak-test-pq55j
time="2025-12-22T09:04:35.428Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1892 namespace=leak-ns-1 workflow=leak-test-pq55j
time="2025-12-22T09:04:35.429Z" level=info msg="Task-result reconciliation" namespace=leak-ns-1 numObjs=0 workflow=leak-test-pq55j
time="2025-12-22T09:04:35.429Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-1 workflow=leak-test-pq55j
time="2025-12-22T09:04:35.429Z" level=info msg=reconcileAgentPod namespace=leak-ns-1 workflow=leak-test-pq55j
time="2025-12-22T09:04:35.454Z" level=info msg="update pod event" pod=leak-test-nv22s
time="2025-12-22T09:04:35.461Z" level=info msg="update pod event" pod=leak-test-pq55j
time="2025-12-22T09:04:36.042Z" level=info msg="update pod event" pod=leak-test-pq55j
time="2025-12-22T09:04:36.459Z" level=info msg="update pod event" pod=leak-test-pq55j
time="2025-12-22T09:04:36.461Z" level=info msg="delete pod event" pod=leak-test-pq55j
time="2025-12-22T09:04:36.488Z" level=info msg="Processing workflow" Phase= ResourceVersion=1908 namespace=leak-ns-6 workflow=leak-test-6xscz
time="2025-12-22T09:04:36.489Z" level=info msg="Task-result reconciliation" namespace=leak-ns-6 numObjs=0 workflow=leak-test-6xscz
time="2025-12-22T09:04:36.489Z" level=info msg="Updated phase -> Running" namespace=leak-ns-6 workflow=leak-test-6xscz
time="2025-12-22T09:04:36.489Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=leak-ns-6 workflow=leak-test-6xscz
time="2025-12-22T09:04:36.489Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=leak-ns-6 workflow=leak-test-6xscz
time="2025-12-22T09:04:36.489Z" level=info msg="Pod node leak-test-6xscz initialized Pending" namespace=leak-ns-6 workflow=leak-test-6xscz
time="2025-12-22T09:04:36.489Z" level=warning msg="couldn't get boundaryTemplate through nodeName leak-test-6xscz" namespace=leak-ns-6 workflow=leak-test-6xscz
time="2025-12-22T09:04:36.493Z" level=info msg="Created pod: leak-test-6xscz (leak-test-6xscz)" namespace=leak-ns-6 workflow=leak-test-6xscz
time="2025-12-22T09:04:36.493Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-6 workflow=leak-test-6xscz
time="2025-12-22T09:04:36.493Z" level=info msg=reconcileAgentPod namespace=leak-ns-6 workflow=leak-test-6xscz
time="2025-12-22T09:04:36.494Z" level=info msg="add pod event" pod=leak-test-6xscz
time="2025-12-22T09:04:36.496Z" level=info msg="Workflow update successful" namespace=leak-ns-6 phase=Running resourceVersion=1911 workflow=leak-test-6xscz
time="2025-12-22T09:04:36.497Z" level=info msg="update pod event" pod=leak-test-6xscz
time="2025-12-22T09:04:36.504Z" level=info msg="update pod event" pod=leak-test-6xscz
time="2025-12-22T09:04:36.948Z" level=info msg="update pod event" pod=leak-test-gljq9
time="2025-12-22T09:04:37.481Z" level=info msg="update pod event" pod=leak-test-6xscz
time="2025-12-22T09:04:37.488Z" level=info msg="update pod event" pod=leak-test-nclrq
time="2025-12-22T09:04:37.629Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1813 namespace=leak-ns-2 workflow=leak-test-nclrq
time="2025-12-22T09:04:37.629Z" level=info msg="Task-result reconciliation" namespace=leak-ns-2 numObjs=0 workflow=leak-test-nclrq
time="2025-12-22T09:04:37.629Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-2 workflow=leak-test-nclrq
time="2025-12-22T09:04:37.629Z" level=info msg=reconcileAgentPod namespace=leak-ns-2 workflow=leak-test-nclrq
time="2025-12-22T09:04:37.631Z" level=info msg="Workflow update successful" namespace=leak-ns-2 phase=Running resourceVersion=1941 workflow=leak-test-nclrq
time="2025-12-22T09:04:37.631Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1941 namespace=leak-ns-2 workflow=leak-test-nclrq
time="2025-12-22T09:04:37.631Z" level=info msg="Task-result reconciliation" namespace=leak-ns-2 numObjs=0 workflow=leak-test-nclrq
time="2025-12-22T09:04:37.631Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-2 workflow=leak-test-nclrq
time="2025-12-22T09:04:37.632Z" level=info msg=reconcileAgentPod namespace=leak-ns-2 workflow=leak-test-nclrq
time="2025-12-22T09:04:38.190Z" level=info msg="update pod event" pod=leak-test-nclrq
time="2025-12-22T09:04:38.487Z" level=info msg="update pod event" pod=leak-test-nclrq
time="2025-12-22T09:04:38.492Z" level=info msg="delete pod event" pod=leak-test-nclrq
time="2025-12-22T09:04:38.705Z" level=info msg="Processing workflow" Phase= ResourceVersion=1951 namespace=leak-ns-7 workflow=leak-test-tk68p
time="2025-12-22T09:04:38.706Z" level=info msg="Task-result reconciliation" namespace=leak-ns-7 numObjs=0 workflow=leak-test-tk68p
time="2025-12-22T09:04:38.706Z" level=info msg="Updated phase -> Running" namespace=leak-ns-7 workflow=leak-test-tk68p
time="2025-12-22T09:04:38.706Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=leak-ns-7 workflow=leak-test-tk68p
time="2025-12-22T09:04:38.706Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=leak-ns-7 workflow=leak-test-tk68p
time="2025-12-22T09:04:38.706Z" level=info msg="Pod node leak-test-tk68p initialized Pending" namespace=leak-ns-7 workflow=leak-test-tk68p
time="2025-12-22T09:04:38.706Z" level=warning msg="couldn't get boundaryTemplate through nodeName leak-test-tk68p" namespace=leak-ns-7 workflow=leak-test-tk68p
time="2025-12-22T09:04:38.710Z" level=info msg="Created pod: leak-test-tk68p (leak-test-tk68p)" namespace=leak-ns-7 workflow=leak-test-tk68p
time="2025-12-22T09:04:38.710Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-7 workflow=leak-test-tk68p
time="2025-12-22T09:04:38.710Z" level=info msg=reconcileAgentPod namespace=leak-ns-7 workflow=leak-test-tk68p
time="2025-12-22T09:04:38.710Z" level=info msg="add pod event" pod=leak-test-tk68p
time="2025-12-22T09:04:38.712Z" level=info msg="Workflow update successful" namespace=leak-ns-7 phase=Running resourceVersion=1954 workflow=leak-test-tk68p
time="2025-12-22T09:04:38.713Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1954 namespace=leak-ns-7 workflow=leak-test-tk68p
time="2025-12-22T09:04:38.713Z" level=info msg="Task-result reconciliation" namespace=leak-ns-7 numObjs=0 workflow=leak-test-tk68p
time="2025-12-22T09:04:38.713Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-7 workflow=leak-test-tk68p
time="2025-12-22T09:04:38.713Z" level=info msg=reconcileAgentPod namespace=leak-ns-7 workflow=leak-test-tk68p
time="2025-12-22T09:04:38.714Z" level=info msg="update pod event" pod=leak-test-tk68p
time="2025-12-22T09:04:38.721Z" level=info msg="update pod event" pod=leak-test-tk68p
time="2025-12-22T09:04:39.171Z" level=info msg="update pod event" pod=leak-test-2pwsb
time="2025-12-22T09:04:39.494Z" level=info msg="update pod event" pod=leak-test-tk68p
time="2025-12-22T09:04:39.500Z" level=info msg="update pod event" pod=leak-test-gljq9
time="2025-12-22T09:04:39.861Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1838 namespace=leak-ns-3 workflow=leak-test-gljq9
time="2025-12-22T09:04:39.861Z" level=info msg="Task-result reconciliation" namespace=leak-ns-3 numObjs=0 workflow=leak-test-gljq9
time="2025-12-22T09:04:39.861Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-3 workflow=leak-test-gljq9
time="2025-12-22T09:04:39.861Z" level=info msg=reconcileAgentPod namespace=leak-ns-3 workflow=leak-test-gljq9
time="2025-12-22T09:04:39.863Z" level=info msg="Workflow update successful" namespace=leak-ns-3 phase=Running resourceVersion=1985 workflow=leak-test-gljq9
time="2025-12-22T09:04:40.600Z" level=info msg="update pod event" pod=leak-test-gljq9
time="2025-12-22T09:04:40.928Z" level=info msg="Processing workflow" Phase= ResourceVersion=1991 namespace=leak-ns-8 workflow=leak-test-flt4t
time="2025-12-22T09:04:40.929Z" level=info msg="Task-result reconciliation" namespace=leak-ns-8 numObjs=0 workflow=leak-test-flt4t
time="2025-12-22T09:04:40.929Z" level=info msg="Updated phase -> Running" namespace=leak-ns-8 workflow=leak-test-flt4t
time="2025-12-22T09:04:40.930Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=leak-ns-8 workflow=leak-test-flt4t
time="2025-12-22T09:04:40.930Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=leak-ns-8 workflow=leak-test-flt4t
time="2025-12-22T09:04:40.930Z" level=info msg="Pod node leak-test-flt4t initialized Pending" namespace=leak-ns-8 workflow=leak-test-flt4t
time="2025-12-22T09:04:40.930Z" level=warning msg="couldn't get boundaryTemplate through nodeName leak-test-flt4t" namespace=leak-ns-8 workflow=leak-test-flt4t
time="2025-12-22T09:04:40.936Z" level=info msg="Created pod: leak-test-flt4t (leak-test-flt4t)" namespace=leak-ns-8 workflow=leak-test-flt4t
time="2025-12-22T09:04:40.936Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-8 workflow=leak-test-flt4t
time="2025-12-22T09:04:40.936Z" level=info msg=reconcileAgentPod namespace=leak-ns-8 workflow=leak-test-flt4t
time="2025-12-22T09:04:40.937Z" level=info msg="add pod event" pod=leak-test-flt4t
time="2025-12-22T09:04:40.939Z" level=info msg="update pod event" pod=leak-test-flt4t
time="2025-12-22T09:04:40.939Z" level=info msg="Workflow update successful" namespace=leak-ns-8 phase=Running resourceVersion=1995 workflow=leak-test-flt4t
time="2025-12-22T09:04:40.939Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1995 namespace=leak-ns-8 workflow=leak-test-flt4t
time="2025-12-22T09:04:40.939Z" level=info msg="Task-result reconciliation" namespace=leak-ns-8 numObjs=0 workflow=leak-test-flt4t
time="2025-12-22T09:04:40.941Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-8 workflow=leak-test-flt4t
time="2025-12-22T09:04:40.941Z" level=info msg=reconcileAgentPod namespace=leak-ns-8 workflow=leak-test-flt4t
time="2025-12-22T09:04:40.955Z" level=info msg="Workflow update successful" namespace=leak-ns-8 phase=Running resourceVersion=1997 workflow=leak-test-flt4t
time="2025-12-22T09:04:40.956Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1997 namespace=leak-ns-8 workflow=leak-test-flt4t
time="2025-12-22T09:04:40.956Z" level=info msg="Task-result reconciliation" namespace=leak-ns-8 numObjs=0 workflow=leak-test-flt4t
time="2025-12-22T09:04:40.956Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-8 workflow=leak-test-flt4t
time="2025-12-22T09:04:40.956Z" level=info msg=reconcileAgentPod namespace=leak-ns-8 workflow=leak-test-flt4t
time="2025-12-22T09:04:40.958Z" level=info msg="update pod event" pod=leak-test-flt4t
time="2025-12-22T09:04:41.371Z" level=info msg="update pod event" pod=leak-test-nv22s
time="2025-12-22T09:04:41.547Z" level=info msg="update pod event" pod=leak-test-flt4t
time="2025-12-22T09:04:41.553Z" level=info msg="update pod event" pod=leak-test-gljq9
time="2025-12-22T09:04:41.555Z" level=info msg="delete pod event" pod=leak-test-gljq9
time="2025-12-22T09:04:42.072Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1858 namespace=leak-ns-4 workflow=leak-test-2pwsb
time="2025-12-22T09:04:42.072Z" level=info msg="Task-result reconciliation" namespace=leak-ns-4 numObjs=0 workflow=leak-test-2pwsb
time="2025-12-22T09:04:42.072Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-4 workflow=leak-test-2pwsb
time="2025-12-22T09:04:42.072Z" level=info msg=reconcileAgentPod namespace=leak-ns-4 workflow=leak-test-2pwsb
time="2025-12-22T09:04:42.074Z" level=info msg="Workflow update successful" namespace=leak-ns-4 phase=Running resourceVersion=2022 workflow=leak-test-2pwsb
time="2025-12-22T09:04:42.074Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=2022 namespace=leak-ns-4 workflow=leak-test-2pwsb
time="2025-12-22T09:04:42.074Z" level=info msg="Task-result reconciliation" namespace=leak-ns-4 numObjs=0 workflow=leak-test-2pwsb
time="2025-12-22T09:04:42.074Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-4 workflow=leak-test-2pwsb
time="2025-12-22T09:04:42.074Z" level=info msg=reconcileAgentPod namespace=leak-ns-4 workflow=leak-test-2pwsb
time="2025-12-22T09:04:42.530Z" level=info msg="update pod event" pod=leak-test-flt4t
time="2025-12-22T09:04:42.538Z" level=info msg="update pod event" pod=leak-test-2pwsb
time="2025-12-22T09:04:42.809Z" level=info msg="update pod event" pod=leak-test-2pwsb
time="2025-12-22T09:04:43.140Z" level=info msg="Processing workflow" Phase= ResourceVersion=2034 namespace=leak-ns-9 workflow=leak-test-pgw9r
time="2025-12-22T09:04:43.141Z" level=info msg="Task-result reconciliation" namespace=leak-ns-9 numObjs=0 workflow=leak-test-pgw9r
time="2025-12-22T09:04:43.141Z" level=info msg="Updated phase -> Running" namespace=leak-ns-9 workflow=leak-test-pgw9r
time="2025-12-22T09:04:43.142Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=leak-ns-9 workflow=leak-test-pgw9r
time="2025-12-22T09:04:43.142Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=leak-ns-9 workflow=leak-test-pgw9r
time="2025-12-22T09:04:43.142Z" level=info msg="Pod node leak-test-pgw9r initialized Pending" namespace=leak-ns-9 workflow=leak-test-pgw9r
time="2025-12-22T09:04:43.142Z" level=warning msg="couldn't get boundaryTemplate through nodeName leak-test-pgw9r" namespace=leak-ns-9 workflow=leak-test-pgw9r
time="2025-12-22T09:04:43.146Z" level=info msg="Created pod: leak-test-pgw9r (leak-test-pgw9r)" namespace=leak-ns-9 workflow=leak-test-pgw9r
time="2025-12-22T09:04:43.147Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-9 workflow=leak-test-pgw9r
time="2025-12-22T09:04:43.147Z" level=info msg=reconcileAgentPod namespace=leak-ns-9 workflow=leak-test-pgw9r
time="2025-12-22T09:04:43.146Z" level=info msg="add pod event" pod=leak-test-pgw9r
time="2025-12-22T09:04:43.149Z" level=info msg="Workflow update successful" namespace=leak-ns-9 phase=Running resourceVersion=2037 workflow=leak-test-pgw9r
time="2025-12-22T09:04:43.149Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=2037 namespace=leak-ns-9 workflow=leak-test-pgw9r
time="2025-12-22T09:04:43.149Z" level=info msg="Task-result reconciliation" namespace=leak-ns-9 numObjs=0 workflow=leak-test-pgw9r
time="2025-12-22T09:04:43.149Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-9 workflow=leak-test-pgw9r
time="2025-12-22T09:04:43.149Z" level=info msg=reconcileAgentPod namespace=leak-ns-9 workflow=leak-test-pgw9r
time="2025-12-22T09:04:43.150Z" level=info msg="update pod event" pod=leak-test-pgw9r
time="2025-12-22T09:04:43.157Z" level=info msg="update pod event" pod=leak-test-pgw9r
time="2025-12-22T09:04:43.534Z" level=info msg="update pod event" pod=leak-test-2pwsb
time="2025-12-22T09:04:43.536Z" level=info msg="delete pod event" pod=leak-test-2pwsb
time="2025-12-22T09:04:43.585Z" level=info msg="update pod event" pod=leak-test-6xscz
time="2025-12-22T09:04:44.290Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1883 namespace=leak-ns-5 workflow=leak-test-nv22s
time="2025-12-22T09:04:44.290Z" level=info msg="Task-result reconciliation" namespace=leak-ns-5 numObjs=0 workflow=leak-test-nv22s
time="2025-12-22T09:04:44.290Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-5 workflow=leak-test-nv22s
time="2025-12-22T09:04:44.290Z" level=info msg=reconcileAgentPod namespace=leak-ns-5 workflow=leak-test-nv22s
time="2025-12-22T09:04:44.293Z" level=info msg="Workflow update successful" namespace=leak-ns-5 phase=Running resourceVersion=2065 workflow=leak-test-nv22s
time="2025-12-22T09:04:44.294Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=2065 namespace=leak-ns-5 workflow=leak-test-nv22s
time="2025-12-22T09:04:44.294Z" level=info msg="Task-result reconciliation" namespace=leak-ns-5 numObjs=0 workflow=leak-test-nv22s
time="2025-12-22T09:04:44.294Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-5 workflow=leak-test-nv22s
time="2025-12-22T09:04:44.294Z" level=info msg=reconcileAgentPod namespace=leak-ns-5 workflow=leak-test-nv22s
time="2025-12-22T09:04:44.547Z" level=info msg="update pod event" pod=leak-test-nv22s
time="2025-12-22T09:04:44.554Z" level=info msg="update pod event" pod=leak-test-pgw9r
time="2025-12-22T09:04:45.063Z" level=info msg="update pod event" pod=leak-test-nv22s
time="2025-12-22T09:04:45.354Z" level=info msg="Processing workflow" Phase= ResourceVersion=2077 namespace=leak-ns-10 workflow=leak-test-scpjf
time="2025-12-22T09:04:45.355Z" level=info msg="Task-result reconciliation" namespace=leak-ns-10 numObjs=0 workflow=leak-test-scpjf
time="2025-12-22T09:04:45.355Z" level=info msg="Updated phase -> Running" namespace=leak-ns-10 workflow=leak-test-scpjf
time="2025-12-22T09:04:45.355Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=leak-ns-10 workflow=leak-test-scpjf
time="2025-12-22T09:04:45.355Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=leak-ns-10 workflow=leak-test-scpjf
time="2025-12-22T09:04:45.355Z" level=info msg="Pod node leak-test-scpjf initialized Pending" namespace=leak-ns-10 workflow=leak-test-scpjf
time="2025-12-22T09:04:45.355Z" level=warning msg="couldn't get boundaryTemplate through nodeName leak-test-scpjf" namespace=leak-ns-10 workflow=leak-test-scpjf
time="2025-12-22T09:04:45.360Z" level=info msg="add pod event" pod=leak-test-scpjf
time="2025-12-22T09:04:45.360Z" level=info msg="Created pod: leak-test-scpjf (leak-test-scpjf)" namespace=leak-ns-10 workflow=leak-test-scpjf
time="2025-12-22T09:04:45.360Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-10 workflow=leak-test-scpjf
time="2025-12-22T09:04:45.360Z" level=info msg=reconcileAgentPod namespace=leak-ns-10 workflow=leak-test-scpjf
time="2025-12-22T09:04:45.363Z" level=info msg="update pod event" pod=leak-test-scpjf
time="2025-12-22T09:04:45.363Z" level=info msg="Workflow update successful" namespace=leak-ns-10 phase=Running resourceVersion=2081 workflow=leak-test-scpjf
time="2025-12-22T09:04:45.364Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=2081 namespace=leak-ns-10 workflow=leak-test-scpjf
time="2025-12-22T09:04:45.364Z" level=info msg="Task-result reconciliation" namespace=leak-ns-10 numObjs=0 workflow=leak-test-scpjf
time="2025-12-22T09:04:45.364Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-10 workflow=leak-test-scpjf
time="2025-12-22T09:04:45.364Z" level=info msg=reconcileAgentPod namespace=leak-ns-10 workflow=leak-test-scpjf
time="2025-12-22T09:04:45.366Z" level=info msg="Workflow update successful" namespace=leak-ns-10 phase=Running resourceVersion=2083 workflow=leak-test-scpjf
time="2025-12-22T09:04:45.367Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=2083 namespace=leak-ns-10 workflow=leak-test-scpjf
time="2025-12-22T09:04:45.367Z" level=info msg="Task-result reconciliation" namespace=leak-ns-10 numObjs=0 workflow=leak-test-scpjf
time="2025-12-22T09:04:45.367Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-10 workflow=leak-test-scpjf
time="2025-12-22T09:04:45.367Z" level=info msg=reconcileAgentPod namespace=leak-ns-10 workflow=leak-test-scpjf
time="2025-12-22T09:04:45.370Z" level=info msg="update pod event" pod=leak-test-scpjf
time="2025-12-22T09:04:45.568Z" level=info msg="update pod event" pod=leak-test-nv22s
time="2025-12-22T09:04:45.570Z" level=info msg="delete pod event" pod=leak-test-nv22s
time="2025-12-22T09:04:45.811Z" level=info msg="update pod event" pod=leak-test-tk68p
time="2025-12-22T09:04:46.496Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1911 namespace=leak-ns-6 workflow=leak-test-6xscz
time="2025-12-22T09:04:46.496Z" level=info msg="Task-result reconciliation" namespace=leak-ns-6 numObjs=0 workflow=leak-test-6xscz
time="2025-12-22T09:04:46.496Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-6 workflow=leak-test-6xscz
time="2025-12-22T09:04:46.496Z" level=info msg=reconcileAgentPod namespace=leak-ns-6 workflow=leak-test-6xscz
time="2025-12-22T09:04:46.499Z" level=info msg="Workflow update successful" namespace=leak-ns-6 phase=Running resourceVersion=2107 workflow=leak-test-6xscz
time="2025-12-22T09:04:46.499Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=2107 namespace=leak-ns-6 workflow=leak-test-6xscz
time="2025-12-22T09:04:46.499Z" level=info msg="Task-result reconciliation" namespace=leak-ns-6 numObjs=0 workflow=leak-test-6xscz
time="2025-12-22T09:04:46.499Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-6 workflow=leak-test-6xscz
time="2025-12-22T09:04:46.499Z" level=info msg=reconcileAgentPod namespace=leak-ns-6 workflow=leak-test-6xscz
time="2025-12-22T09:04:46.571Z" level=info msg="update pod event" pod=leak-test-scpjf
time="2025-12-22T09:04:46.577Z" level=info msg="update pod event" pod=leak-test-6xscz
time="2025-12-22T09:04:47.404Z" level=info msg="update pod event" pod=leak-test-6xscz
time="2025-12-22T09:04:47.566Z" level=info msg="Processing workflow" Phase= ResourceVersion=2119 namespace=leak-ns-11 workflow=leak-test-q6tw2
time="2025-12-22T09:04:47.567Z" level=info msg="Task-result reconciliation" namespace=leak-ns-11 numObjs=0 workflow=leak-test-q6tw2
time="2025-12-22T09:04:47.567Z" level=info msg="Updated phase -> Running" namespace=leak-ns-11 workflow=leak-test-q6tw2
time="2025-12-22T09:04:47.567Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=leak-ns-11 workflow=leak-test-q6tw2
time="2025-12-22T09:04:47.567Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=leak-ns-11 workflow=leak-test-q6tw2
time="2025-12-22T09:04:47.567Z" level=info msg="Pod node leak-test-q6tw2 initialized Pending" namespace=leak-ns-11 workflow=leak-test-q6tw2
time="2025-12-22T09:04:47.567Z" level=warning msg="couldn't get boundaryTemplate through nodeName leak-test-q6tw2" namespace=leak-ns-11 workflow=leak-test-q6tw2
time="2025-12-22T09:04:47.572Z" level=info msg="add pod event" pod=leak-test-q6tw2
time="2025-12-22T09:04:47.572Z" level=info msg="Created pod: leak-test-q6tw2 (leak-test-q6tw2)" namespace=leak-ns-11 workflow=leak-test-q6tw2
time="2025-12-22T09:04:47.572Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-11 workflow=leak-test-q6tw2
time="2025-12-22T09:04:47.572Z" level=info msg=reconcileAgentPod namespace=leak-ns-11 workflow=leak-test-q6tw2
time="2025-12-22T09:04:47.577Z" level=info msg="update pod event" pod=leak-test-q6tw2
time="2025-12-22T09:04:47.578Z" level=info msg="Workflow update successful" namespace=leak-ns-11 phase=Running resourceVersion=2123 workflow=leak-test-q6tw2
time="2025-12-22T09:04:47.578Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=2123 namespace=leak-ns-11 workflow=leak-test-q6tw2
time="2025-12-22T09:04:47.578Z" level=info msg="Task-result reconciliation" namespace=leak-ns-11 numObjs=0 workflow=leak-test-q6tw2
time="2025-12-22T09:04:47.578Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-11 workflow=leak-test-q6tw2
time="2025-12-22T09:04:47.578Z" level=info msg=reconcileAgentPod namespace=leak-ns-11 workflow=leak-test-q6tw2
time="2025-12-22T09:04:47.584Z" level=info msg="Workflow update successful" namespace=leak-ns-11 phase=Running resourceVersion=2126 workflow=leak-test-q6tw2
time="2025-12-22T09:04:47.584Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=2126 namespace=leak-ns-11 workflow=leak-test-q6tw2
time="2025-12-22T09:04:47.584Z" level=info msg="Task-result reconciliation" namespace=leak-ns-11 numObjs=0 workflow=leak-test-q6tw2
time="2025-12-22T09:04:47.584Z" level=info msg="TaskSet Reconciliation" namespace=leak-ns-11 workflow=leak-test-q6tw2
time="2025-12-22T09:04:47.584Z" level=info msg=reconcileAgentPod namespace=leak-ns-11 workflow=leak-test-q6tw2
time="2025-12-22T09:04:47.586Z" level=info msg="update pod event" pod=leak-test-6xscz
time="2025-12-22T09:04:47.588Z" level=info msg="delete pod event" pod=leak-test-6xscz
time="2025-12-22T09:04:47.593Z" level=info msg="update pod event" pod=leak-test-q6tw2
time="2025-12-22T09:04:48.023Z" level=info msg="update pod event" pod=leak-test-flt4t
time="2025-12-22T09:04:48.583Z" level=info msg="update pod event" pod=leak-test-q6tw2
time="2025-12-22T09:04:48.589Z" level=info msg="update pod event" pod=leak-test-tk68p
time="2025-12-22T09:04:48.711Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1954 namespace=leak-ns-7 workflow=leak-test-tk68p
(truncated for length)
Logs from in your workflow's wait container
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
(none, they were deleted after the namespace is deleted)