You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
## Fix degradation of Workflow runtime performance over time
10
14
@@ -26,23 +30,48 @@ This caused Jobs to fail, and enter failure policy retry loops.
26
30
27
31
Refactor the Scheduler connection pool logic to properly prune stale connections to prevent job execution occurring on stale connections and causing failure policy loops.
28
32
29
-
## Allow Service Account for MetalBear mirrord operator in sidecar injector
33
+
## Fix remote Actor invocation 500 retry
30
34
31
35
### Problem
32
36
33
-
Mirrord Operator is not on the allow list of Service Accounts for the dapr sidecar injector.
37
+
An actor invocation across hosts which result in a 500 HTTP header response code would result in the request being retried 5 times.
34
38
35
39
### Impact
36
40
37
-
Running mirrord in `copy_target` mode would cause the pod to initalise with without the dapr container.
41
+
Services which return a 500 HTTP header response code would result in requests under normal operation to return slowly, and request the service on the same request multiple times.
38
42
39
43
### Root cause
40
44
41
-
Mirrord Operator is not on the allow list of Service Accounts for the dapr sidecar injector.
45
+
The Actor engine considered a 500 HTTP header response code to be a retriable error, rather than a successful request which returned a non-200 status code.
42
46
43
47
### Solution
44
48
45
-
Add the Mirrord Operator into the allow list of Service Accounts for the dapr sidecar injector.
49
+
Remove the 500 HTTP header response code from the list of retriable errors.
50
+
51
+
### Problem
52
+
53
+
## Fix Global Actors Enabled Configuration
54
+
55
+
### Problem
56
+
57
+
When `global.actors.enabled` was set to `false` via Helm or the environment variable `ACTORS_ENABLED=false`, the Dapr sidecar would still attempt to connect to the placement service, causing readiness probe failures and repeatedly logged errors about failing to connect to placement.
58
+
Fixes this [issue](https://github.com/dapr/dapr/issues/8551).
59
+
60
+
### Impact
61
+
62
+
Dapr sidecars would fail their readiness probes and log errors like:
63
+
```
64
+
Failed to connect to placement dns:///dapr-placement-server.dapr-system.svc.cluster.local:50005: failed to create placement client: rpc error: code = Unavailable desc = last resolver error: produced zero addresses
65
+
```
66
+
67
+
### Root cause
68
+
69
+
The sidecar injector was not properly respecting the global actors enabled configuration when setting up the placement service connection.
70
+
71
+
### Solution
72
+
73
+
The sidecar injector now properly respects the `global.actors.enabled` helm configuration and `ACTORS_ENABLED` environment variable. When set to `false`, it will not attempt to connect to the placement service, allowing the sidecar to start successfully without actor functionality.
74
+
46
75
47
76
## Prevent panic of reminder operations on slow Actor Startup
48
77
@@ -61,3 +90,57 @@ The Dapr runtime would attempt to use the reminder service before it was initial
61
90
### Solution
62
91
63
92
Correctly return an errors that the actor runtime was not ready in time for the reminder operation.
93
+
94
+
## Remove client-side rate limiter from Sentry
95
+
96
+
### Problem
97
+
98
+
A cold start of many Dapr deployments would take a long time, and even cause some crash loops.
99
+
100
+
### Impact
101
+
102
+
A large Dapr deployment would take a non-linear more amount of time that a smaller one to completely roll out.
103
+
104
+
### Root cause
105
+
106
+
The Sentry Kubernetes client was configured with a rate limiter which would be exhausted when services all new Dapr deployment at once, cause many client to wait significantly.
107
+
108
+
### Solution
109
+
110
+
Remove the client-side rate limiting from the Sentry Kubernetes client.
111
+
112
+
## Allow Service Account for MetalBear mirrord operator in sidecar injector
113
+
114
+
### Problem
115
+
116
+
Mirrord Operator is not on the allow list of Service Accounts for the dapr sidecar injector.
117
+
118
+
### Impact
119
+
120
+
Running mirrord in `copy_target` mode would cause the pod to initalise without the dapr container.
121
+
122
+
### Root cause
123
+
124
+
Mirrord Operator is not on the allow list of Service Accounts for the dapr sidecar injector.
125
+
126
+
### Solution
127
+
128
+
Add the Mirrord Operator into the allow list of Service Accounts for the dapr sidecar injector.
129
+
130
+
## Fix Scheduler Client connection pruning
131
+
132
+
### Problem
133
+
134
+
Daprd would attempt to connect to stale Scheduler addresses.
135
+
136
+
### Impact
137
+
138
+
Network resource usage and error reporting from service mesh sidecars.
139
+
140
+
### Root cause
141
+
142
+
Daprd would not close Scheduler gRPC connections to hosts which no longer exist.
143
+
144
+
### Solution
145
+
146
+
Daprd now closes connections to Scheduler hosts when they are no longer in the list of active hosts.
// Set addresses for actor services only if it's not explicitly globally disabled
84
84
// Even if actors are disabled, however, the placement-host-address flag will still be included if explicitly set in the annotation dapr.io/placement-host-address
85
85
// So, if the annotation is already set, we accept that and also use placement for actors services
0 commit comments