Commit ff82a88
authored
[CLD-559]: test(job): fix race condition (#330)
Recently, we been [getting race
condition](https://github.com/smartcontractkit/chainlink-deployments-framework/actions/runs/17254351292/job/48963163120?pr=328#step:2:491)
on the mocked JobClient when ProposeJob is called.
From the stack trace, we know there is a read ProposeJob from log:
```
github.com/stretchr/testify/mock.(*Mock).Called()
/home/runner/go/pkg/mod/github.com/stretchr/[email protected]/mock/mock.go:481 +0x191
github.com/smartcontractkit/chainlink-deployments-framework/offchain/jd/internal/mocks.(*MockJobServiceClient).ProposeJob()
/home/runner/work/chainlink-deployments-framework/chainlink-deployments-framework/offchain/jd/internal/mocks/mock_job_service_client.go:546 +0x34c
```
and a write at
```
Previous write at 0x00c000188060 by goroutine 15:
sync/atomic.StoreInt64()
/opt/hostedtoolcache/go/1.24.4/x64/src/runtime/race_amd64.s:237 +0xb
sync/atomic.StorePointer()
/opt/hostedtoolcache/go/1.24.4/x64/src/runtime/atomic_pointer.go:92 +0x44
github.com/smartcontractkit/chainlink-protos/job-distributor/v1/job.(*ProposeJobRequest).ProtoReflect()
/home/runner/go/pkg/mod/github.com/smartcontractkit/chainlink-protos/[email protected]/v1/job/job.pb.go:791 +0x93
```
We know it is the test `TestJDClient_ProposeJob` from the logs below
```
Goroutine 15 (running) created at:
testing.(*T).Run()
/opt/hostedtoolcache/go/1.24.4/x64/src/testing/testing.go:1851 +0x8f2
github.com/smartcontractkit/chainlink-deployments-framework/offchain/jd.TestJDClient_ProposeJob()
/home/runner/work/chainlink-deployments-framework/chainlink-deployments-framework/offchain/jd/client_test.go:308 +0xae4
```
After some investigation, we use the same pointer for
`jobv1.ProposeJobRequest` on all the test cases for
`TestJDClient_ProposeJob`.
```
--- FAIL: TestJDClient_ProposeJob (0.00s)
--- FAIL: TestJDClient_ProposeJob/success_with_empty_request (0.00s)
testing.go:1490: race detected during execution of test
--- FAIL: TestJDClient_ProposeJob/success_with_valid_proposal (0.00s)
testing.go:1490: race detected during execution of test
--- FAIL: TestJDClient_ProposeJob/error_when_ProposeJob_service_call_fails (0.00s)
testing.go:1490: race detected during execution of test
```
How it happen:
```
// Shared ProposeJobRequest object
request := &jobv1.ProposeJobRequest{...}
// Goroutine 15 (Test A)
mockJob.EXPECT().ProposeJob(ctx, request) // Triggers:
└─ Arguments.Diff()
└─ fmt.Sprintf("%v", request)
└─ request.String()
└─ request.ProtoReflect()
└─ atomic.StorePointer(&metadata) // WRITE!
// Goroutine 16 (Test B) - happening simultaneously
mockJob.EXPECT().ProposeJob(ctx, request) // Triggers:
└─ Arguments comparison
└─ reflect.DeepEqual()
└─ reflect.Value.IsNil()
└─ reads the same memory location // READ!
```
Looks like for mocked object, when using `Arguments.Diff()` for
comparing arguments called and expected, it will call `String()` of the
object, which for protobuf object, it eventually calls ProtoReflect
which performs a write
[here](https://github.com/smartcontractkit/chainlink-protos/blob/main/job-distributor/v1/job/job.pb.go#L174),
this may be the reason for the race condition.
Not super confident, but lets see if it happens again.
JIRA: https://smartcontract-it.atlassian.net/browse/CLD-5591 parent 240a44f commit ff82a88
1 file changed
+6
-14
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
198 | 198 | | |
199 | 199 | | |
200 | 200 | | |
201 | | - | |
202 | | - | |
203 | | - | |
204 | | - | |
205 | | - | |
206 | | - | |
207 | | - | |
208 | 201 | | |
209 | 202 | | |
210 | | - | |
| 203 | + | |
211 | 204 | | |
212 | 205 | | |
213 | 206 | | |
214 | 207 | | |
215 | 208 | | |
216 | 209 | | |
217 | | - | |
| 210 | + | |
218 | 211 | | |
219 | 212 | | |
220 | 213 | | |
| |||
244 | 237 | | |
245 | 238 | | |
246 | 239 | | |
247 | | - | |
| 240 | + | |
248 | 241 | | |
249 | 242 | | |
250 | 243 | | |
| |||
260 | 253 | | |
261 | 254 | | |
262 | 255 | | |
263 | | - | |
| 256 | + | |
264 | 257 | | |
265 | 258 | | |
266 | 259 | | |
| |||
280 | 273 | | |
281 | 274 | | |
282 | 275 | | |
283 | | - | |
| 276 | + | |
284 | 277 | | |
285 | 278 | | |
286 | | - | |
287 | 279 | | |
288 | 280 | | |
289 | 281 | | |
| |||
310 | 302 | | |
311 | 303 | | |
312 | 304 | | |
313 | | - | |
| 305 | + | |
314 | 306 | | |
315 | 307 | | |
316 | 308 | | |
| |||
0 commit comments