Actions created by bazel's --runs_per_test are treated as new

[GabrielCFigueira]

Actions created by bazel's --run_per_test flag inject new environment variables such as TEST_RANDOM_SEED and TEST_RUN_NUMBER, which cause remote execution systems such as Buildbarn to interpret them as new actions. In the case of Buildbarn, this prevents it from using size classes correctly, as the tests are treated as new and any prior learning on the execution of these tests is not considered.

Not sure what the solution might be, but is it possible to have some key that identifies actions without considering this env variables that bazel injects?

[peterebden]

This would be nice. We ended up doing something similar in Please so multiple concurrent test runs weren't combined - there's no way of requesting that through the API - the closest thing is skip_cache_lookup, but that still explicitly allows combining multiple in-flight executions.

Conversely though it might be tricky for a client to know whether the server is going to honour this or not. It'd really need to be an advertised capability so the client could decide whether it needs to poison them with env vars or not.

[EdSchouten]

Peter, I think you’re thinking of another kind of feature.

What Gabriel is requesting is that clients send some kind of fingerprint alongside the action (e.g., as part of ExecuteRequest) that tells the scheduler: “this action is similar to […] from a resource utilization perspective”.

There is no need to announce a feature like this as part of the capabilities. It could just be a new field in ExecuteRequest (e.g., a “bytes” field) that a scheduler may decide to use or ignore.

[sluongng]

Not sure what the solution might be, but is it possible to have some key that identifies actions without considering this env variables that bazel injects?

I don't think this issue is as relevant for other RBE clients, such as Buck2 or Siso right? So it would be best if you could raise this on the Bazel / BuildBarn side first to discuss potential solutions. Though I guess raising this in the remote-apis repository suggests that you did have a solution in mind. 😉

Once the client/server implementations agreed on the possible solution, if a change in remote-apis is needed, then we would be able to review and process. It would be weird if we agreed on an API change without such a consensus, because subsequent changes on the Bazel side would face the rejection risk.

---

What Gabriel is requesting is that clients send some kind of fingerprint alongside the action (e.g., as part of ExecuteRequest) that tells the scheduler: “this action is similar to […] from a resource utilization perspective”.

On top of my head, the RequestMetadata should already include sufficient information to group similar actions?

What else is required?

[EdSchouten]

What Gabriel is requesting is that clients send some kind of fingerprint alongside the action (e.g., as part of ExecuteRequest) that tells the scheduler: “this action is similar to […] from a resource utilization perspective”.

On top of my head, the RequestMetadata should already include sufficient information to group similar actions?

I'm not sure about that. Sure, there's action_id. But nowhere is it explicitly stated that clients should ensure it is set to the same value if they expect that instances of such actions share the same resource usage characteristics.

[sluongng]

I was thinking of a combination of action mnemonic + target id. You can also add configuration id into that mix if you want to separate actions based on different platforms.

if they expect that instances of such actions share the same resource usage characteristics.

Is this a reasonable expectation for the client to make?

[EdSchouten]

That’s clearly not sufficient. Many actions belonging to the same target can have the same mnemonic. For example, for tests, the shard index should be taken into account.

[peterebden]

Peter, I think you’re thinking of another kind of feature.

What Gabriel is requesting is that clients send some kind of fingerprint alongside the action (e.g., as part of ExecuteRequest) that tells the scheduler: “this action is similar to […] from a resource utilization perspective”.

There is no need to announce a feature like this as part of the capabilities. It could just be a new field in ExecuteRequest (e.g., a “bytes” field) that a scheduler may decide to use or ignore.

It's similar - I'm thinking of a different way of solving it, where the client no longer needs to explicitly poison the env vars any more in order to separate the executions. In that case I'd assume Buildbarn's size classes would be able to use previous learnings.

[sluongng]

That’s clearly not sufficient. Many actions belonging to the same target can have the same mnemonic. For example, for tests, the shard index should be taken into account.

You can use the primary output of the action, which should include runs and shards information

[EdSchouten]

Exactly. And that will turn it into a Bazel specific solution.

@GabrielCFigueira filed this issue, because he (and other Buildbarn users) would like it if REv2 added a generic solution for this. One where Buildbarn doesn't need to apply heuristics, but Bazel tells Buildbarn what it's sending over.

[sluongng]

Now that we have ironed out the issue, I do not object to adding this information to the spec.

I still want to point out that if folks want this to be implemented in Bazel, they should talk with the Bazel team first to avoid any downstream rejection.

If any non-Bazel clients are interested in implementing this, it would be nice to know as well.