Flip incompatible_strict_action_env

[hofbi]

Work towards bazel-contrib/SIG-rules-authors#42.

Fixes #7026

[hofbi]

I guess now I need someone from the bazel team to trigger https://github.com/bazelbuild/continuous-integration/blob/337638d2fcff6d86b66035901c09d37c9199b38a/buildkite/bazelci.py#L99 so that we can see all downstream projects.

[fmeum]

I started a downstream run at https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4824

[fmeum]

I also sent a PR to add the flag to the BCR pipeline: bazelbuild/bazel-central-registry#5319

[hofbi]

I started a downstream run at https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4824

Thanks @fmeum. Do we actually have a baseline for the downstream job? I was looking at a few of the failures showing up and a significant number fails because of Error in rule: rule() got unexpected keyword argument 'incompatible_use_toolchain_transition'. Is this something that is expected to be fixed with #14127?

[fmeum]

I don't see any failures that I would attribute to the flip, but it's true that downstream is in bad shape. That makes the signal worse.

Ideally we would be able to run a downstream test with Bazel 8 and the flag flipped instead, which should avoid most of the preexisting failures.

@fweikert Is that possible?

[hofbi]

@fweikert ping

[meteorcloudy]

I re-opened #6648 and added migration-ready label, we'll see the test result with BCR modules in https://buildkite.com/bazel/bcr-bazel-compatibility-test tomorrow.

But based on historical experience, this one is very hard to flip as many actions do depend on the value of PATH and some other env var.

[hofbi]

Thanks. Let's see what we get tomorrow.

But based on historical experience, this one is very hard to flip as many actions do depend on the value of PATH and some other env var.

Depending how many fixed we would have to do, how about setting --incompatible_strict_action_env=false in projects that heavily depend on it for starters? This would allow us to at list flip the flag here and give projects more time to work this off in their own speed.

[meteorcloudy]

Depending how many fixed we would have to do, how about setting --incompatible_strict_action_env=false in projects that heavily depend on it for starters?

Yeah, this would work for leaf projects, but if a common dependency actually needs this flag, we'll still need to fix it and migrate the ecosystem first. Let's see the result, I started a manual run at https://buildkite.com/bazel/bcr-bazel-compatibility-test/builds/492

[meteorcloudy]

Looks like only rules_python might be affected on Windows, can you please take a quick look? If it's not serious, I'm fine with moving this forward and fix new breakages as we discover them.

[hofbi]

Created bazel-contrib/rules_python#3160 to reproduce this on a PR

[hofbi]

@meteorcloudy I tried to reproduce the windows error in bazel-contrib/rules_python#3160, but on rules_python CI this job passes. See https://buildkite.com/bazel/rules-python-python/builds/12739/steps/canvas?sid=0198f0e1-0e1a-4f05-8f7e-2769e21a7640#0198f0e1-0f47-4ee1-9940-61d1a48af7db/278-292 where the flag is set on the windows CI.

Do you have any idea why the windows build of https://buildkite.com/bazel/bcr-bazel-compatibility-test/builds/492 could have failed? Are they running on different windows environments?

[meteorcloudy]

I think the only difference is that in the rules_python pipeline, it's running again Bazel 7.6.1, however in the BCR pipeline, we are testing against 8.3.1. Can you try upgrade Bazel and see if it's reproducible?