[Bazel CI] Failed to copy inputs to sandbox permission denied error.

[sgowroji]

CI: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4542#0195ac95-7f90-454f-ab3f-dd894217575d

Platform: MacOS

Logs:

(05:08:04) ERROR: /Users/buildkite/builds/bk-macos-arm64-q1go/bazel-org-repo-root/bazel-skylib/tests/copy_directory/BUILD.bazel:15:15: Copying directory tests/copy_directory/dir_with_subdir failed: Could not copy inputs into sandbox: /private/var/tmp/_bazel_buildkite/0f7a0c0e5068cbecc82869727306c10a/sandbox/darwin-sandbox/12/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/tests/copy_directory/empty_dir_copy -> /private/var/tmp/_bazel_buildkite/0f7a0c0e5068cbecc82869727306c10a/sandbox/_moved_trash_dir/10 (Permission denied)

Culprit:

Steps:

git clone https://github.com/bazelbuild/bazel-skylib
git reset 25a8e9d60687fee887e54dcb9560ef08777235de  --hard
export USE_BAZEL_VERSION=69726de6a42f2795071c10514e36a2439a7d44de
bazel build --remote_download_toplevel --incompatible_config_setting_private_default_visibility --incompatible_disallow_empty_glob --noenable_workspace  //... @external_directory_tests//... @bazel_skylib_gazelle_plugin//... 

CC Greenteam @Wyverald

[sgowroji]

@meteorcloudy @Wyverald Could you please confirm whether this is an Infra issue ?

[Wyverald]

This happens quite often on CI (intermittently...), though I have no idea how to reproduce it locally. @tjgq for a look -- maybe this is indicating that something is actually wrong with the skylib impl of copy_directory?

[tjgq]

I think @fmeum also ran into this earlier.

The error message says "copying inputs into sandbox", but the failed operation tries to move a file from the sandbox into the trash. So this looks more like a failure to clear the previous sandbox contents (likely callsite: https://cs.opensource.google/bazel/bazel/+/master:src/main/java/com/google/devtools/build/lib/sandbox/SandboxHelpers.java;l=422;drc=8587f7442e9bf6f63910986d8e07c390df73bee2 or https://cs.opensource.google/bazel/bazel/+/master:src/main/java/com/google/devtools/build/lib/sandbox/SandboxHelpers.java;l=438;drc=8587f7442e9bf6f63910986d8e07c390df73bee2)

In order for the move to fail with access denied, one of the two parent directories must be non-writable (or not have search permission, I guess). But the destination directory should be writable (it's controlled by the AsynchronousTreeDeleter) and I don't see evidence that the rules in question are messing with the permissions of the origin directory.

I suppose we could have AsynchronousTreeDeleter retry after fixing permissions, but I'd like to have a theory for what's happening before blindly attempting to fix it.

[sgowroji]

@Wyverald We are still seeing this failure in the Downstream https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4707#01971f5f-5ad5-4158-b780-c6c89ecf3ddf.

[satyanandak]

Team, Given that the current bug has caused our Downstream CI pipeline to fail for over two months, we're temporarily moving it to a disabled pipeline. It will be reinstated in the Downstream CI once it successfully passes its tests in the disabled pipeline.