Refactor: gsutil to gcloud storage Migration with Per-Command Feature Flags (#4930)

# Refactor: `gsutil` to `gcloud storage` Migration with Per-Command
Feature Flags

## Summary

This pull request fundamentally refactors our `gsutil` to `gcloud
storage` migration strategy. The previous approach used a single global
feature flag, `USE_GCLOUD_STORAGE`, which activated the migration for
all commands simultaneously. This PR replaces that global flag with a
granular, per-command flag system, allowing for a safer, more
controlled, and incremental transition.

The following environment variables have been introduced to individually
control their corresponding commands:

  * `USE_GCLOUD_STORAGE_CP`
  * `USE_GCLOUD_STORAGE_RSYNC`
  * `USE_GCLOUD_STORAGE_LS`
  * `USE_GCLOUD_STORAGE_MB`
  * `USE_GCLOUD_STORAGE_CORS_SET`
  * `USE_GCLOUD_STORAGE_DEFSTORAGECLASS_SET`
  * `USE_GCLOUD_STORAGE_IAM_CH`

-----

## Motivation

The migration from `gsutil` to the more modern `gcloud storage` tools is
an important step to modernize our codebase and leverage the performance
and feature improvements of `gcloud`. However, a "big bang" migration
with a single global flag posed a significant risk. A bug or performance
issue in a single command could impact all storage operations, causing
widespread instability.

This new approach mitigates that risk by allowing us to enable and test
the migration for each command in isolation. This makes it easier to
identify issues, roll back specific changes, and ensure a smooth, stable
transition to the new tooling.

-----

## Implementation Details

The main technical changes include:

  * **Dynamic Flag Function:**

      * The global `use_gcloud_storage()` function has been removed.
* A new function, `use_gcloud_for_command(command: str)`, has been
introduced in `src/clusterfuzz/_internal/google_cloud_utils/gsutil.py`.
This function dynamically constructs the environment variable name
(e.g., `USE_GCLOUD_STORAGE_CP`) based on the provided command and checks
if it is set.

  * **`GSUtilRunner` Refactoring:**

* The `GSUtilRunner` in `gsutil.py` no longer relies on an internal
state (`self.use_gcloud_storage`).
* Its methods (`rsync`, `download_file`, etc.) now invoke
`use_gcloud_for_command` with the relevant command to decide at runtime
which CLI (`gsutil` or `gcloud storage`) to use.

  * **Direct `gsutil` Call Updates:**

* Scripts that invoked `gsutil` directly, such as
`src/local/butler/common.py` and
`src/clusterfuzz/_internal/scripts/copy_corpus.py`, have been refactored
to use the new feature flag logic.
* Shell scripts in `configs/` and `docker/` have also been updated to
check the new command-specific environment variables.

  * **User Interface:**

* The upload instruction message in the `corpora` handler
(`src/appengine/handlers/corpora.py`) now dynamically displays the
`gsutil rsync` or `gcloud storage rsync` command, depending on the state
of the `USE_GCLOUD_STORAGE_RSYNC` flag, ensuring users see the correct
command.

-----

## Rollout Strategy and How to Use

Activating the new functionality is done via environment variables. By
default, all commands will continue to use `gsutil`.

To enable `gcloud storage` for a specific command, set the corresponding
environment variable to a truthy value (e.g., `"1"` or `"True"`).

**Example:** To test the `cp` command using `gcloud storage`, set the
following environment variable:

```bash
export USE_GCLOUD_STORAGE_CP=1
```

To revert to `gsutil`, simply unset the environment variable or set it
to an empty string.

### Recommended Rollout Strategy:

1. **Staging Environment**: Start by enabling the flags one by one in a
staging or testing environment.
2. **Monitoring**: After enabling each flag, closely monitor logs,
performance metrics, and related functionality to ensure there are no
regressions.
3. **Incremental Rollout**: We suggest the following order for a
production rollout, starting with lower-risk commands:
      * `ls`
      * `mb`, `cors set`, `defstorageclass set`
      * `cp`
      * `rsync`
4. **Production**: Once a command has been thoroughly tested and
validated in staging, the corresponding flag can be enabled in
production.
5. **Migration Completion**: The ultimate goal is to have all flags
enabled in production. After a stabilization period, we can plan a
future refactoring to completely remove the legacy `gsutil` code and the
feature flags themselves.

-----

## Tests

* All existing unit tests in `gsutil_test.py` and `deploy_test.py` have
been updated to use the new per-command feature flags.
* Tests have been parameterized to ensure each command functions
correctly with both `gsutil` (flag disabled) and `gcloud storage` (flag
enabled).
* All tests, linters, and formatters are passing, ensuring code quality
and consistency.

Author: hunsche

.github/workflows/tests.yaml

@@ -23,9 +23,8 @@ jobs:
 
     steps:
       - uses: actions/checkout@v3
-      - run: |  # Needed for git diff to work.
-          git fetch origin master --depth 1
-          git symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/master
+        with:
+          fetch-depth: 0
 
       - name: Setup python environment
         uses: actions/setup-python@b55428b1882923874294fa556849718a1d7f2ca5

AGENTS.md

@@ -65,6 +65,14 @@ python butler.py py_unittest -t core -m
 
 The `-m` flag runs the tests in parallel, which is recommended.
 
+### Running a single test file
+
+To run a single test file, you can use the `-p` or `--pattern` flag. For example, to run the tests in `deploy_test.py`, you can use the following command:
+
+```bash
+python butler.py py_unittest -t core -p deploy_test.py
+```
+
 ## Linting
 
 To check the code for style and linting issues, run the following command:

Pipfile

@@ -9,6 +9,7 @@ crcmod = "==1.7"
 future = "==0.17.1"
 protobuf = "==4.23.4"
 psutil = "==5.9.4"
+google-cloud-ndb = "==2.3.4"
 
 [dev-packages]
 Fabric = "==1.14.1"

Pipfile.lock

@@ -24,6 +24,7 @@ steps:
     # the repository to grab the .git directory.
     git clone 'https://github.com/google/clusterfuzz' tmp
     git -C tmp fetch origin "$COMMIT_SHA"
+    git -C tmp fetch origin master
     git -C tmp checkout -qf FETCH_HEAD
     mv tmp/.git .git
     rm -rf tmp

cloudbuild.yaml

@@ -74,20 +74,26 @@ fi
 echo "Activating credentials with the Google Cloud SDK."
 $GSUTIL_PATH/gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS
 
-# Otherwise, gsutil will error out due to multiple types of configured
-# credentials. For more information about this, see
-# https://cloud.google.com/storage/docs/gsutil/commands/config#configuration-file-selection-procedure
-echo "Specifying the proper Boto configuration file."
-BOTO_CONFIG_PATH=$($GSUTIL_PATH/gsutil -D 2>&1 | grep "config_file_list" | egrep -o "/[^']+gserviceaccount\.com/\.boto") || true
-if [ -f $BOTO_CONFIG_PATH ]; then
-  export BOTO_CONFIG="$BOTO_CONFIG_PATH"
-else
-  echo "WARNING: failed to identify the Boto configuration file and specify BOTO_CONFIG env."
-fi
-
 echo "Downloading ClusterFuzz source code."
 rm -rf clusterfuzz
-$GSUTIL_PATH/gsutil cp gs://$DEPLOYMENT_BUCKET/$DEPLOYMENT_ZIP clusterfuzz-source.zip
+if [ "$USE_GCLOUD_STORAGE_CP" = "1" ]; then
+  echo "Downloading ClusterFuzz source code using gcloud."
+  $GSUTIL_PATH/gcloud storage cp gs://$DEPLOYMENT_BUCKET/$DEPLOYMENT_ZIP clusterfuzz-source.zip
+else
+  # Otherwise, gsutil will error out due to multiple types of configured
+  # credentials. For more information about this, see
+  # https://cloud.google.com/storage/docs/gsutil/commands/config#configuration-file-selection-procedure
+  echo "Specifying the proper Boto configuration file."
+  BOTO_CONFIG_PATH=$($GSUTIL_PATH/gsutil -D 2>&1 | grep "config_file_list" | egrep -o "/[^']+gserviceaccount\.com/\.boto") || true
+  if [ -f $BOTO_CONFIG_PATH ]; then
+    export BOTO_CONFIG="$BOTO_CONFIG_PATH"
+  else
+    echo "WARNING: failed to identify the Boto configuration file and specify BOTO_CONFIG env."
+  fi
+
+  echo "Downloading ClusterFuzz source code using gsutil."
+  $GSUTIL_PATH/gsutil cp gs://$DEPLOYMENT_BUCKET/$DEPLOYMENT_ZIP clusterfuzz-source.zip
+fi
 unzip -q clusterfuzz-source.zip
 
 echo "Installing ClusterFuzz package dependencies using pipenv."

configs/test/bot/setup/android.bash

@@ -69,20 +69,26 @@ fi
 echo "Activating credentials with the Google Cloud SDK."
 $GSUTIL_PATH/gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS
 
-# Otherwise, gsutil will error out due to multiple types of configured
-# credentials. For more information about this, see
-# https://cloud.google.com/storage/docs/gsutil/commands/config#configuration-file-selection-procedure
-echo "Specifying the proper Boto configuration file."
-BOTO_CONFIG_PATH=$($GSUTIL_PATH/gsutil -D 2>&1 | grep "config_file_list" | egrep -o "/[^']+gserviceaccount\.com/\.boto") || true
-if [ -f $BOTO_CONFIG_PATH ]; then
-  export BOTO_CONFIG="$BOTO_CONFIG_PATH"
-else
-  echo "WARNING: failed to identify the Boto configuration file and specify BOTO_CONFIG env."
-fi
-
 echo "Downloading ClusterFuzz source code."
 rm -rf clusterfuzz
-$GSUTIL_PATH/gsutil cp gs://$DEPLOYMENT_BUCKET/$DEPLOYMENT_ZIP clusterfuzz-source.zip
+if [ "$USE_GCLOUD_STORAGE_CP" = "1" ]; then
+  echo "Downloading ClusterFuzz source code using gcloud."
+  $GSUTIL_PATH/gcloud storage cp gs://$DEPLOYMENT_BUCKET/$DEPLOYMENT_ZIP clusterfuzz-source.zip
+else
+  # Otherwise, gsutil will error out due to multiple types of configured
+  # credentials. For more information about this, see
+  # https://cloud.google.com/storage/docs/gsutil/commands/config#configuration-file-selection-procedure
+  echo "Specifying the proper Boto configuration file."
+  BOTO_CONFIG_PATH=$($GSUTIL_PATH/gsutil -D 2>&1 | grep "config_file_list" | egrep -o "/[^']+gserviceaccount\.com/\.boto") || true
+  if [ -f $BOTO_CONFIG_PATH ]; then
+    export BOTO_CONFIG="$BOTO_CONFIG_PATH"
+  else
+    echo "WARNING: failed to identify the Boto configuration file and specify BOTO_CONFIG env."
+  fi
+
+  echo "Downloading ClusterFuzz source code using gsutil."
+  $GSUTIL_PATH/gsutil cp gs://$DEPLOYMENT_BUCKET/$DEPLOYMENT_ZIP clusterfuzz-source.zip
+fi
 unzip -q clusterfuzz-source.zip
 
 echo "Installing ClusterFuzz package dependencies using pipenv."

configs/test/bot/setup/linux.bash

@@ -65,17 +65,23 @@ echo "Specifying the proper Boto configuration file."
 # Otherwise, gsutil will error out due to multiple types of configured
 # credentials. For more information about this, see
 # https://cloud.google.com/storage/docs/gsutil/commands/config#configuration-file-selection-procedure
-BOTO_CONFIG_PATH=$($GSUTIL_PATH/gsutil -D 2>&1 | grep "config_file_list" | egrep -o "/[^']+gserviceaccount\.com/\.boto")
-
-if [ -f $BOTO_CONFIG_PATH ]; then
-  export BOTO_CONFIG="$BOTO_CONFIG_PATH"
-else
-  echo "WARNING: failed to identify the Boto configuration file and specify BOTO_CONFIG env."
-fi
-
 echo "Downloading ClusterFuzz source code."
 rm -rf clusterfuzz
-$GSUTIL_PATH/gsutil cp gs://$DEPLOYMENT_BUCKET/$DEPLOYMENT_ZIP clusterfuzz-source.zip
+if [ "$USE_GCLOUD_STORAGE_CP" = "1" ]; then
+  echo "Downloading ClusterFuzz source code using gcloud."
+  $GSUTIL_PATH/gcloud storage cp gs://$DEPLOYMENT_BUCKET/$DEPLOYMENT_ZIP clusterfuzz-source.zip
+else
+  BOTO_CONFIG_PATH=$($GSUTIL_PATH/gsutil -D 2>&1 | grep "config_file_list" | egrep -o "/[^']+gserviceaccount\.com/\.boto")
+
+  if [ -f $BOTO_CONFIG_PATH ]; then
+    export BOTO_CONFIG="$BOTO_CONFIG_PATH"
+  else
+    echo "WARNING: failed to identify the Boto configuration file and specify BOTO_CONFIG env."
+  fi
+
+  echo "Downloading ClusterFuzz source code using gsutil."
+  $GSUTIL_PATH/gsutil cp gs://$DEPLOYMENT_BUCKET/$DEPLOYMENT_ZIP clusterfuzz-source.zip
+fi
 unzip -q clusterfuzz-source.zip
 
 echo "Installing ClusterFuzz package dependencies using pipenv."

configs/test/bot/setup/mac.bash

@@ -92,7 +92,11 @@ cd $INSTALL_DIRECTORY
 
 echo "Downloading ClusterFuzz source code."
 rm -rf $ROOT_DIR
-$GSUTIL_PATH/gsutil cp gs://$DEPLOYMENT_BUCKET/linux.zip clusterfuzz-source.zip
+if [ "$USE_GCLOUD_STORAGE_CP" = "1" ]; then
+  $GSUTIL_PATH/gcloud storage cp gs://$DEPLOYMENT_BUCKET/linux.zip clusterfuzz-source.zip
+else
+  $GSUTIL_PATH/gsutil cp gs://$DEPLOYMENT_BUCKET/linux.zip clusterfuzz-source.zip
+fi
 unzip -q clusterfuzz-source.zip
 
 echo "Setting up android."

configs/test/gce/android-init.bash

@@ -270,9 +270,14 @@ rm "C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\sym" -Recurse -Force
 $deploymentBucket = $webClient.DownloadString('http://metadata.google.internal/computeMetadata/v1/project/attributes/deployment-bucket')
 
 # Download ClusterFuzz source.
+Write-Host "Downloading ClusterFuzz source code."
 rm c:\clusterfuzz -Recurse -Force
 $fileName = "$tmp\clusterfuzz.zip"
-gsutil cp gs://$deploymentBucket/windows-3.zip $fileName
+if ($env:USE_GCLOUD_STORAGE_CP -eq "1") {
+  gcloud storage cp gs://$deploymentBucket/windows-3.zip $fileName
+} else {
+  gsutil cp gs://$deploymentBucket/windows-3.zip $fileName
+}
 unzip $fileName
 
 # Resize partition to max available size.