Skip to content

refactor(cron): use gcloud instead of gsutils #3950

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

refactor(cron): use gcloud instead of gsutils #3950

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
14b1c19
Update all references of gsutil to use gcloud storage instead
jess-lowe Sep 12, 2025
fab3915
try gzip in flight
jess-lowe Sep 12, 2025
5eb8667
Update vulnfeeds/cmd/nvd-cve-osv/run_cve_to_osv_generation.sh
jess-lowe Sep 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/api/post-v1-determineversion.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ To confirm if the package you are interested in can be versioned by the determin

1. All available package information can be found [here](https://storage.googleapis.com/osv-indexer-configs).
2. You can look up your specific package using a url in the form <!-- markdown-link-check-disable --> `https://storage.googleapis.com/osv-indexer-configs/generated/{your-package}.yaml` <!-- markdown-link-check-enable--> For example, if you are interested in the library `protobuf`, you can find information for it at [`https://storage.googleapis.com/osv-indexer-configs/generated/protobuf.yaml`](https://storage.googleapis.com/osv-indexer-configs/generated/protobuf.yaml).
3. You can use [gsutil](https://cloud.google.com/storage/docs/gsutil) to copy everything: `gsutil -m cp -r gs://osv-indexer-configs/ .`
3. You can use [gcloud storage](https://cloud.google.com/sdk/gcloud/reference/storage) to copy everything: `gcloud storage cp -r gs://osv-indexer-configs/ .`

## Try the API with our tool

Expand Down
2 changes: 1 addition & 1 deletion docs/data.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ E.g. for PyPI vulnerabilities:

```bash
# Or download over HTTP via https://osv-vulnerabilities.storage.googleapis.com/PyPI/all.zip
gsutil cp gs://osv-vulnerabilities/PyPI/all.zip .
gcloud storage cp gs://osv-vulnerabilities/PyPI/all.zip .
```

### Downloading recent changes
Expand Down
2 changes: 1 addition & 1 deletion gcp/workers/cron/generate_sitemap/generate_and_upload.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@ echo "Begin sitemap generation for $BASE_URL_PATH"

echo "Begin Syncing with cloud to $OUTPUT_BUCKET"

gsutil -m rsync -c -d $SITEMAP_OUTPUT "gs://$OUTPUT_BUCKET/"
gcloud storage rsync $SITEMAP_OUTPUT "gs://$OUTPUT_BUCKET/" --delete-unmatched-destination-objects -c
2 changes: 1 addition & 1 deletion vulnfeeds/cmd/alpine/run_alpine_convert.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ rm -rf $OSV_PARTS_OUTPUT && mkdir -p $OSV_PARTS_OUTPUT

./alpine-osv
echo "Begin Syncing with cloud"
gsutil -q -m rsync -c -d $OSV_PARTS_OUTPUT "gs://$OUTPUT_BUCKET/$OSV_PARTS_OUTPUT"
gcloud storage rsync $OSV_PARTS_OUTPUT "gs://$OUTPUT_BUCKET/$OSV_PARTS_OUTPUT" --delete-unmatched-destination-objects -q -c
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is gone anyway once conflict is resolved.

echo "Successfully synced with cloud"
4 changes: 2 additions & 2 deletions vulnfeeds/cmd/combine-to-osv/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,6 @@ This statically overrides the record generated, meaning any and all of the input

#### Procedure

1. `gsutil cp gs://cve-osv-conversion/osv-output/CVE-YYYY-NNNN.json`
1. `gcloud storage cp gs://cve-osv-conversion/osv-output/CVE-YYYY-NNNN.json`
2. manually edit the file
3. `gsutil cp gs://cve-osv-conversion/osv-output-overrides/CVE-YYYY-NNNN.json`
3. `gcloud storage cp gs://cve-osv-conversion/osv-output-overrides/CVE-YYYY-NNNN.json`
2 changes: 1 addition & 1 deletion vulnfeeds/cmd/combine-to-osv/run_combine_to_osv_convert.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,5 +49,5 @@ echo "Override"
gcloud --no-user-output-enabled storage rsync "gs://${INPUT_BUCKET}/osv-output-overrides/" $OSV_OUTPUT

echo "Begin syncing output to GCS bucket ${OUTPUT_BUCKET}"
gsutil -q -m rsync -c -d "${OSV_OUTPUT}" "gs://${OUTPUT_BUCKET}/osv-output/"
gcloud storage rsync "${OSV_OUTPUT}" "gs://${OUTPUT_BUCKET}/osv-output/" --delete-unmatched-destination-objects -q -c
echo "Successfully synced to GCS bucket"
4 changes: 2 additions & 2 deletions vulnfeeds/cmd/cpe-repo-gen/cpe-repo-gen_map.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ unzip -o "${WORK_DIR}/cpe.zip" -d "${CPE_JSON_DIR}"

MAYBE_USE_DEBIAN_COPYRIGHT_METADATA=""
if [[ -n "${DEBIAN_COPYRIGHT_GCS_PATH}" ]]; then
gsutil ${BE_VERBOSE="-q"} cp "${DEBIAN_COPYRIGHT_GCS_PATH}" "${WORK_DIR}"
gcloud storage cp "${DEBIAN_COPYRIGHT_GCS_PATH}" "${WORK_DIR}" ${BE_VERBOSE="-q"}
tar -C "${WORK_DIR}" -xf "${WORK_DIR}/$(basename ${DEBIAN_COPYRIGHT_GCS_PATH})"
MAYBE_USE_DEBIAN_COPYRIGHT_METADATA="--debian_metadata_path ${WORK_DIR}/metadata.ftp-master.debian.org"
fi
Expand All @@ -49,4 +49,4 @@ fi
--output_dir="${WORK_DIR}"


gsutil ${BE_VERBOSE="-q"} cp "${WORK_DIR}/cpe_product_to_repo.json" "${CPEREPO_GCS_PATH}"
gcloud storage cp "${WORK_DIR}/cpe_product_to_repo.json" "${CPEREPO_GCS_PATH}" ${BE_VERBOSE="-q"}
8 changes: 4 additions & 4 deletions vulnfeeds/cmd/cve-bulk-converter/run-cvelist-converter.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,19 +64,19 @@ find "${LOCAL_OUT_DIR}" -type f -name \*.json \

# Copy (and remove any missing) results to GCS bucket, with some sanity
# checking.
objs_present=$(gsutil ls "${OSV_OUTPUT_GCS_PATH}" | wc -l)
objs_deleted=$(gsutil -m rsync -c -n -d "${LOCAL_OUT_DIR}/gcs_stage" "${OSV_OUTPUT_GCS_PATH}" 2>&1 | grep "Would remove" | wc -l)
objs_present=$(gcloud storage ls "${OSV_OUTPUT_GCS_PATH}" | wc -l)
objs_deleted=$(gcloud storage rsync --dry-run --delete-unmatched-destination-objects -c "${LOCAL_OUT_DIR}/gcs_stage" "${OSV_OUTPUT_GCS_PATH}" 2>&1 | grep "Would remove" | wc -l)

threshold=$(echo "scale=2; ${objs_present} * (${SAFETY_THRESHOLD_PCT:-2} / 100)" | bc)

# # Bash can't deal with floats
if (( $(echo "${objs_deleted} > ${threshold}" | bc -l) )); then
echo "Aborting. Unexpectedly high (${objs_deleted}) number of CVE records would be deleted!" >> /dev/stderr
gsutil -m rsync -c -n -d "${LOCAL_OUT_DIR}/gcs_stage" "${OSV_OUTPUT_GCS_PATH}" 2>&1 | grep "Would remove" >> /dev/stderr
gcloud storage rsync --dry-run --delete-unmatched-destination-objects -c "${LOCAL_OUT_DIR}/gcs_stage" "${OSV_OUTPUT_GCS_PATH}" 2>&1 | grep "Would remove" >> /dev/stderr
exit 1
fi

echo "Copying CVEList records successfully converted to GCS bucket"
gsutil -q -m rsync -c -d "${LOCAL_OUT_DIR}/gcs_stage" "${OSV_OUTPUT_GCS_PATH}"
gcloud storage rsync -q --delete-unmatched-destination-objects -c "${LOCAL_OUT_DIR}/gcs_stage" "${OSV_OUTPUT_GCS_PATH}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since you are already writing this just do this in go

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can probably just use that existing function that does the checksums in metadata and stuff, since that's the most accurate way to specify a modified date.


echo "Conversion run complete"
2 changes: 1 addition & 1 deletion vulnfeeds/cmd/debian-copyright-mirror/debian-copyright-mirror.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,4 +33,4 @@ python debian-copyright-mirror.py "${WORK_DIR}/metadata.ftp-master.debian.org/ch

tar -C "${WORK_DIR}" -cf "${WORK_DIR}/$(basename ${GCS_PATH})" .

gsutil ${BE_VERBOSE="--quiet"} cp "${WORK_DIR}/$(basename ${GCS_PATH})" "${GCS_PATH}"
gcloud storage cp "${WORK_DIR}/$(basename ${GCS_PATH})" "${GCS_PATH}" ${BE_VERBOSE="-q"}
2 changes: 1 addition & 1 deletion vulnfeeds/cmd/debian/run_debian_convert.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ rm -rf $OSV_PARTS_OUTPUT && mkdir -p $OSV_PARTS_OUTPUT

./debian-osv
echo "Begin Syncing with cloud, GCS bucket: ${OUTPUT_BUCKET}"
gsutil -q -m rsync -c -d $OSV_PARTS_OUTPUT "gs://$OUTPUT_BUCKET/$OSV_PARTS_OUTPUT"
gcloud storage rsync $OSV_PARTS_OUTPUT "gs://$OUTPUT_BUCKET/$OSV_PARTS_OUTPUT" --delete-unmatched-destination-objects -q -c
echo "Successfully synced with cloud"
2 changes: 1 addition & 1 deletion vulnfeeds/cmd/download-cves/mirror_nvd.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,4 +35,4 @@ mkdir -p "${WORK_DIR}/nvd"

echo "Copying files to GCS bucket"
gcloud config set storage/parallel_composite_upload_enabled True
gcloud --no-user-output-enabled storage rsync "${WORK_DIR}/nvd/" "gs://${BUCKET}/nvd/" --checksums-only -c --delete-unmatched-destination-objects -q
gcloud --no-user-output-enabled storage rsync "${WORK_DIR}/nvd/" "gs://${BUCKET}/nvd/" --checksums-only -c --delete-unmatched-destination-objects -q -J
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is -J?

Also generally for scripts I think the best practice is to use the full flags to make it easier to read, so it's great that a lot of the gsutil flags are being change to full flags in this PR :D

10 changes: 5 additions & 5 deletions vulnfeeds/cmd/nvd-cve-osv/run_cve_to_osv_generation.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,19 +68,19 @@ done

# Copy (and remove any missing) results to GCS bucket, with some sanity
# checking.
objs_present=$(gsutil ls "${OSV_OUTPUT_GCS_PATH}" | wc -l)
objs_deleted=$(gsutil -m rsync -c -n -d "${WORK_DIR}/nvd2osv/gcs_stage" "${OSV_OUTPUT_GCS_PATH}" 2>&1 | grep "Would remove" | wc -l)
objs_present=$(gcloud storage ls "${OSV_OUTPUT_GCS_PATH}" | wc -l)
objs_deleted=$(gcloud storage rsync --dry-run --delete-unmatched-destination-objects -c "${WORK_DIR}/nvd2osv/gcs_stage" "${OSV_OUTPUT_GCS_PATH}" 2>&1 | grep "Would remove" | wc -l)

threshold=$(echo "scale=2; ${objs_present} * (${SAFETY_THRESHOLD_PCT:-2} / 100)" | bc)

# Bash can't deal with floats
if (( $(echo "${objs_deleted} > ${threshold}" | bc -l) )); then
echo "Warning. Unexpectedly high (${objs_deleted}) number of CVE records would be deleted!" >> /dev/stderr
gsutil -m rsync -c -n -d "${WORK_DIR}/nvd2osv/gcs_stage" "${OSV_OUTPUT_GCS_PATH}" 2>&1 | grep "Would remove" >> /dev/stderr
# TODO: add back in once nvd-mirror issue fixed: exit 1
gcloud storage rsync --dry-run --delete-unmatched-destination-objects -c "${WORK_DIR}/nvd2osv/gcs_stage" "${OSV_OUTPUT_GCS_PATH}" 2>&1 | grep "Would remove" >> /dev/stderr
exit 1
fi

echo "Copying NVD CVE records successfully converted to GCS bucket"
gsutil -q -m rsync -c "${WORK_DIR}/nvd2osv/gcs_stage" "${OSV_OUTPUT_GCS_PATH}"
gcloud storage rsync -q --delete-unmatched-destination-objects -c "${WORK_dIR}/nvd2osv/gcs_stage" "${OSV_OUTPUT_GCS_PATH}"

echo "Conversion run complete"
5 changes: 3 additions & 2 deletions vulnfeeds/pypi/cloudbuild.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,6 @@ steps:
args:
- -c
- bq query --max_rows=10000000 --format=json --nouse_legacy_sql < vulnfeeds/pypi/pypi_versions.sql > pypi_versions.json
- name: gcr.io/cloud-builders/gsutil
args: ['cp', 'pypi_links.json', 'pypi_versions.json', 'gs://pypa-advisory-db/triage/']
- name: 'gcr.io/google.com/cloudsdktool/google-cloud-cli'
entrypoint: 'gcloud'
args: ['storage', 'cp', 'pypi_links.json', 'pypi_versions.json', 'gs://pypa-advisory-db/triage/']
6 changes: 3 additions & 3 deletions vulnfeeds/tools/debian/run_convert_debian.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ poetry run python3 convert_debian.py --adv_type=DTSA -o $OSV_DTSA_OUT $WEBWML_PA
popd

echo "Begin Syncing with cloud"
gsutil -m rsync -c -d $OSV_DSA_OUT gs://$OUTPUT_BUCKET/dsa-osv
gsutil -m rsync -c -d "$OSV_DLA_OUT" "gs://${OUTPUT_BUCKET}/dla-osv"
gsutil -m rsync -c -d "$OSV_DTSA_OUT" "gs://${OUTPUT_BUCKET}/dtsa-osv"
gcloud storage rsync $OSV_DSA_OUT gs://$OUTPUT_BUCKET/dsa-osv --delete-unmatched-destination-objects -c
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep these as gsutil for now

gcloud storage rsync "$OSV_DLA_OUT" "gs://${OUTPUT_BUCKET}/dla-osv" --delete-unmatched-destination-objects -c
gcloud storage rsync "$OSV_DTSA_OUT" "gs://${OUTPUT_BUCKET}/dtsa-osv" --delete-unmatched-destination-objects -c
echo "Successfully synced with cloud"
2 changes: 1 addition & 1 deletion vulnfeeds/tools/debian/run_first_package_finder.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ echo "Finding first packages"
poetry run python3 first_package_finder.py

echo "Syncing with cloud first_package_output ${OUTPUT_BUCKET}"
gsutil -q -m rsync -c -d 'first_package_output' "gs://${OUTPUT_BUCKET}/first_package_output"
gcloud storage rsync 'first_package_output' "gs://${OUTPUT_BUCKET}/first_package_output" --delete-unmatched-destination-objects -q -c
echo "Successfully synced with cloud"

popd
Loading