GCS connector throws rate limit errors #1231
Description
Using spark 3.5.1 on GKE, using connector https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.14, ran into below errors.
24/07/29 23:25:14 INFO c.go.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.util.RetryHttpInitializer: Encountered status code 429 when sending DELETE request to URL 'https://storage.googleapis.com/storage/v1/b/xxx/o/granular%2Frams_video_ad_requests%2F.spark-staging-431ff8ba-d71b-431d-81f4-a4e13178636e%2F_temporary%2F0%2F_temporary%2Fattempt_202407292251203861221280246034281_0004_m_000538_3024%2Fdate_key=2024-06-23%2Fhour=19%2F00242_0_part-00538-431ff8ba-d71b-431d-81f4-a4e13178636e_00242.c000.snappy.parquet?ifGenerationMatch=1722294586557203'. Delegating to response handler for possible retry. [CONTEXT ratelimit_period="10 SECONDS" ] 24/07/29 23:25:24 INFO c.go.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.util.RetryHttpInitializer: Encountered status code 429 when sending DELETE request to URL 'https://storage.googleapis.com/storage/v1/b/xxx/o/granular%2Frams_video_ad_requests%2F.spark-staging-431ff8ba-d71b-431d-81f4-a4e13178636e%2F_temporary%2F0%2F_temporary%2Fattempt_202407292251203861221280246034281_0004_m_000538_3024%2Fdate_key=2024-06-23%2Fhour=20%2F00059_0_part-00538-431ff8ba-d71b-431d-81f4-a4e13178636e_00059.c000.snappy.parquet?ifGenerationMatch=1722294618547766'. Delegating to response handler for possible retry. [CONTEXT ratelimit_period="10 SECONDS [skipped: 9]" ]