Commit 8024082
authored
Script to get sizes of all s3 buckets (#6484)
This code was used when recently analyzing our S3 buckets to see which
ones were the largest and may have room for optimization (resulting in
us setting a new lifecycle policy:
meta-pytorch/pytorch-gha-infra#626)
What it does:
* It gets the size of each bucket using CloudWatch, which is much faster
than checking directly in S3. The latter iterates over every item in the
bucket and can take hours to complete.
* It sorts the output to show the largest buckets.
Checking in the code in case we want to use it again
Output looks like this (dummy data below):
```
> python bucket_size_metrics.py
🔍 Fetching S3 bucket list...
✅ Found 7 buckets. Fetching storage metrics...
(1/7): Bucket marketing-assets
💿 Storage used: 100.0 GB
(2/7): Bucket devops-tools
💿 Storage used: 50.0 GB
(3/7): Bucket software-releases
💿 Storage used: 200.0 GB
(4/7): Bucket analytics-reports
🫥 No metrics found for bucket: analytics-reports. Is it unused?
(5/7): Bucket product-images
💿 Storage used: 10.0 GB
(6/7): Bucket company-videos
💿 Storage used: 500.0 GB
(7/7): Bucket sales-documents
💿 Storage used: 20.0 GB
📌 Storage Usage Summary:
Size (GB) Bucket Name
----------------------------
500.00 company-videos
200.00 software-releases
100.00 marketing-assets
50.00 devops-tools
20.00 sales-documents
10.00 product-images
0.00 analytics-reports
```1 parent e2efc38 commit 8024082
1 file changed
+110
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
0 commit comments