Skip to content

Commit e4b194c

Browse files
authored
add Intelligent tiering
1 parent 987b250 commit e4b194c

File tree

1 file changed

+3
-3
lines changed

1 file changed

+3
-3
lines changed

data-exports/README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -75,13 +75,13 @@ Cross-account access is possible but can be difficult to maintain, considering t
7575
Yes. Throughout an organization's lifecycle, mergers and acquisitions may occur, so this approach prepares you for potential future scenarios.
7676

7777
### Can I use S3 Intelligent Tiering or S3 Infrequent Access (IA) for my CUR data connected to Athena?
78-
We strongly recommend **against** using S3 IA or other storage tiers for CUR data that is connected to Athena, especially if you have active FinOps users querying this data. Here's why:
78+
We strongly recommend **against** using S3 IA for CUR data that is connected to Athena, especially if you have active FinOps users querying this data. Here's why:
7979
- CUDOS typically only retrieves data for the last 7 months, so theoretically older data could be moved to S3 IA or managed with Intelligent Tiering.
8080
- Moving older CUR parquet files to IA could potentially reduce storage costs by up to 45%.
8181
- **However**, this only saves money if the data isn't frequently accessed. With S3 IA, you're charged $0.01 per GB retrieved.
8282
- Athena uses multiple computational nodes in parallel, and complex queries can multiply data reads dramatically. For every 1GB of data you want to scan, Athena might perform up to 75GB of S3 reads.
8383
- If someone runs a query without properly limiting it to specific billing periods, the retrieval costs can be astronomical. For example:
8484
* Scanning a full CUR of 600GB: `600GB × 75 × $0.01/GB` = `$450.00` for just one query!
8585
- Due to this risk of human error, we do not use storage tiering as a default and strongly advise against it for CUR data connected to Athena.
86-
87-
If you still want to optimize storage costs, consider replicating your CUR to another bucket and configuring a lifecycle policy to delete older data from the main bucket connected to Athena.
86+
We also advise agains Intelligent Tiering by default.
87+
- KPI Dashboard - one of our foundational dashboards - scans the entire CUR (Cost and Usage Report) data to detect the first snapshot and determine its age. This prevents AWS Intelligent Tiering from functioning effectively, as it forces all data to remain in frequent access tiers and result is unnecessary additional monitoring costs with no cost-saving benefits.

0 commit comments

Comments
 (0)