Move to Intelligent Tiering for nix cache or how to save $4k per month with this one trick

[arianvp]

The Amazon S3 Intelligent-Tiering storage class automatically stores objects in three access tiers. One tier is optimized for frequent access, one lower-cost tier is optimized for infrequent access, and another very low-cost tier is optimized for rarely accessed data. For a low monthly object monitoring and automation charge, S3 Intelligent-Tiering monitors access patterns and automatically moves objects to the Infrequent Access tier when they haven't been accessed for 30 consecutive days. After 90 days of no access, the objects are moved to the Archive Instant Access tier without performance impact or operational overhead.

INTELLIGENT_TIERING charges a management fee of $0.0025 per 1000 objects https://aws.amazon.com/s3/pricing/?nc=sn&loc=4 and only applies to files bigger than 128KB.

By default, AWS doesn't move things betweens tiers that are smaller than 128KB

We have 53,819,565 objects in STANDARD_IA with an average size of 2.4MiB We have 970,978,934 objects in STANDARD with an average size of 122KiB

From this we can deduce that enabling intelligent tiering will cost:

53,819,565 * $0.0025 / 1,000 = $ 134.54 per month

But for that monthly management fee AWS will automatically move objects to Infrequent Access after 30 days and Archive Infrequent Access after 90 days of no access. If an object is accessed, it's moved back to the standard storage class. So we aren't punished with high retrieval fees if we move something into archive that later becomes a hot path again.

Actually Intelligent Tiering doesn't charge a retrieval fee at all. whilst STANDARD_IA does. We spent $87.16 on retrieval fees last month.

This means that enabling intelligent tiering will cost us $134.54 - $87.16 = $47.38 per month

Intelligent Tiering Infrequent Access is charged the same as STANDARD_IA at $0.0125 per GB Archive Infrequent Access is charged at $0.004 per GB (Same as GLACIER_INSTANT_RETRIEVAL)

Currently we have 545.7TB in STANDARD_IA which costs 545.7 * 1000 * $0.0125 = $6821.25 per month

If most of our storage is going to end up in Archive Infrequent Access we could pay as low as

545.7 * 1000 * $0.004 = $2182.80 per month

So we have a potential savings of a few thousand dollars per month by enabling this

[arianvp]

We can maybe enable this only for files with prefix nar/a and after 366 days to smoke test this a bit and then roll it out for all objects

[arianvp]

So we aren't punished with high retrieval fees if we move something into archive that later becomes a hot path again.
This means that enabling intelligent tiering will cost us $134.54 - $87.16 = $47.38 per month

This might not be entirely accurate and an oversimplification. As intelligent tiering will move it back to standard access and keep it there for at least 30 days. So the storage cost of a moved object might be a bit higher. e.g. if someone decides to fetch NixOS images from 21.05 those files will be moved back into standard access and then it takes 30 days before they're transitioned back into infrequent access.

But still. The amount of money we might save on storage cost is most definitely worth it

[infinisil]

Randomly stumbling upon this, this is great!

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2025-s3-sponsorship-more-resources-for-a-sustainable-nix/67019/1

[arianvp]

I really want to pick this up again. But was very busy