Move to Intelligent Tiering for nix cache

INTELLIGENT_TIERING charges a management fee of $0.0025 per 1000 objects
https://aws.amazon.com/s3/pricing/?nc=sn&loc=4 and only applies to files
bigger than 128KB.

By default, AWS doesn't move things betweens tiers that are smaller than
128KB

We have 53,819,565 objects in STANDARD_IA with an average size of 2.4MiB
We have 970,978,934 objects in STANDARD with an average size of 122KiB

From this we can deduce that enabling intelligent tiering will cost:

53,819,565 * $0.0025 / 1,000 = $ 134.54

But for that monthly management fee AWS will automatically move objects
to Infrequent Access after 30 days and Archive Infrequent Access after 90 days
of no access.  If an object is accessed, it's moved back to the standard
storage class. So we aren't punished if we move something into archive
that later becomes a hot path again.

Intelligent Tiering also doesn't charge a retrieval fee. whilst
STANDARD_IA does. We spent $87.16 on retrieval fees last month.

This means that enabling intelligent tiering will cost us
$134.54 - $87.16 = $47.38 per month

Infrequent Access is charged the same as STANDARD_IA at $0.0125 per GB
Archive Infrequent Access is charged at $0.004 per GB

Currently we have 545.7TB in STANDARD_IA which costs
545.7 * 1000 * $0.0125 = $6821.25 per month

If most of our storage is going to end up in Archive Infrequent Access
we could pay as low as

545.7 * 1000 * $0.004 = $2182.80 per month

So we have a potential savings of $4638.45 by enabling this

Author: arianvp

terraform/cache.tf

@@ -22,8 +22,11 @@ resource "aws_s3_bucket_lifecycle_configuration" "cache" {
     }
 
     transition {
+      # TODO: Set this to 0 as we rely on intelligent tiering to move between
+      # standard and infrequent access after 30  days.
+      # but I want to test that this is working first.
       days          = 365
-      storage_class = "STANDARD_IA"
+      storage_class = "INTELLIGENT_TIERING"
     }
   }
 }