Cost Export backfill

[wozzer72]

Like the Carbon Export, we require Cost data (FOCUS) to be backfilled from a given start date: YYYY-MM-DD (parameter).

The back dated data should pass through the existing queue used to process ongoing data, thus reusing existing solution to process the data and export to AWS.

[wozzer72]

The terraform code already includes the creation of backfill as "cost management export onetime runs " - one for each month from a 2022-01 until a hardcoded month in 2025:

So inactive. You can run the runs manually with "Export Selected Dates":

The jobs take a good five minutes to run. The job did post on the target queue which invoked the lambda and wrote the backfill data to S3.

The fact that you have to enter the dates to run manually, suggests you can enter any date. Have proven this by entering "01/03/2025 - 31/03/2025" against the export job created for "2025-06".

Checked both the exports to S3 (parquet) and they have data and in the same format as most recent export. The function logs resulting from the manual exports look equivalent too in respect of processing.

And need to be sure we can safely trigger exports for dates where there is no data (even before the account was created say). But it appears from the manual run, there is a "one year limitation" (the following resulted on 22nd Dec 2025):

And finally. Rather than relying on a manual trigger post deploying (and certainly, having to manually run with start/end date is laborious, need to be able to trigger the backfill automatically from code, and only once.

Summary:

• The logic for creating backfill exports assumes a hardcoded end date
• There is no reason of necessity for creating a backfill of exports, when to run them, you have to give an arbitrary start and end date; so create a single export that can be triggered multiple times.
• Need to trigger backfill, and trigger only once
  • Yet still handle failure and retry
• The export range on the export schedule is only for one year
• That said, there are examples on the appvia S3 bucket that suggest you can export beyond the one year limit

[wozzer72]

Documented a solution and reviewed it with Mike. Mike highlighted we can avoid the "one year" time range when using "exported selected dates" for a given report job, by simply selecting one of the generated backfill scheduled reports and selecting "run":

Have updated the proposed solution:

1. Fix the terraform module and introduce backfill start and end date as a new mandatory required module properties (rather than hard coded). Looking to automate the backfill dates is harder than it looks, so simply make then required.
2. A new "cost export backfill" http event in the function - this takes backfill start date and will iterate each month, first checking if exported object does not already exist, forces a run on the export job. The iteration is until last month only, as the daily cost export covers the current month every day. Will report as error if the backfill report job does not exist, highlighting a need to update the backfill end date in terraform and redeploy to create the backfioll job.
3. A new "backfill" daily (Mon to Fri) timer event in the function, that calls the new cost export backfill" function event and the existing carbon export backfill event, having first checked if there does not exists a complementary backfill lock object on the S3 bucket. Why the bucket? Because it is only the analytics team in AWS who see any data generated (the source Azure engineers). And if the code had been updated and the backfill need to be recreated for any reason, the appvia AWS analytics team is best placed to then delete the lock object and wait for the next daily timer.

[wozzer72]

I've been able to experiment much more with the Cost Management terraform and restful API, and have come to the conclusion that:

1. the function (python) - cost export backfill will create the monthly Cost Management Export jobs - for which custom jobs can only have a single month period.
2. Create the job if it doesn't exist.
3. Clean up old jobs on the next timer event.
4. Always create new jobs based on the current timestamp, just in case appvia analytics team wants to recreate backfill in which case, it is up to the month prior to the current month (not the month the deployment was done).

Having the cost export in code as with the carbon export is more consistent - and removes deploy time restrictions.

Need also to extend the schedule "to" to more than just five years from deployment - it's a time bomb waiting to go of and cause heinous debugging.