ECS image prune remove old images memory issue.

[rafaelcarmilema]

Hello
Im running into memory issues while using bottlerocket in ecs. I wonder if there is an equivalent to remove old docker images like
ECS_NUM_IMAGES_DELETE_PER_CYCLE

so far I checked docs and was able to find settings.ecs.task-cleanup-wait and settings.ecs.reserved-memory. Any ideas on how to achieve this ?

Thanks

[yeazelm]

If you set settings.ecs.task-cleanup-wait this will change ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION from the default of 3 hours down to as low as 1 minute. Nonetheless, we don't currently expose ECS_NUM_IMAGES_DELETE_PER_CYCLE and only support task-cleanup-wait. Adjusting this setting could help, but the default of 5 per 30 minutes might not be enough by default. Do you know if your workload would need this number increased? If so, we could add a feature request to add this setting into Bottlerocket.

[rafaelcarmilema]

The workload is for our dev environment. We do multiple deployments one after the another to test features. When testing the bottlerocket ami, we noticed memory issues due to old images and old task still there filling up everything.
We tried the following in user-data toml:
[settings.ecs]
cluster = "${ecs_cluster}"
task-cleanup-wait = "1h"
reserved-memory = 256
The memory still filled out after multiple deploys, (we can adjust the task cleanup to less than 1h but still old images are there).
Can you elaborate on "5 per 30 mins"?
Will it be possible to add ECS_NUM_IMAGES_DELETE_PER_CYCLE as well ?

[yeazelm]

Can you elaborate on "5 per 30 mins"?

The defaults from https://docs.aws.amazon.com/AmazonECS/latest/developerguide/automated_image_cleanup.html call out that every 30 minutes, at most 5 images will be removed from the defaults. So launching faster than that could end up in the state you are talking about.

Will it be possible to add ECS_NUM_IMAGES_DELETE_PER_CYCLE as well?

I cut #3167 to track adding more settings.