SRV records for stopped instances don't always get removed

[aaronbuchanan]

We were getting intermittent 502 errors following deployments, upon investigation we found SRV records existed for tasks that had been stopped. Curious if anyone has seen similar issues with this approach? Is there a best practice for keeping the SRV records healthy?

[jogster]

I found the same. This is because the ecssd_agent is listening to events on the docker port to register/deregister the Route53 SRV records.

Are you running the healthcheck lamda? That should be able to detect dead SRV record entries and remove them automatically.

[wprater]

Are you running the healthcheck lamda? That should be able to detect dead SRV record entries and remove them automatically.

Can look into this, but I imagine there will be more lag than if the agent could listen to container events as well.

[aaronbuchanan]

Hi @jogster is this healthcheck lambda configuration defined anywhere? Route 53's health checks don't seem to supports SRV records (or any multi-answer DNS lookups)?