Fallback behavior when IAM is down

[zxlin]

I'm sure many people noticed a very brief IAM outage earlier this week. During the outage, IAM was not responsive and as a result, this script would go and delete all of the local users synced from IAM as IAM did not return a list of users.

I was hoping to discuss what are the options for some fallback behavior in the event of IAM outage or actually just plain network connectivity outage.

[michaelwittig]

if the IAM API is down the CLI will fail. As a result, the script will fail. It should not delete all users.
Why do you think that this will happen?

[zxlin]

It happened with the IAM outage earlier this week, in /var/log/auth.log it shows aws-ec2-ssh deleting all the users during the outage. Perhaps IAM returned 200 and an empty list while it was still recovering?

Nonetheless, this is still a case we have to consider.

[michaelwittig]

that's kind of unexpected.... the problem is that we can not really decide that an empty list means no IAM users or something is broken...

[zxlin]

It would be kind of annoying to implement, but a configurable n number of "confirms" that a user is deleted before actually deleting the user would be nice.

Keep some state of the list of users and a counter of how many times a user's been missing from the returned list, on the n-th sync of it missing, delete the user.

[michaelwittig]

I don't see how this would solve the problem. Depending on the length of the outage the script would still delete all the users?

[richard-scott]

Curl has lots of exit codes so you can see exactly why the download of data failed and act accordingly.

[zxlin]

@michaelwittig n can be set to a org's fault tolerance preference and if the outage does last longer than however long the n confirm allows, then it at least gives time for the org to react to system failures.

This is not the only solution to fix it, it's just what I thought of first. I'm completely open to other ways to introduce some fault tolerance into this.

[michaelwittig]

@richard-scott there is no curl involved here at all...

[assertnotnull]

I saw this issue and maybe it's the problem I am facing here.
Today in the same afternoon, the import_user deleted the users twice.

[michaelwittig]

@assertnotnull we log to system log (priority auth.info | tag aws-ec2-ssh) when users are created or deleted by ec2-ssh. Depending on the configuration of your OS, the logs will be placed in different log files. Usually it is /var/log/messages. Could you provide the relevant log lines?

[assertnotnull]

Here's what I found in the logs
https://defuse.ca/b/ToCgVcUDGNTQj4gwv0YhS3

[michaelwittig]

@assertnotnull ok. and on what time was the user added again after Nov 1 20:40:20?

[MassimoSporchia]

Same is happening to me during an outage on IAM: https://twitter.com/gbcis/status/993502731762655232
Even having the users during an outage like this means that users won't be able to log-in your system, is there a way to cache public-keys?

[packetfairy]

After having this code shoot us in the face on three discrete events, here are the changes I made to better protect myself:

• removed the -r from the userdel instruction, so that home directories would be left in place during an outage which resulted in users being deleted
• added extra bailouts to import_users, because, at least for our environment, YES WE CAN conclude that an empty user list means that something is horribly horribly broken
• added generation of authorized_keys files to import_users and stripped the sshd configurations back out