What’s your worst ops blunder?

Mine is the time I took down an entire AWS account by miss-spelling an environment variable 😳

Orrrr the time I accidentally deleted all the code from self hosted GitLab right before end of day

#infrastructure #devops

@rachel oh god 😭

I hope everything was backed up ok!

Edit, seeing the dates: …not entirely sure why mastodon just notified me of this

@ellie not mine but a co-worker was going to delete a folder on traffic node for a customer, and issued the command rm -rf <foldername>*

Except his copy/paste had grabbed a space at the end of the folder name…….

@ellie For some reason, I thought I needed access to prod DB from my dev machine (basically me being lazy and not doing the debugging and reproduction properly)

So the DB_URL env var, pointed to the live database with payment details of over a million users in it.

Guess what happened when an hour later, I ran the tests in that terminal? Yeah, those with the before() function that called db_reset().

(We had backups. And failover. We're down for less than a minute. Still...)

@berkes oh ouchhhh I bet that one stung 😭😭😭

@ellie i deleted a full prod database because someone didn't enable backups and did not name it and I thought it's a leftover database.

That was a nice post mortem.

@ellie

The scene: AWS console, EC2 instances tab.
Me: clicks "select all", and then "terminate instances"
Me, a half second later: "oh SHIT it didn't save my filters, I just terminated every EC2 instance in the account."

Fortunately it was just our dev account and we could recreate everything with terraform but.... Still.

#devops #horrorstories

@ellie many many years ago, running a large Solaris prod box for the first time, after Linux. ‘killall’ command behaviour was different.