Today I got to tell my students that in the bad old days we used to write scripts that just SSHed into every machine and ran sed on config files but today we had puppet and I regret to inform you that based on their facial expressions we apparently still write scripts that just SSH into every machine and run sed on config files

@mjg59 people love to hate on Puppet, but I have seen it used to do remarkable things across a fleet larger than any the haters have managed

@fivetonsflax I could enable security config on over 100,000 systems with reasonable confidence it would work and even 20 years ago I would not have thought that was a good idea with SSH

@mjg59 after having used puppet pretty heavily for a couple years, I think I'd prefer ssh+shell scripts. (I mostly don't use ssh+scripts, I mostly use config packages distributed via apt for stuff that's consistent across all devices of a type and a custom config service for the stuff that isn't. Puppet was just too heavy and fragile for our use case. Also complicated. Our field ops folks had a lot of fights with it, because they didn't understand what it controlled and how to change it.)

@swelljoe I think this is an argument against the idea that DevOps is a way to get rid of sysadmins because you still very much need people who understand this shit works

@swelljoe @mjg59 i prefer taco bell over puppets http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html

@mjg59 I can't tell if by "SSH into every machine and run sed on config files" you're making jokes of ansible or if it's meant literally. Can you please clarify? 😂

@liskin @mjg59 boy this opens old wounds, doesn't it

@liskin @mjg59 were you on board already when i've been implementing 2-factor auth. ended up with a puppet resource that sed in a totp module into pam configuration that was conditional on production environment.
sadly the precondition regexp was botched and the resource kept adding a line each time the recipe run.
combined with linux_pam's BASIC-y conditional jumps (with targets being essentially a line offset) this has been slowly shifting pam_permit.so into the most awkward spot. only on prod.

@lkundrak @mjg59 Yep, remember that. :-)

@lkundrak @liskin @mjg59 I believe I can clarify for Matthew: yes.

@mjg59 For my latest project, I wrote a bunch of shell scripts that SSH into the remote hosts and run shell scripts and sed on configuration files. Feels good. Liberating.

Don't get me wrong, I still love ansible. But there's just something... Je ne sais quoi....

@otakup0pe @mjg59 ansible -m command -a "sed -i..."

It's just like a shell script that SSHs into the remote hosts and runs sed on configuration files, but with free parallelism!

@otakup0pe @mjg59 ansible is just shell scripts written in yaml.

@otakup0pe @mjg59 isn't ansible a glorified version of those shell scripts ?

And yes, I'm using ansible on a daily bases.

@nicolas17 @otakup0pe @mjg59 sometimes you want to limit the parallelism. As a wise SRE once told me, "computers are machines for making mistakes very fast".

@evana @otakup0pe @mjg59 There's several other reasons why that command in my previous post is a terrible idea and you shouldn't do it ;)

@nicolas17 @otakup0pe @mjg59
With cssh you don't even need sed

@mjg59 if the scripts are idempotent and get tested, isn't that okay?

@malin does it deal correctly with machine-specific config? What happens if a machine is down when you run it? What happens if someone manually reverts the change?