Periodically enforce state on running servers with ansible pull-like logic

[msheiny]

Feature request

Description

We use a TON of ansible to enforce state at run-time but do not have any smart logic to maintain system state over the life of the system. Right okay.. we do have a few examples where we have pushed changes out via debian post-install hooks but this is really frail. Until we have something like #2966 or a magical FPF Leprechaun that travels to each instance to tuck it in each night -- we need a better solution for the time being. So WHAT do We DO!??

🎉 INTRODUCING 🎉 ANSIBLE playbooks in a debian package ✨

Straight from the factory of RedHat comes the latest in system state management! Follow me into my lab! ....... Come on!!

The Gist

The package would look like this from a high-level (we've have a separate package for the app and monitor server):

• Package up a pip wheel of ansible requirements from our requirements files. SEE securedrop-app-code for similar strategy. Install those into a virtualenv. Include ansible logic we usually run on ansible-admin THAT doesnt rely on secrets on the adjacent server AND doesnt require a reboot.
• Lay out appropriate paxflags
• As part of postinstall hook, run ansible playbook logic now
• Put in a cron job to run playbook daily
• Sit back on the beach and reminisce about all the past troubles we've had maintaining remote SD instances melt away. Pina Colada anyone?

Caveats to be careful of ?

• We dont want to trigger new ossec alerts during run-time. This is important since we already have a lot of alert fatigue.
• We'll have to watch carefully vet ansible logic to avoid another spookyinstall situation. Best to stay away from community maintained modules.
• Every push of new ansible logic should be made as part of a release... we need to fully QA whenever we make changes.

User Stories

As a securedrop maintainer, I want to have more control over long running SD system state for stability + security.

As an ansible playbook, I want to be run daily and not just at install time.

[redshiftzero]

Thanks for filing @msheiny - this definitely seems like a promising approach.

We'll have to watch carefully vet ansible logic to avoid another spookyinstall situation. Best to stay away from community maintained modules.

For the interested observer, "spookyinstall" refers to the security vulnerability in #2472. I certainly agree that we want to carefully vet everything (Ansible or otherwise) that will run on the servers, but in terms of whether it's better to use our own roles of community maintained ones, there are some good points in #2360 in favor of using solid community maintained roles.

[msheiny]

but in terms of whether it's better to use our own roles of community maintained ones, there are some good points in #2360 in favor of using solid community maintained roles.

Whoopss , apologies for any confusion here. To clarify, when i said community maintained modules, i am referring to ansible modules themselves (pieces of python code that will actually do the leg work on the server in question). This is different than a role, which contains the playbook logic (written in yaml) which is what #2360 refers to. I have no comment either way on community role inclusion here :)

We arent currently using an community maintained modules in our codebase (at least in the install logic , we have some custom code in CI). It's possible for a community role to embed a module... but thats a whole different can of worms.

[msheiny]

Note to self here .. remove libpam-google-authenticator from installed packages.

[eloquence]

This is very much worthy of consideration after the Focal migration, and in the context of larger discussions about the future of the server OS and provisioning story (e.g., #5517).