A partial solution to this evil-maid attack vector is Heads firmware (a replacement for the bios/uefi itself), which lets you sign the contents of your unencrypted boot partition using a gpg key on a hardware token, and verify the integrity of the firmware itself using a totp/hotp key stored in the tpm.
All the benefits of secure boot, but you get to control the signing keys yourself instead of relying on a vendor. It’s great stuff.
Everything I run, I deploy and manage with ansible.
When I’m building out the role/playbook for a new service, I make sure to build in any special upgrade tasks it might have and tag them. When it’s time to run infrastructure-wide updates, I can run my single upgrade playbook and pull in the upgrade tasks for everything everywhere - new packages, container images, git releases, and all the service restart steps to load them.
It’s more work at the beginning to set the role/playbook up properly, but it makes maintaining everything so much nicer (which I think is vital to keep it all fun and manageable).
A friend of mine was once the organist at a cathedral with a grand pipe organ. He invited me to see it one day and hear him play, and for the finale he had me climb up into the forest of towering pedal pipes, crouching between the rows, dwarfed by their looming height, while he played Bach’s Toccata and Fugue in D Minor.
The sound hit me like a wave, so vast and tremendous and perfect. I felt utterly annihilated - tiny and shaken apart into nothing, a speck swept away in a cascading ocean of music, like the whole world was exploding in cataclysm and fractal rebirth all around me. Dazzling and enormous.
And when the fugue began, I think that’s the closest to nirvana I’ve ever been. Just blown clean off the face of the earth.