[–] lrvick 4 days ago
[–] kccqzy 4 days ago
The most grating thing is requiring every workload to be in a dedicated VM. You are throwing so much performance away. What kind of threat model would justify this? Usually there’s no threat model analysis at all.
Banning opensshd is again mostly a fool’s errand. It’s much more productive to design a system that writes the authorized_keys file read by opensshd. For example you can design a system that requires peer approval before your own key gets written to authorized_keys.
[–] lrvick 4 days ago
From there the spec I described writes itself. No one can ever have the ability to mutate a system once launched.
A new server spec should be defined in a signed change, it is approved with a signed review, it is compiled deterministically from source and signed by multiple independently controlled systems which both must match, and then given multiple signatures. Then any engineer can deploy those immutable images, other engineers can remotely attest the expected PCRs from the independent signed builds. Finally secrets can be released to the machine by an HSM that validates m-of-n approval signatures.
This is table stakes zero trust deployment and again, the bare minimum.
Humans manually sshing to production machines and running commands is a 90s way of doing sysadmin that is exactly why Binance was just compromised and a thousand orgs before it. Survivors bias is not a reasonable security posture.
Also with even nested virtualization on modern hardware the perf hit is minimal, though distinct servers for each workload is fine too. At -least- sandbox each workload with something like gvisor at a minimum. Linux kernel namespaces are not enough as evidenced by endless exploits.
I practice what I preach here too. I spent the last decade building this way for high risk orgs responsible for hundreds of billions of dollars in value. I also founded https://stagex.tools and co-founded https://distrust.co and https://caution.co to make it easier for others to follow these basic tactics. Much more to come soon on this. All FOSS of course.
We need adoption on these tactics to be as wide spread as TLS if the internet is going to survive an army of AI armed skiddies.
Add to that aging and wildly insecure ssh-based centralized ruby based infra management solution and I can only conclude everyone invovled has been asleep at the wheel on industry best practices for a over a decade.
Production servers should be immutable once deployed, remotely attestable at all times, with quorum signed secret management that trusts no single admin, with every workload in a dedicated vm with only a kernel, reaper, and target daemon, under an inaccessible offline base OS with TPM gated full disk encryption that fails closed if the system is ever modified after first boot.
And all of that is the bare -minimum-. Unfortunately negligence is industry standard since security engineers that actually know how to do system hardening are crazy rare in the bay.