Hacker News

points by lrvick 4 days ago | 0 comments

Apple still runs iCloud with Puppet managed unhardened desktop Linux distros last I heard with most devops engineers having plaintext God access to all of it from their individual engineer ssh keys.

They are one of the last companies we should be trusting with hosting infrastructure.

[–] greggsy 4 days ago

Like most bold claims about security, it omits a whole swathe of controls that allow that to happen.

[–] paul_h 4 days ago

Cattle-pets: smart!

[–] lrvick 4 days ago

Any server with a shell and opensshd is a pet, and has no business being near sensitive production workloads.

Add to that aging and wildly insecure ssh-based centralized ruby based infra management solution and I can only conclude everyone invovled has been asleep at the wheel on industry best practices for a over a decade.

Production servers should be immutable once deployed, remotely attestable at all times, with quorum signed secret management that trusts no single admin, with every workload in a dedicated vm with only a kernel, reaper, and target daemon, under an inaccessible offline base OS with TPM gated full disk encryption that fails closed if the system is ever modified after first boot.

And all of that is the bare -minimum-. Unfortunately negligence is industry standard since security engineers that actually know how to do system hardening are crazy rare in the bay.

[–] kccqzy 4 days ago

People love to throw out these buzzwords describing their ideal “maximum security” setup suitable for “sensitive production workloads” without ever defining what the threat model is.

The most grating thing is requiring every workload to be in a dedicated VM. You are throwing so much performance away. What kind of threat model would justify this? Usually there’s no threat model analysis at all.

Banning opensshd is again mostly a fool’s errand. It’s much more productive to design a system that writes the authorized_keys file read by opensshd. For example you can design a system that requires peer approval before your own key gets written to authorized_keys.

[–] lrvick 4 days ago

Any service that has the potential to attract state actors, or attackers with access to frontier AI, MUST have a threat model that trusts no single employee or workstation.

From there the spec I described writes itself. No one can ever have the ability to mutate a system once launched.

A new server spec should be defined in a signed change, it is approved with a signed review, it is compiled deterministically from source and signed by multiple independently controlled systems which both must match, and then given multiple signatures. Then any engineer can deploy those immutable images, other engineers can remotely attest the expected PCRs from the independent signed builds. Finally secrets can be released to the machine by an HSM that validates m-of-n approval signatures.

This is table stakes zero trust deployment and again, the bare minimum.

Humans manually sshing to production machines and running commands is a 90s way of doing sysadmin that is exactly why Binance was just compromised and a thousand orgs before it. Survivors bias is not a reasonable security posture.

Also with even nested virtualization on modern hardware the perf hit is minimal, though distinct servers for each workload is fine too. At -least- sandbox each workload with something like gvisor at a minimum. Linux kernel namespaces are not enough as evidenced by endless exploits.

I practice what I preach here too. I spent the last decade building this way for high risk orgs responsible for hundreds of billions of dollars in value. I also founded https://stagex.tools and co-founded https://distrust.co and https://caution.co to make it easier for others to follow these basic tactics. Much more to come soon on this. All FOSS of course.

We need adoption on these tactics to be as wide spread as TLS if the internet is going to survive an army of AI armed skiddies.

[–] fastily 4 days ago

Citation needed. Would love to hear more about this if verifiably true

[–] lrvick 4 days ago

Apple does not brag about using Linux, and while I will not call anyone out directly because it is a great resource, resumes of current and former Apple DevOps/Security engineers describing what they worked on are quite telling.

All sorts of intel you can get about all sorts of companies that way.

[–] fastily 4 days ago

Yeah but that’s just hearsay right? To reiterate my original point, I’m not disagreeing with you, however I’d love to see some documentation and/or hard evidence of these alleged shoddy security practices at Apple

[–] lrvick 4 days ago

What company publicly documents their shoddy security practices?

That is part of the entire problem in the industry. No standards and no accountability. Those with the purse strings only invest in security that is marketable or protects IP. Anyone that invested in proper remotely attestable provably quorum managed systems is going to be bragging about it and happy to show off the proof to earn trust from their customers. I do.

When no one is bragging about doing these things, it is because they are not doing them, and they just have a bunch of humans running around with god powers to execute any code they want on production systems or in binaries that will end up on your laptop.

To get a clue about their internal practices, lets see what they do when everyone is watching in public.

https://github.com/apple/foundationdb/commits/main/

No signed commits, no signed reviews. Merges done via browser. One Github API token away from being supply chain attacked. Also put one of those recent usernames into "forgot my password" and you can verify they do not mandate hardware 2FA. One compromised email account and you get to make a PR as a rando, then merge it with an Apple account and huzzah, you now have code in a critical repo used internally at Apple. Also releases are not signed or reproducible and are just cut by github actions. Another API key you could use to be even sneakier.

Almost any community maintained Linux distro does way better than this. Inexcusable for an org of their resources.

If this is how poorly Apple handles supply chain integrity in public, why would anyone believe they do any of this internally. I have also talked with multiple current and former Apple employees close to such things off the record but obviously will not out which ones.

[–] fastily 4 days ago

Respectfully, I think you're missing my point here. When someone presents a claim, I hope you understand that the onus is on the claimant to provide evidence of their claim. Extraordinary claims require extraordinary evidence. I could just as easily assert that aliens from Jupiter visited me yesterday afternoon for tea but I can imagine you would like to see some evidence before wholeheartedly agreeing with me. Again, I don't think you're necessarily lying about the "plaintext god-mode access", but I am hoping you'll show us some evidence; "trust me bro" isn't good enough

[–] lrvick 4 days ago

I -did- give you clear public evidence of severe gaps in their supply chain security on at least one heavily relied on Apple software library. Same is true of all their public repos and code.

What better evidence could I possibly provide?

Also see the other commenter in this thread who identified themselves as a former Apple contractor admitting Apple uses Jenkins. -Jenkins-.

[–] fastily 4 days ago

> plaintext God access to all of it from their individual engineer ssh keys

Scroll up, I asked for evidence of this specifically. This is a pretty damning allegation and worthy of further examination given the security implications for many, many users. Frankly, whatever they're doing in unrelated public repositories is concerning in a different context but otherwise thoroughly irrelevant to this claim above

[–] doctorpangloss 4 days ago

i think they run icloud on gcp dude

[–] madeofpalk 4 days ago

“icloud” is not one single piece of infrastructure that runs in one place.

[–] lrvick 4 days ago

Sure for bulk storage, but at least as of a few years ago Apple controls a lot of their own metal for the trusted backbone. I would not be shocked if they are crazy enough to manage GCP with puppet too.

[–] bradknowles 4 days ago

Back when I did a consulting job for Apple Retail Software Engineering, the first part was getting a Jenkins CI/CD server up and running for the people who developed and supported the tools that were used by Apple to communicate with the employees in the stores, provide trading for them, etc….

Think of it as a predecessor to Facebook, plus whatever training platform you like. One of these guys wrote a complete software simulator for the iPod Touch-based device they were using in all the stores to handle point-of-sale transactions, so that he could include that in the training.

Content was created on macOS. Content was consumed on iPhones and iPads. The backend servers ran on Linux. It was my job to get all that compiling on the Jenkins server sitting under my desk, instead of on their laptops. I had to make minor changes to each of those code bases to get that to work.

Their build processes were quite advanced for the times. Thousands and thousands of regression tests. Thousands of UI conformance tests.

Then I had to get that Jenkins server up and running on the in-house VMware based cloud they had, where they sucked up their entire last two runs of rackmount xserve hardware and turned it around for their own internal use.

I was given a VM and remote access, and then that was that. My job was done. My contract was over, and I moved back to Austin. But that was a few years ago.

[–] lrvick 4 days ago

Jenkins. Wow. I thought Puppet was the worst of it.

Thank you for having the balls to admit this in public.

Hopefully others can stop putting Apple on a pedestal as some gold standard for high accountability and rigorous security.

[–] bradknowles 3 days ago

You know, Puppet isn’t a massively bad solution, if you’re managing your own hardware. I’ve done Puppet before. IMO, Chef is better, but Puppet still works reasonably well.

And for that matter, if you’re running your own CI/CD solution, Jenkins isn’t that bad, either.

Also mind that this work was done about a decade ago, and Puppet and Jenkins were newer and shinier tools back then.

Now, if you can operate in the cloud, then I can point you at solutions that I think are better — depending one which cloud you’re in, and what tools are natively available there. But on-premise hasn’t changed all that much in a decade.