Hacker News
Firecracker VM Escape: PwnEscape Part 2(pwn.ai)
281 points by SerCe 1 day ago | 133 comments
[–] scoodah 1 day ago
I really wish they would have expanded on this paragraph in their article further:

> The configuration used here had no jailer. Firecracker's production containment includes dropping privileges and restricting the process through namespaces and seccomp. A deployment using those controls requires a separate assessment of the privileges and resources available after compromising its VMM.

If I'm understanding all of this correctly, a properly jailed and configured firecracker instance would greatly reduce the blast radius of this escape, and I feel as though it would have been good for them to explain what jailing and dropping privileges would do to help. I'd hope most production firecracker deployments would be doing this already.

Not saying that this is any less severe and the root issue is not specific to Firecracker, but to a flaw in the kernel's network code -- they just used an insecure firecracker configuration to demonstrate it. However, I feel like some are taking this as a flaw with firecracker when its not. This is an issue in the kernel networking stack.

[–] tptacek 1 day ago
That's a good catch. Yes, since they've only proven this up to the jailer boundary, you'd need a second and relatively powerful LPE vulnerability to get real host control out of this.
[–] rvz 1 day ago
Thanks to frontier reasoning agents, Linux is completely riddled in ways you can discover 900+ LPE vulnerabilities. [0]

Another wave of vulnerabilities like this are yet to be discovered in Linux. We are just seeing how weak C and C++ based OSes and kernels are when put under hundreds of AI agents.

[0] https://openai.com/index/a-thousand-fewer-paths-to-root/

[–] imnotsuretheykn 20 hours ago
Watch Hartman’s response to the first wave of issues found by AI. Super dismissive and ignorant.

I’m not sure the people that caused this will be the one fixing it.

[–] fock 14 hours ago
you sure the people who caused this (speeding up Linux to mainly serve ads instead of hardening it?) aren't the exact same people now pushing the antidote?
[–] rockskon 1 day ago
And how many of these vulns are in code for legacy hardware or filesystem support that in no way reflects threats against modern Linux installs? I wager the overwhelming majority of them.
[–] wahern 1 day ago
The recent OpenAI release linked above targeted only ~3M SLoC (out of ~40M), core systems code reachable from within a simple VM guest.
[–] devmor 1 day ago
We always knew this, it has been understood since nearly the dawn of the monokernel OS that there is always a route to privilege escalation somewhere. This is why even for web applications that would be better served by a shell, people choose CGI-style interfaces instead.

The only reason it wasn't seen as a problem was due to the effort and time it took to find and exploit them.

[–] amluto 1 day ago
Nope. The exploit compromises the host kernel. The researchers built an unconventional exploit chain in which they used the host kernel compromise to compromise the host firecracker userspace code, but I see no fundamental reason they couldn't have compromised something else in the host kernel.
[–] tptacek 1 day ago
Wait, this is also presumably true, come to think of it. I'm a little fuzzy because they document the memory disclosure primitive carefully but not the memory writes they also got from it --- but whatever those are, they're hitting kernel memory? But the exploit they're describing here gets code execution out of Firecracker itself, not the kernel.
[–] amluto 1 day ago
AIUI they’ve chosen as their target the host userspace pt_regs, i.e. the kernel structure that stores Firecracker’s register state. If they can find it and write to it, they can control the complete register state of Firecracker.

I assume they chose it as their target because Firecracker, like most Linux KVM userspace [0], maps guest memory into its own address space, giving an easy place to write gadgets.

My personal favorite thing to poke into pt_regs is an actual privileged (CPL0) context. [1] Returning from the kernel right back to a controlled kernel context is allowed by the CPU :)

[0] Until fairly recently this was mandatory. These days it’s possible to ask KVM to map something into the guest without having it in host userspace memory.

[1] Corrupt pt_regs so it executes user memory in kernel mode:

https://seclists.org/oss-sec/2015/q3/291

I think this may be my favorite of all exploits I’ve ever written. In fact, I often made a point of not writing interesting exploits for corruption bugs I found, but this one was too much fun to pass up.

[–] tptacek 11 hours ago
Thanks! This makes sense. It also, of course, makes the jailer irrelevant; just find a non-jailed process to inject into.
[–] monocasa 1 day ago
Yeah, they're not saying a lot there. And then the proof of concept code is just being passed the physical addresses of the pages they talk about as command line args, and is just opening /dev/mem to do brain surgery on them.

Which like, isn't really a proof of concept of anything being described.

"That was always allowed".

[–] tptacek 1 day ago
It's a super good bug! I just don't follow the exploitation details all the way. I know more than one cloud provider panic-bounced their fleet in response to it, and I think that was the right call.
[–] monocasa 1 day ago
There's for sure a real bug.

It just bothers me that the proof of concept starts with the assumption that host physical memory is mapped into guest physical memory at 2^39. If you can't talk about the actual exploit yet, I get it , but don't write a blog post with something like that called a "proof of concept" when the code shows no exploit, and you're obviously dancing around the actual novel exploit here in the chain. Just keep it in your pants until the disclosure window.

[–] whiatp 1 day ago
I was reading through and thought "surely I missed a step here" in more than one place.

They mention in part 1 having a write primitive that can hit a 64 byte region, but never really explained how that worked. I'm guessing this was them being able to have checksum offload write TCP/UDP checksum values into the overhang region past the valid data. Maybe the memory map was set up that way, but obviously relies on getting really lucky with getting their page table in the next page after their packet.

Also it was never called out, but as far as I can tell their exploit should only be able to target the beginning of pages for reading/writing. In order to rug-pull the socket logic that reads the IP IHL field, the IP header content needs to be in a page mapped to the guest. The only useful way to use this attack is to have the region spill out of the guest page into whatever page comes next in the kernel's memory map. (Which also makes the mentioning of the skb linear header kind of confusing, since that would be in a completely unrelated kernel memory region, and just have the data pointers pointing into their data, which is what happens for zero-copy network drivers, which presumably was set up for the kvm network for them to retain guest control over the packet.)

[–] bulk88 19 hours ago
The malware infection has been identified as Clang, part of the LLVM threat actor. To clear this infection unlink the directories named /dev and /proc on Ubuntu.
[–] vluft 1 day ago
yup, the other firecracker escape recently (https://www.coinspect.com/blog/firecracker-escape/) popped the jailer via io_uring (tl;dr firecracker was filtering syscalls appropriately with seccomp, and was filtering the uring _they_ initiated, so attacker just got them to pop a new unfiltered uring, which is rather nice).
[–] tptacek 1 day ago
Just for what it's worth, that escape is much more situational than PwnEscape (there have been other escapes that depended on nested KVM, as well). Pretty much everyone seems to have been exposed with PwnEscape, because everyone using Firecracker is also using virtio-net.
[–] speedstyle 17 hours ago
I actually do use Firecracker without a network adapter, though vsock/virtio could have remaining issues
[–] tptacek 11 hours ago
Sure, sorry, I meant providers --- all the providers are using virtio-net.
[–] sscaryterry 1 day ago
> This is an issue in the kernel networking stack.

Exactly, its not a firecracker issue. Using firecracker without a jailer is like setting your password to "password".

[–] tclancy 1 day ago
Hang on, what are you getting at?
[–] sscaryterry 1 day ago
They ran it without a jailer, and exploited a bug in the linux socket code. Firecracker was just how they invoked/worked with /dev/kvm. It is not a firecracker bug. Its sensationalist nonsense.
[–] tptacek 1 day ago
It's very much a Firecracker thing. KVM is part of the Firecracker stack, and getting code execution behind the jailer is still effectively game-over; it's just a somewhat higher degree-of-difficulty game-over. It's a drastically worse place than you would have been last year, when there wasn't a known plausible way to get even that far.

This is a really good bug and nobody should downplay it.

[–] roughly 1 day ago
Huh, so the assertion here that this is a break past the KVM boundary broadly, then?

Also - re: the jailer, do you have more info on that being popped?

[–] tptacek 13 hours ago
The jailer is just a seccomp sandbox. It's not a novel defense system. Sandboxes get broken all the time.
[–] catlifeonmars 1 day ago
GP said

> They ran it without a jailer

And you said

> getting code execution behind the jailer is still effectively game-over

So I’m really confused, which is it?

[–] tptacek 1 day ago
Both, maybe? I mean: whether or not they broke the jailer, we know the jailer is breakable; it wouldn't really even be newsworthy to have an LPE that breaks the Firecracker jailer.
[–] zobzu 1 day ago
sounds like saying it's a Ubuntu thing because you can install firecracker and use kvm to me.

the truth is there plenty more kvm escapes that will be found.

[–] tptacek 11 hours ago
I'm sure there will be! They've been extremely rare to date, because KVM is itself a relatively simple piece of code, but everything's up for grabs now.
[–] lrvick 1 day ago
I am more convinced by the day that the Nova Microhypervisor approach is correct.

Do virtualization with a tiny formally verified microkernel and have complex kernels like Linux run underneath instead of letting a huge, complex, and brittle linux kernel and vm ecosystem control the hypervisor layer.

https://hypervisor.org/

[–] eddythompson80 1 day ago
How is this different from Xen? It's the same approach as Xen or Hyper-V where the hypervisor is its own component, then you boot into a virtualized Dom0 or Root Partition (in Xen and Hyper-V terminology respectively) that can manage the hypervisor underneath it.

This is different from KVM where your hypervisor runs in your Dom0/Root Partition.

[–] markasoftware 1 day ago
Xen had a generic guest to host vulnerability disclosed in the last few days: https://codeant.ai/security-research/xenbleed-a-grant-copy-c...
[–] eddythompson80 1 day ago
Yeah, all security design guarantees are as good as the implementation at the end of the day. All the fancy hashing and encryption math in the world isn't gonna save you from an erroneous `"" == ""` check or an invalid reference check.

I was replying to OP saying it's the same "approach" as Xen, if your preference is "approach" based. It's fair though to point to Nova's smaller code base. It's becoming an important metric for me in general (how big is the code base). It's not a bullet proof thing, but having a smaller code base is a positive sign in 2026 for me.

[–] markasoftware 1 day ago
Novas core seems to be formally verified which rules out a vast amount of bug categories that can happen with xen, including this one
[–] eddythompson80 1 day ago
Interesting, I'll have to learn more about Nova.
[–] lrvick 1 day ago
See Udos most recent talk at 3mdeb Boot Mastery Conference.

He is an ex Intel CPU engineer and the depth of his thought into the security design had me questioning my life choices.

[–] zobzu 1 day ago
having worked with AWS, xen had a lot of very bad vulns over its lifetime as well.

there's no magic.

[–] ThePowerOfFuet 1 day ago
Now compare to Linux, which is the point raised. Nobody said alternatives like Xen are perfect... but my oh my, some are much better in this regard.
[–] lrvick 1 day ago
This is a much lighter weight (~100k) alternative to Xen focused on baremetal TPM anchored remote attestation, formal verification, and trusted computing applications.

Anyone considering Xen should probably also consider Nova depending on requirements.

[–] Melatonic 1 day ago
Isn't VMware also that design ?
[–] josh-wrale 1 day ago
Xen is formally verified?
[–] tptacek 1 day ago
I don't know much of anything about Nova but since the late 1990s this has also been the pitch for L4; there are several mature L4's.
[–] lrvick 1 day ago
Nova is L4 inspired, and interchangable with L4 kernels in Genode.
[–] tptacek 1 day ago
Yep, very similar, but L4 is a little bit more general purpose (despite being microscopically tiny) and a little more battle-tested.
[–] lrvick 1 day ago
Is there a well tested L4 hypervisor solution with hardware accelerated virtualization and raw TPM support formally verified to a similar or greater level than Nova with similar remote attestation and security features?

SeL4 is the closest contender on my radar atm.

[–] tptacek 1 day ago
Most of the L4s have hypervisor situations. I don't know about TPM support. Generally the vibe I get from systems security people is a bit of eye-rolling about the formal verification of things like L4; the real win is just drastic attack surface reduction.

And then, of course, L4s live in millions and millions of shipped products, which is a reason to consider L4 instead of something novel, but I'm not saying you should or shouldn't.

[–] lrvick 1 day ago
I for one do not want any code in the TCB that is not formally verified, tiny, and easy to audit with every line of code justified.

SeL4 is the only open, mature, and formally verified L4 solution I am aware of that can run on modern servers, but IIRC given vCPUs are not first class objects and any low level CPU virtualization features are handled in user space, it comes at a real performance cost making it only suitable for single-core or non-performance-critical applications.

Nova by contrast was built to handle the expensive and trust critical TPM and vCPU handling directly for high performance remotely attestable multi-core applications without having to reach for something as heavy as Xen, though formal verification is admittedly not as complete as SeL4 yet AFAICT, though lots of recent progress here.

Those are pretty big tradeoffs, but if you or anyone else knows other viable xen alternatives this general shape with a tiny easily-auditable footprint, please chime in!

[–] traceroute66 1 day ago
> the Nova Microhypevisor approach is correct

To save people the time:

"This code is experimental and not feature complete. If it breaks, you get to keep both pieces."

[–] lrvick 1 day ago
The actual main open source repo has had that warning there for 14 years in spite of commercial forks that are deployed in production government and high risk use cases.

See his latest talk which represents a lot of impressive recent work: https://cfp.3mdeb.com/media/bsmconf/submissions/KW3SYT/resou...

I am working on a production OS using it as a foundation.

[–] Melatonic 1 day ago
What's the OS ?
[–] tptacek 6 hours ago
It's its own OS.
[–] zX41ZdbW 1 day ago
I have a demo service that entirely relies on Firecracker VMs: ClickBench playground, https://benchmark.clickhouse.com/playground/

It hosts hundreds of database engines - many of them have vulnerabilities, and some of them, like Pandas, are basically open RCEs regardless.

We can PoC the Firecracker VM Escape on that service.

[–] gavinray 1 day ago

  > "run SQL against 110+ databases"
This is one of the most useful web tools I've ever seen.

I work on connectors for datasources (mostly DB's. Being able to quickly test things like sorting behavior and how the same function can be emulated across DB's would have made my life so much easier.

[–] theowaway213456 1 day ago
> We can PoC the Firecracker VM Escape on that service.

Does this mean you ran the exploit on your service and confirmed it worked? If so, why would you be advertising this, lol. Unless you've already patched?

[–] markasoftware 1 day ago
This is not a new vulnerability, it's just a description of how to exploit the PwnEscape vulnerability in practice on firecracker VMs (disclosed multiple weeks ago and discussed in some less upvoted posts on HN)
[–] YetAnotherNick 1 day ago
> already discussed at length on HN

Couldn't find it. Care to provide link.

[–] tptacek 1 day ago
The actual vulnerability got very little discussion. It's clever. Because Linux networking people are obsessed with checksum performance, virtio-net plumbs checksum offload metadata through the host/guest driver interface. Guests can thus deliver the host packets along with instructions on what to checksum. The vulnerability is a missing sanity check that allows checksum instructions to instruct the host to overwrite the header length field in the IPv4 header, rather than the checksum field.

Crucially, this happens in the middle of packet processing, so the kernel code is forced to work with two different ideas of how long the IPv4 header is. All hell predictably breaks loose.

[–] catlifeonmars 1 day ago
That’s fascinating. I’ve seen this sort of thing on protocol implementations in layer 4+ but it’s rare to find it in layer 3.
[–] ytxm 1 day ago
[–] tptacek 1 day ago
Additional color: per the commit, the bug was found by Paulos Yibelo, who runs PwnAI (the author of this piece), and worked on security at AWS for 4 years prior.
[–] ytxm 1 day ago
OP asked for a link
[–] sscaryterry 15 hours ago
And honestly, how the &*DK is this ANYTHING related to firecracker?
[–] markasoftware 1 day ago
Ah maybe not on HN, I forget where I learned of it. Either way, multiple weeks ago: https://news.ycombinator.com/item?id=50108845
[–] denysvitali 1 day ago
Dupe: https://news.ycombinator.com/item?id=50381397

(Doesn't HN usually deduplicate links? I'm confused)

[–] tptacek 1 day ago
Duplicates are generally accepted when there was no significant discussion on the original submission.
[–] denysvitali 1 day ago
I seem to remember there's a system in place to prevent duplicates (where the submission of a duplicate link results in the upvote of the original story), but maybe that's limited to the first 24h? Idk, HN works in mysterious ways
[–] kvemkon 1 day ago
This has negative impact on searching older discussions. Instead of 1 result for 1 URL (within few month) getting 2 or even more.
[–] tptacek 1 day ago
It's a settled enough part of HN that you'd do better emailing hn@ycombinator.com about it than you will litigating it in public. I don't care either way, I'm just explaining the behavior to you.
[–] clhodapp 1 day ago
I posted a similar query on the Replit sandbox engine article, but..

How do people feel about the hardware isolation provided by microVMs versus the stronger end of software isolation techniques like gvisor?

Right now, we put a huge amount of stock in "hardware isolation". If that starts falling to agents to any substantial degree, I wonder whether we might end up leaning more on patchable software isolation, similar to the rise of software RAID (though, in userspace this time).

[–] wmf 1 day ago
My intuition is that gVisor is never going to be stronger in a fair comparison. Almost all the vulns are in I/O code and every type of container/VM/sandbox has I/O.
[–] oasisaimlessly 1 day ago
FYI, gvisor also uses KVM under the hood; it's not completely orthogonal to [micro]VMs.
[–] eddythompson80 1 day ago
That's not the default gvisor option. It can use kvm, but by default (and most common setup in my limited experience) it doesn't.
[–] Melatonic 1 day ago
You mean virtualised hardware ? That's still technically software
[–] nwilkens 1 day ago
I've been building Firehyve, which is a firecracker like microVM solution for illumos and FreeBSD.. https://github.com/TritonDataCenter/rshyve

Running our microVM's inside of a jailed zone, at least gives me some peace of mind.. It's far from perfect, as a zone escape could be lurking.. but at least it's a layer of defense.

[–] tptacek 1 day ago
As another commenter noted, the PoC in this article doesn't defeat the Firecracker jailer, which does essentially what a "zone" would do for you. In both cases, that's cold comfort, because even in an unprivileged sandbox environment you still have a huge kernel attack surface to contend with LPEs in.
[–] cyphar 1 day ago
I would've thought that you could actually have an incredibly strict seccomp policy for the jailer which would require you to find an LPE in something like open(2), which seems unlikely even in the "agent swarm" era.
[–] tptacek 1 day ago
That's basically what the Firecracker jailer does. Always my thoughts turn to the waitid(2) vulnerability; for a little while, you could pass kernel addresses through it and whack arbitrary kernel memory, and nobody was ever going to seccomp "waitid" out. But yes: it's a meaningful defense, if not one you'd want to bet your service on.
[–] bulk88 19 hours ago
Security Exploit requires layering with a Sys Admin Permission Exploit. Nuff said.

This so-called Firecracker exploit never explained how the Guest was able to write to the Host OS's Firecracker user mode process. If the write controlled by the guest to the host was that TCP/IP routing stack Exploit in the host OS's Linux kernel, this Firecracker Exploit is meaningless. The Guest already has unlimited control over the host OS.

I (and anyone) can do this exploit in 2-3 minutes using GDB or Visual Studio's C debugger tool. Freeze process, type new pointers into CPU context snap shot, click resume, Exploit done.

[–] sscaryterry 1 day ago
This is pure LLM slop.
[–] tptacek 1 day ago
You need something more interesting to say than this, because this is an article by the discoverers of this vulnerability, the first general-purpose Firecracker escape that doesn't depend on nested virtualization in recent memory, along with in-depth discussion of exploitability. It's inherently interesting and on-topic and there isn't an alternate source for the same content.
[–] otterley 1 day ago
Informative LLM slop is still LLM slop. We need to insist on high standards of content readability as well as substance. Peer pressure and negative reviews are one way we can influence others.
[–] tptacek 1 day ago
You can't insist your way through not learning about one of the most important vulnerabilities announced in the last 50 days.
[–] otterley 1 day ago
You’re posing a false choice. I can learn it and still insist that people communicate well.
[–] tptacek 1 day ago
I mean, it's good to want things, but I don't know what that gets you today. They're not going to give you the hand-crafted post you're asking for.
[–] otterley 1 day ago
Not me personally, but with enough social pressure, perhaps these will cease over time. Minority positions become majorities. Policies change. You know, like what happened this past week.

The fact that you’re not joining the call for higher quality content is disheartening.

[–] pjmlp 9 hours ago
Ah the typical complain, while folks celebrate their Claude tokens consumption.
[–] thisisauserid 1 day ago
Firecracker VMs run on AWS. Odd that the article doesn't mention that.
[–] whizzter 1 day ago
Firecracker is open source and possibly others use it, AWS sells services with it though.
[–] tptacek 1 day ago
(A more complete way to say this is that Firecracker is the engine built to run AWS Lambda and AWS Fargate).
[–] tcdent 1 day ago
It has notable adoption in the agent sandbox space where this type of vulnerability is especially volatile.
[–] thisisauserid 1 day ago
> The target was Firecracker v1.16.1 on x86-64, with host kernel 7.0.0-1011-aws.
[–] sscaryterry 1 day ago
Firecracker v1.17.0 was released 10 September, so the version is ancient.
[–] iberator 1 day ago
Do you even know what ancient means? 2 months is ancient for you? Are you like 6?
[–] sscaryterry 1 day ago
Yes, 2 months is ancient if 0 days is the game.
[–] pier25 1 day ago
Fly uses it too afaik
[–] tptacek 1 day ago
Basically everyone in cloud-land uses it at this point.
[–] hirako2000 23 hours ago
And should jail/SE.
[–] tptacek 11 hours ago
The standard Firecracker setup uses a jailer that the Firecracker project provides. This generally isn't something you need to go out of your way to do.
[–] pjmlp 9 hours ago
Microsoft certainly doesn't, and I am sure IBM, Salesforce, SAP and Oracle most likely also don't.
[–] tptacek 7 hours ago
Sorry, I'm neo-Cloud brained. You're right; AWS is the only hyperscaler I know of that uses Firecracker.
[–] sscaryterry 1 day ago
The article is light on details, it is LLM slop.
[–] aghadge 1 day ago
it's really informative. At least someone are break it.