Hacker News

points by gandreani 4 days ago | 0 comments

A lot of people are focusing on how irresponsible the engineer was about using the work computer. That's obviously an issue but the bigger failure mode to me is the Engineer C's container deployment token.

That's a huge red-flag. In a company of this size, the deployments should be via CI only. CI isn't just to increase the speed and reliability of deployments but also to provide a clean room to build the software! Even if it's a token that was copied and pasted temporarily, the systems in place should not have to require this type of 1. copy 2. paste 3. remember-to-delete the token from my PC.

Also, Binance should be paying for intrusion testing. Better to pay a trusted company to do this kind of thing instead of letting North Korea show you how it's done. I don't think this exact exploit path would have been found but I bet the intrusion testing would have figured out another way to use Engineer C's token.

[–] panicinducer 3 days ago

Nobody in the entire company should have access to move $billions in crypto, much less a random engineer, even if they're part of the deploy process.

[–] Quinner 3 days ago

They had controls in place to prevent large transfers, so the attackers structured it as a lot of small transfers very quickly

[–] melenaboija 3 days ago

Well, then they need another control that prevents “small transfers very quickly” happening, I guess.

[–] theoreticalmal 3 days ago

Aren’t “small transfers very quickly” kind of the whole point of what Binance does?

[–] IAmBroom 3 days ago

Great hindsight! Anything you want to suggest in advance of it happening?

[–] yobananaboy 3 days ago

Not just that! The EIP-7702 delegation to make the wallet into a smart contract bypassed all of the controls that they had in place on the signer. It's wild that they didn't have a system in place to stop it from signing that transaction type.

[–] tennysont 3 days ago

From my rough understanding, this seems like the main exploit. It seems like the machine with access to the wallet key was presented with a crazy EIP-7702 transaction and signed it (which effectively owned its wallet in a single transaction)[1]. Without knowing their security model, it seems like biggest trust boundary should be between that signing machine and everything else.

[1] The fact that the needed to sign 50 of these transactions to solve a race condition seems like an implementation detail

[–] yobananaboy 1 day ago

They had to sign 50 of them because they had to compete timing the transaction with the normal transactions. Transactions have to be in nonce order for the wallet, and if two transactions with the same nonce are submitted to a block, it is up to the miner / validator to determine which one to put in. So if you spray 50, you can almost guarantee that the transaction you want will get in.

[–] stogot 3 days ago

Is quorum signing not a thing in crypto?

[–] cowlevel 3 days ago

In Ethereum it would be implemented by linking your account to smart contract code, then throwing away your private key so you can only make transactions through the smart contract. The approval of linking to smart contract code is exactly what they signed here - only it wasn't a multisig smart contract, it was something malicious.

[–] tennysont 3 days ago

It is---though I don't see the connection

[–] PeeMcGee 3 days ago

A disgruntled employee wouldn't do the exact same thing?

[–] neutronicus 3 days ago

That's the plot of Office Space!

[–] wahern 3 days ago

Also Superman III, which predates it and presumably was well known to the writers. But I have an inkling the scheme and it's infamy go back much further. (Precious metal coin clipping/debasement?) It's one of those ethical questions, like a pauper stealing bread, or thought experiments, like "what if reality is just a dream" that many if not most people encounter during their formative years. Per usual, though, the chasm between theory and application (even just identifying real world potential when it matters) is the toughest one to cross.

[–] vondur 3 days ago

They also mention that in the movie.

[–] otterley 3 days ago

https://en.wikipedia.org/wiki/Salami_slicing_tactics

[–] wolfi1 3 days ago

there's even a Peter Ustinov film from the 60s

[–] theoreticalmal 3 days ago

Where’s Milton when you need him??

[–] burningChrome 3 days ago

If they did and didn't account for a simple scenario like this happening and built in roadblocks and friction to prevent it from happening? Then it would stand to reason their entire EIS team should be fired.

The fact nobody thought of this is pretty staggering.

[–] littlecranky67 3 days ago

It is hard to imagine they lost 6.3$bn and are not firing people...

[–] Betelbuddy 3 days ago

They just spent 6.3$bn to train them....

[–] stinkbeetle 3 days ago

> They had controls in place to prevent large transfers, so the attackers structured it as a lot of small transfers very quickly

lol that's the worst control I've ever heard.

[–] SpaceNoodled 3 days ago

OK, so nobody else has seen Superman III?

[–] RHSeeger 3 days ago

It is such a weird combination of "I enjoyed this movie" and "this really isn't a good superman movie"

[–] wolfi1 3 days ago

at that time I was not old enough to see it

[–] Animats 3 days ago

Nope.

[–] ForHackernews 3 days ago

Be your own IRM 4.26.13!

[–] long-time-first 3 days ago

I think they did that in Superman 3

[–] jMyles 3 days ago

This isn't a mundane detail, Michael!

[–] theatomheart 2 days ago

best part of this thread so far

[–] fhn 3 days ago

most employees shouldn't be able to touch the funds at all except a handful and probably require supervisor verification (dual factor authentication).

[–] ffsm8 3 days ago

I mean sure, but as long as you can deploy directly any permissions models are meaningless for an attacker? They can just be removed.

[–] benoau 4 days ago

> Even if it's a token that was copied and pasted temporarily, the systems in place should not have to require this type of 1. copy 2. paste 3. remember-to-delete the token from my PC.

One of the first things I did in my last job was shift processes like this off my team's computers, aside from the security ramifications all it takes is one dumb mistake, one up-arrow too many, one didn't-read-the-whole-line, and now you're doing something to production by accident!

[–] hirako2000 3 days ago

I also did that, for a team deploying smart contract on chain from local.

It took a couple months to get the pipeline out of their laptop.

Worth it, yes, as we found interesting things in there which explained why they were so reluctant to deploy upgrades.

[–] pjc50 3 days ago

> as we found interesting things in there which explained why they were so reluctant to deploy upgrades

.. backdoors which enabled the dev to deploy exploits of their own which wouldn't be visible through CI?

[–] hirako2000 3 days ago

No, rather a particular environment with monkey patched libraries, alias for various versions of compliers, and dozens of env variables that happen to work or not work well with the desired config.

[–] nightpool 3 days ago

What sorts of processes related to secret storage were you able to shift off of team computers? From my perspective, if I need to get e.g. an API key from Twitter and put it into my prod environment, i have to be doing that with my computer somehow. There's still millions of different types of secrets that need meat puppets to shuffle them around.

[–] nutjob2 3 days ago

> There's still millions of different types of secrets that need meat puppets to shuffle them around.

Sure, but those meat puppets shouldn't be developers, they should be administrators, using proper procedures.

Separation of powers seems so fundamental yet also so totally ignored these days.

[–] jacekm 3 days ago

TBH CI does not prevent this kind of mistakes. I've seen people triggering the wrong CI pipeline.

[–] steve1977 3 days ago

I guess people triggering the pipeline is already part of the problem

[–] rafabulsing 3 days ago

No matter what you do, at some point you have some human performing some action which will, after N indirection layers, result in a change in prod, unless you have the mythical fully automated agentic software factory.

[–] yubblegum 3 days ago

The article says that a software engineer interviewed with the North Koreans and the NKs then gained access to engineer A's machine (which had the deployment token) and then finally passed that token to run on engineer C's machine (who apparently did deployments). They knew of the team makeup from their first (caught) attempt.

That unlettered (npi) engineer who downloaded and ran the interview software on his or her machine is what the human weak link here. Let's call them Engineer F. Seems like this person was specifically selected from the dev team.

[–] burningChrome 3 days ago

I've had this happen a few times getting interview requests from Indian recruiting firms. It sounded familiar. You go through a screening, then get passed onto the "code challenge" part. You have to download a GitHub file and then work on it, check it back in and push it so they can examine what you did.

Twice I scanned the file after downloading it to a sandbox and twice it found malware that would've been installed.

While this is by far the worst case of this happening, I've read at least a half dozen other accounts of recruiting companies attempting to install malware through the same process. I'm not saying the person should have known, but it's scary this is becoming more and more common.

[–] fc417fc802 3 days ago

> I'm not saying the person should have known

This person definitely should have known. You don't download and run arbitrary binaries outside of a VM. At this point you don't even compile code from an untrusted source. This is the power user equivalent of falling for a phishing email.

[–] saalweachter 3 days ago

One of my favorite stories from a security talk was the speaker recounting the surrealism of (while doing a forensic investigation of a security incident) watching the attacker read your internal documentation to try to figure out how the hell anything works mid-attack, and running into all the same pitfalls as your new employees.

[–] rafabulsing 3 days ago

That suggests a creative new justification for not documenting your code :p

[–] eptcyka 3 days ago

No such thing as deleting secrets. You decommission them and replace them. Deleting them off disk is impractical to do, at least sufficiently.

[–] _heimdall 3 days ago

I don't even leave deploy tokens on my laptop for personal side projects. It isn't difficult to setup CI deploys, if nothing else the extra upfront cost let's me be lazy and have a consistent deploy pipeline.

[–] suzzer99 3 days ago

100% this. I can deploy to our sandbox environment directly, but only our CI/CD has the permissions to build/deploy to our master environment. And the stuff I work on is 1000x less impactful than $billions in crypto!

[–] jandrese 3 days ago

Maybe that second engineer who's laptop they were laser focused on was the CI/CD?

[–] r_lee 3 days ago

> Better to pay a trusted company to do this kind of thing instead of letting North Korea show you how it's done.

but enterprises usually prefer the latter, I guess it's more convenient since it bypasses the bureaucratic processes?

[–] AlexITC 4 days ago

I have saw companies doing social engineering as part of their intrution testing, I imagine fake interviews are (or will become) part of these.

[–] Aurornis 3 days ago

I think it will remain uncommon. I agree it would be a good test, but companies and employees alike get very angry when pentesting gets too close to sensitive subjects.

Anyone who fell for it would not only get in trouble for poor security practice, they would reveal that they were job searching. That opens an entire can of worms that creates problems beyond the pentest. This also angers other employees who see it as a double-trap being sprung on employees.

Another major problem is that responsible employees who accept the fake interview but then use their personal laptop to do the interview will have their personal laptop compromised by the pentesters. Responsible testers will immediately abort, but some will try to use the personal laptop to pivot deeper which is a huge ick. It’s also really bad if the company is compromising personal laptops of people who are doing the right thing (switching to personal gear for interviews).

[–] quaffcode 3 days ago

The pentesters wouldn't have to actually pwn a laptop, just getting the employee to download and run anything could go in a report without having the file actually be malicious.

[–] NikolaNovak 3 days ago

Would they have legal right to do that, out of curiosity?

Hired oentest has company permission to do stuff on company property. I do not imagine they have right to do so to personal property.

[–] manphone 3 days ago

Absolutely not. There is literally no framework, which would give you the right to exploit an employee’s personal laptop, that’s just cyber crime.

[–] realo 3 days ago

Then that is something the pentesters cannot do, but malicious actors will do.

Game over.

[–] PeterisP 3 days ago

What pentesters can do, should do, and often do is simply start the test from a workstation that is given to them - you assume a successful social engineering attack as the starting point, and test whether your process and technical controls can properly limit the impact of such a compromise without trying to replicate the actual social engineering step and the associated disruption to the targeted employees.

Taking over that first engineer's workstation shouldn't ever have resulted in $6.3B of damages. I would consider it reasonable if those privileges could be abused (by e.g. the same engineer intentionally) to cause $6.3M of damages and the rest would be mitigated or prevented; so for this incident we could say that 0.1% of the impact was caused by the initial compromise of that workstation and 99.9% of the losses were caused by all the other failures - so it's effective for any pentest to focus on the latter part, ignore the social engineering and phishing part (since you can't really eliminate it anyway - unlike many other flaws) and assume initial compromise as a starting point.

[–] manphone 2 days ago

Pen testers generally won’t physically assault your employees, bring guns and wave them around or shoot people, or kidnap their family members and torture them until they do what they want. What’s your point? Nothing.

[–] manwe150 3 days ago

I think my security form said something along the lines of “using personal laptops to access company resources grants implicit permission for my company to audit my laptop.” It goes on to explain that if you don’t agree to it, simply use the company device only.

[–] fc417fc802 3 days ago

> responsible employees who accept the fake interview but then use their personal laptop to do the interview will have their personal laptop compromised by the pentesters.

If your laptop is successfully compromised by the pentesters then pretty much by definition you weren't being responsible. Though I agree that pentesters going after personal devices is ethically dubious.

[–] Aurornis 3 days ago

Compromising employee personal devices is strictly out of scope for any security testing I've seen.

> Though I agree that pentesters going after personal devices is ethically dubious.

It's not just ethically dubious. It crosses a lot of legal boundaries. A company can consent to have their personal devices hacked. An employee doesn't automatically consent to let a company's contractors access their personal devices.

[–] skeeter2020 3 days ago

fake interviews might be excessive but lots of companies send tempting phishing emails as part of the secops, then both collect org-level stats and educate (or shame depending on your take) the employee for getting duped.

[–] ShinyLeftPad 3 days ago

Companies don't try to set up job interviews to test their employees loyalty?

[–] 0cf8612b2e1e 3 days ago

Everyone has their price. It is foolish to think that most people would not leave for better title/pay/job alignment/whatever.

[–] ShinyLeftPad 3 days ago

Well also to test that they don't just casually drop corporate secrets...

[–] thomc 3 days ago

Sometimes called "Full Spectrum Attack Simulation" where initial entry can include physical access or spear phishing. They can span months in elapsed time (not effort) so as to be more realistic. However there are still ethical and safety constraints and even phishing may not be able to use the (effective) technique of stressing the recipient too much or of course blackmail or coercing. So I think fake interviews may not be allowed in many cases, to protect the employees.

[–] shepherdjerred 3 days ago

Is that even legal? Wouldn’t a fake job interview be, in a sense, tricking you into taking time off and working without pay?

[–] phyzome 3 days ago

I think it would be legal but unethical -- unless you also then gave the employee substantial compensation for their time and opportunity cost.

[–] nradov 3 days ago

Legality would be very jurisdiction dependent. It might be illegal in most developed Western countries, depending on the exact facts of the case. I suspect it wouldn't be illegal in China — unless the trick happened to hit a ranking Communist Party member.

[–] FitchApps 3 days ago

And you never, ever, ever install software as part of the interview. Decline and hang up if they're asking you to install stuff. Visiting websites - maybe but even that unless it's a well known website I wouldn't go there but download and install stuff, hells no

[–] qurren 3 days ago

I fully agree with this.

If you ever meet with Alibaba for any reason they'll ask you to download their conference software. Unfortunately most people in the China ecosystem don't have the guts to say no and will just gleefully install any .exe thrown at them.

[–] zaphirplane 3 days ago

Hmm like building an node project, actually any project can have a dependency on a made up libraries that does the exploitation or even the project itself as made infamous by chrome extensions

[–] cldellow 3 days ago

"fork this repo and fix this bug" is a common job interview request, isn't it?

For many build chains, that involves implicitly installing software.

[–] fc417fc802 3 days ago

Such toolchains are dumpster fires whose use invites supply chain attacks. They should never have gained mainstream acceptance and should never be run outside of a sandbox.

That said I wouldn't trust a compiler to be hardened against malicious input and anyway you presumably need to run the output so all of that should be happening in a hardened sandbox regardless.

[–] dylan604 4 days ago

It'll be hard to time a job opening receiving interviews to line up with intrusion tests in a way that people would not be on guard against it in a way not normal.

[–] philipwhiuk 3 days ago

Fake interviews would be a huge time sink and probably piss off your staff.

[–] hirako2000 3 days ago

They don't have to respond. It gives employers an even stronger signal they should be kept right?

[–] Aurornis 3 days ago

Using pentesting to also test employee loyalty is a great way to get the entire company to hate working there.

[–] optimalsolver 3 days ago

Which is why it's going to happen.

[–] solyin 3 days ago

Only if your manager is an incompetent moron.

Which is also why we, collectively, as an industry needs to not put up with that. It's easy for me to say from the comfort of it not being my job on the line, that if this happens to anybody they should quit and then name and shame the company, but a competent security team is going to have the idea of doing a fake phishing interview and then not do it because of how morale destroying that would be. The security awareness team sending fake phishing emails are embarrassing enough.

[–] Naru41 3 days ago

Yes. The situation improves considerably if, when making server changes or retrieving data, programmers are required to enter a code generated from a hardware TOTP token isolated from both PCs and smartphones.

Every service provider system should be designed with the assumption that programmers' PCs and smartphones may be compromised.

[–] WesolyKubeczek 3 days ago

What if I plant a hook on the engineer’s laptop that will slightly modify their commits in such a way that eventually I smuggle malicious code over? First, exfiltrate the code over to me, learn from previous commits, start poisoning. Stamped by the CI, no less. In the days of slop PRs thousands of lines each shouldn’t be even too hard.

The point is that at some point you do need to open the door and let the code in.

[–] zaphirplane 3 days ago

Wouldn’t most companies worth exploiting run some static analysis in the ci cd

[–] stickfigure 3 days ago

The company should require both automated and human review of PRs. Mine does.

[–] mianos 3 days ago

This is the standard at most professional companies but it's now becoming a problem due to AI and undisciplined people submitting 10,000 lines MRs.

[–] zaphirplane 3 days ago

mine mandates no security bugs in PR :)

[–] WesolyKubeczek 3 days ago

"too long, LGTM, now buzz off"

[–] baal80spam 3 days ago

Well, that's still a human review. If something breaks, you (as in: this reviewer) is at least partly responsible.

[–] cyanydeez 2 days ago

shouldn't we focus on a unregulated monetrary system whose sole existence appears to be to support the Epstein class against democracy?