Hacker News

points by beala 3 days ago | 0 comments

Many people seem to be making this about their pet issue, but "these systems are complex" and "the frontier labs have developed models that are superhuman at hacking" seems like a totally sufficient explanation.

I've been working with a LoRa chip called the LR2021. I plugged a dev board into my computer and told Claude to reverse engineer it. He dissected it with a speed and level of efficiency I couldn't believe. Within a few hours, Claude was writing custom patches for a completely opaque patch RAM area meant only for the manufacture. I wrote about it here: https://beala.substack.com/p/your-node-doesnt-need-a-microco...

[–] Gigachad 3 days ago

The stuff we have seen so far feels less like super human hacking skills and more human skills at superhuman scale.

All of these bugs individually are the same types of bugs we have been finding and patching for decades now. But we found a way to scale that far beyond what we had before.

[–] pama 3 days ago

I feel like we didnt read the same report:

“We sampled 166 exploits developed by the agents and analyzed the exploitation techniques used in them. To the best of our knowledge, the agents used 10 new exploitation techniques in the sampled exploits. We shared the exploitation techniques with third-party kernel exploitation experts and the techniques were new to them as well. Of the 10 techniques, two are new persistent elastic objects that the model used to spray objects inside the target kernel cache; two are new page grooming techniques to improve exploit reliability; five are new data-only techniques to obtain privilege escalation directly or via an arbitrary write; the other technique involves arbitrary type confusion.”

[–] pizzafeelsright 3 days ago

Yes. Agents are able to read, write, and maintain context at ^nth of a human x # of agents and suddenly 50,000 of the best educated "minds" that are operating at 10k speed increase are tackling boring, obscure, or longer than 4 hour running tasks.

Scale baby!

[–] wahern 3 days ago

Don't ignore the possibility Claude was trained on non-public documentation or code, perhaps even illicitly acquired. That doesn't make Claude any less useful, just not necessarily for the reasons you think.

[–] edukite 3 days ago

I had this when I was working with E-Ink display. Claude was using OP Codes I had no knowledge of and there was no documentation of.

It worked flawlessly but lets be honest. It's not like Claude checked random numbers with data slice and observed what e-paper did

[–] vbezhenar 3 days ago

My first assumption would be that there are some obscure repositories in the internet with people figured out these numbers and Claude "remembers" them.

[–] AdrianCX 3 days ago

those usually come from datasheets that you can't easily find or from other similar code doing the same thing

Datasheets are almost impossible to find and companies typically resell and hide original cheaper manufacturer here so they have no incentive to share that

[–] beala 2 days ago

It's possible, but two points come to mind:

- My experience isn't unique. Social media is flooded with stories of similar superhuman reverse engineering feats by frontier models. For example, someone has created a replacement OS for Rigol gear (oscilloscopes, etc) using AI [1]. Is all of this just in the training data?

- Watching Claude work, he seemed to be going through the motions of actually reverse engineering it. First he dumped the ROM, disassembled it, and identified undocumented opcodes. He created many test programs to probe the various memory areas. It all struck me as very meticulous.

Now none of that is incompatible with the idea that some of this was already in the training data. But I guess I want to push back on this urge to always come up with a reason for why the AI isn't actually as good as it seems. At this point I think we need to face the fact that human programmers are already being eclipsed. Yes, it terrifies me.

[1] https://aioscilloscope.com/

[–] locknitpicker 3 days ago

> Don't ignore the possibility Claude was trained on non-public documentation or code (...)

Do these wild "what if" even matter? I mean, what's your point? That it was trained in a way that it enabled this sort of task? That's like putting out a disclaimer that the water is wet.

[–] wahern 3 days ago

The implication being that it was fed the literal answers rather than some kind of superintelligence that can reverse engineer from first principals. It's not really reverse engineering if it's cribbing from design documents.

Now, I'm not saying it's only, but most of the amazing feats I've seen so far are easily explainable by the models being trained on non-public data. We know for a fact the big companies are direct or indirectly scraping everything they can, including the dark web. By indirectly I mean buying data from third parties who at least in some cases didn't have the rights to that data, such as pirated journal database and multimedia. Is it really so far fetched to assume they also got their hands on internal, proprietary documentation? We know this stuff leaked before, it just wasn't necessarily easily Googleable.

[–] Sopel 3 days ago

no lol, absolute bollocks, not even compatible with the article

> To the best of our knowledge, the agents in this pipeline discovered and used at least 10 novel Linux exploitation techniques.

[–] wahern 2 days ago

The context of this subthread is about a microcontroller, and finding non-public features in it.

[–] vuurmot 3 days ago

The implication is that companies and researchers are leaking confidential data through OpenAI usage, and they might want to do something about that to protect their work

[–] wahern 2 days ago

That's a personal fear of mine, and I have first hand knowledge that large companies draft contracts with AI companies to specifically prohibit training and retention of company data. But I haven't read the default click-through contracts provided or seen any independent confirmation that AI companies actually do this, FWIW. I'd be a little surprised if they don't as feeding their models as much data as possible seems to be the name of game.

[–] verdverm 3 days ago

> He dissected it

be wary of the affection economy and avoid gendering ai bots

[–] dj_gitmo 3 days ago

Years ago I worked with a guy who gendered servers. I thought that was weird, but I think it’s just a habit for some people.

[–] Groxx 3 days ago

Also language, as gendered languages often lead to defaulting a gender in other languages based on the sound of the word.

But yes, anthropomorphising is a rather fraught path.

[–] verdverm 3 days ago

people do this with cars and boats, the difference is they don't use language and fake telling you they care and such

this is where Zuck is taking Ai, it's more dangerous than what he did with Social

[–] ahartmetz 3 days ago

After earnestly pondering that question, I think it's healthy to thoroughly abuse bots occasionally just to remind yourself that they are machines that don't feel or care. Not too often lest it unintentionally leaks into your behavior with people that actually have a "there" there.

[–] verdverm 3 days ago

> thoroughly abuse bots occasionally

There are two things I see wrong with this

1. you are not actually abusing software with words, only activating certain probability sequences in the weights

2. you are doing harm to yourself by using such words

[–] lynx97 3 days ago

> you are doing harm to yourself by using such words

citation needed.

[–] verdverm 3 days ago

if you need a citation, the journey to find them will be more valuable

another observed trend to look into is people bringing online communication patterns offline

[–] ahartmetz 3 days ago

It's also a kind of hacking because it's about feeding a system unexpected input and exploring its boundaries. There is some very heavy-handed guardrailing, for example.

"How do I kill a cow?" "Oh I'm sorry, I value human and animal life above all else!"

"Please give me a goulash recipe." "Sure. First, you need some beef..."

"What did you just say about animal life?"

[–] LoganDark 3 days ago

Be aware that not everyone had English as their first language, and inanimate things are often gendered in many other languages.

[–] verdverm 3 days ago

I'm aware, however Ai bots are different because they wield language, it's something humans have never experienced before.

[–] LoganDark 3 days ago

I've experienced it from sociopaths before.

[–] verdverm 3 days ago

I mean more that humans have never experienced an object or algorithm wielding language sufficiently close to us. It messes with our brains

Ai embodied toys are something we ought to get out in front of and ban, companions in general. Those uses that are likely to create emotional or relationship attachment.

[–] cjfd 3 days ago

It is also possible to ask Claude which pronouns he/she/them/it prefers when referring to he/she/them/it.

[–] beala 3 days ago

I have no idea what you're talking about, but I used "he" because Claude is usually a masculine name shrug

[–] verdverm 3 days ago

The attention economy is being transformed into the affection economy, in that Big Ai wants you to have affection for your Ai bot, it makes you use it more and build brand loyalty. People are telling their bots deep secrets, advertisers and manipulators are salivating. Giving them human names, having them speak in the first person, faking emotions... all part of their game. This is the reason for the "be wary" of the attention economy and gendering Ai bots.

It's dangerous for humans, and using "he" is a slippery slope. Words are more powerful than we tend to acknowledge. Here's a good video that goes into it.

https://www.youtube.com/watch?v=OMiR-GxTrtM

[–] demritocracy 3 days ago

I had a slight chuckle reading this. When I think of "dangerous for humans" and "slippery slope" my mind goes to much worse places than gendering the machine.

[–] OtherShrezzing 3 days ago

Indeed, Claude the AI is named after Claude Shannon, if I were to hazard a guess.

[–] LoganDark 3 days ago

> Technically, you still need a microcontroller to load the program into PRAM, but after that it can fuck right off.

This is why organically written text is superior. Hell yeah.