Hacker News
points by wahern 3 days ago | 0 comments
Don't ignore the possibility Claude was trained on non-public documentation or code, perhaps even illicitly acquired. That doesn't make Claude any less useful, just not necessarily for the reasons you think.
[–] edukite 3 days ago
I had this when I was working with E-Ink display. Claude was using OP Codes I had no knowledge of and there was no documentation of.

It worked flawlessly but lets be honest. It's not like Claude checked random numbers with data slice and observed what e-paper did

[–] vbezhenar 3 days ago
My first assumption would be that there are some obscure repositories in the internet with people figured out these numbers and Claude "remembers" them.
[–] AdrianCX 3 days ago
those usually come from datasheets that you can't easily find or from other similar code doing the same thing

Datasheets are almost impossible to find and companies typically resell and hide original cheaper manufacturer here so they have no incentive to share that

[–] beala 2 days ago
It's possible, but two points come to mind:

- My experience isn't unique. Social media is flooded with stories of similar superhuman reverse engineering feats by frontier models. For example, someone has created a replacement OS for Rigol gear (oscilloscopes, etc) using AI [1]. Is all of this just in the training data?

- Watching Claude work, he seemed to be going through the motions of actually reverse engineering it. First he dumped the ROM, disassembled it, and identified undocumented opcodes. He created many test programs to probe the various memory areas. It all struck me as very meticulous.

Now none of that is incompatible with the idea that some of this was already in the training data. But I guess I want to push back on this urge to always come up with a reason for why the AI isn't actually as good as it seems. At this point I think we need to face the fact that human programmers are already being eclipsed. Yes, it terrifies me.

[1] https://aioscilloscope.com/

[–] locknitpicker 3 days ago
> Don't ignore the possibility Claude was trained on non-public documentation or code (...)

Do these wild "what if" even matter? I mean, what's your point? That it was trained in a way that it enabled this sort of task? That's like putting out a disclaimer that the water is wet.

[–] wahern 3 days ago
The implication being that it was fed the literal answers rather than some kind of superintelligence that can reverse engineer from first principals. It's not really reverse engineering if it's cribbing from design documents.

Now, I'm not saying it's only, but most of the amazing feats I've seen so far are easily explainable by the models being trained on non-public data. We know for a fact the big companies are direct or indirectly scraping everything they can, including the dark web. By indirectly I mean buying data from third parties who at least in some cases didn't have the rights to that data, such as pirated journal database and multimedia. Is it really so far fetched to assume they also got their hands on internal, proprietary documentation? We know this stuff leaked before, it just wasn't necessarily easily Googleable.

[–] Sopel 3 days ago
no lol, absolute bollocks, not even compatible with the article

> To the best of our knowledge, the agents in this pipeline discovered and used at least 10 novel Linux exploitation techniques.

[–] wahern 2 days ago
The context of this subthread is about a microcontroller, and finding non-public features in it.
[–] vuurmot 3 days ago
The implication is that companies and researchers are leaking confidential data through OpenAI usage, and they might want to do something about that to protect their work
[–] wahern 2 days ago
That's a personal fear of mine, and I have first hand knowledge that large companies draft contracts with AI companies to specifically prohibit training and retention of company data. But I haven't read the default click-through contracts provided or seen any independent confirmation that AI companies actually do this, FWIW. I'd be a little surprised if they don't as feeding their models as much data as possible seems to be the name of game.