Hacker News

points by sean_pedersen 2 days ago | 0 comments

An ill advised undertaking IMO: Any attempt to build AI generated content detection systems is flawed, since the outputs of such a system may be used to train an even better fake data generator. This leads to an equilibrium state of digital uncertainty: nothing in the digital realm can be deemed as real anymore - only as digital.

[–] modeless 2 days ago

AI detectors can't work in theory, but Pangram works well in practice. This is because the AI companies are not trying to fool Pangram. They have no incentive to; in fact the incentive goes the opposite way. They want to watermark their outputs to leave no doubt as to whether something was generated by their AI.

[–] orbital-decay 2 days ago

AI companies might have no incentive but determined actors do, by definition. Anything Pangram could possibly do (even semantic variation detection) can be trivially bypassed by doing img2img on a custom trained local model, which is in itself done trivially in practice by many people. Same for text, train it on anything and use distribution-aware sampling. From your words, you already have false sense of reliable detection, so a work of a determined actor will trick you into believing false negatives.

[–] eranation 2 days ago

I was a skeptic. Couldn’t trick pangram yet.

[–] vintermann 2 days ago

You can't trick it with AI generated text, but can you trick it into calling something AI generated which isn't? From what I've seen these AI detection tools are very good at detecting AI generation, they're also very good at calling anything AI generated.

[–] mjdv 2 days ago

Pangram boasts an extremely low false positive rate, in contrast to other AI detection tools. This has matched my limited experience.

If you have text that you can prove is not AI-generated (e.g. it is demonstrably older than about 5 years) that gets an AI flag, you could try tweeting at the Pangram founder, I've seen screenshots of him offering bounties (admittedly mostly as a dare to claims he doesn't believe).

[–] Hoasi 2 days ago

Really? Pangram is very easy to bypass in written form.

[–] gunalx 2 days ago

Yes. Even as little as a "humanize and rephrast this to be more natural" prompt with high temperature on a uncommon model reduces the confidence greatly.

[–] dr_kiszonka 11 hours ago

And does it preserve the original meaning, including various nuances? I have never been able to trick Pangram so I am very curious of the details.

[–] buf 2 days ago

removed

[–] cheshire_cat 2 days ago

Seems not so simple after all. How would one even go about training a local model on their own writing voice?

[–] bananaflag 2 days ago

This is what I've been saying for years, then I was surprised by Pangram actually working.

Sometimes it pays to attempt the impossible.

[–] futune 1 day ago

Yeah. Same. I found pangram baffling when I encountered it.

But by analogy compression is also impossible. And yet zip moves...

[–] bemtevi 2 days ago

One thing I've observed is that people who use AI rarely want to hide that they're using it. With text or even images, it's sort of like a dialect. AI-generated/assisted stuff is becoming somewhat like a fashion trend where people tend to use it and it becomes part of them, rather than something they want to pass off as entirely human-made.

So I think there's very little incentive for an arms race to occur in this case.

[–] gunalx 2 days ago

I dunno. Personally I like it to boost my self confidence in the text I write by doing a pass, but I really hate the overly verbose and unopinioned generic ai voice it always reflavours into.

[–] rtpg 2 days ago

there's some truth to what one says but after hearing how Pangram works I'm pretty impressed at the methodology

I think the founder once said something like "well you can try to fight our detection and do a bunch of stuff, you just might find yourself with a pile of extremely garbled text that is less than useless".

[–] theahura 2 days ago

yep, max said that at an agentics spotlight meet up last month

https://youtu.be/ntyZwokAfvU?si=NeA2rvBJVHAJ-qSS

[–] monster_truck 2 days ago

I'm not all that worried about this, I do have a very small concern about there only being one gig in town doing this. Creates a situation where they could be co-opted to label legitimate images as fake.

[–] esafak 2 days ago

I doubt AI labs have beating Pangram as an optimization goal.

[–] svantana 2 days ago

Not AI labs perhaps, but there are a bunch of homework-cheating services that boast low detectability on Pangram and other AI detector services.

[–] rwillmann 2 days ago

I can imagine labs competing against a Pangram benchmark to prove their models can output realistic images

[–] program_whiz 2 days ago

In theory you are correct, in practice it works. In theory you could use it as a training signal, in practice you cannot. In theory, theory and practice are the same, in practice they are not.

[–] zeroonetwothree 2 days ago

How is that a downside exactly? If we can’t tell then we are no worse off.

[–] sean_pedersen 2 days ago

It will create false trust in their score - giving people a false sense of certainty.

[–] karahime 2 days ago

Adding to this, the most motivated actors to get around it or find an edge case, e.g. in politics, will use it to say "Look, of course the scene depicted in this compromising image actually happened, the detector said so".

[–] brookst 2 days ago

Same way weather forecasts are a bad thing?

[–] edude03 2 days ago

This is how ai is trained anyway - generative adversarial networks

[–] 80fj0348i 2 days ago

> the outputs of such a system may be used to train an even better fake data generator.

[citation needed] - my understanding of how entropy works in real photos vs. synthetic images (AI or not) is that this is extremely difficult without severe degradation of image quality.