Hacker News

points by jszymborski 2 days ago | 0 comments

My trouble with Pangram is that the calibration is totally off. It can tell you if there is a drop or an ocean of AI in the text you submit. The "X% of AI" message doesn't really mean much. You can test this yourself: Slowly add generated text to a human write paragraph and you'll see it hit "100% AI" quite quickly.

[–] Matticus_Rex 2 days ago

It analyzes in segments, and the percentage is of the segments, not of the words.

[–] jszymborski 2 days ago

Yes but the segment sizes are huge and any amount of LLm generated sequences within that block cause all of it to be flagged as AI.

[–] Matticus_Rex 1 day ago

That's the state of the tech right now. It'll be nice if we can get it and keep it more granular than that, but even at the current state it's pretty impressive.

[–] jrhizor 2 days ago

Good?

[–] Alpha3031 2 days ago

I don't really know why you would expect a "X% chance of containing AI" number to rise slowly or smoothly alongside the percentage of text that has a 100% chance of containing AI.

[–] kingofmen 2 days ago

Because it isn't an "X% chance". It's an "X% of words".

[–] Alpha3031 2 days ago

Surely not at the sub-paragragh level? OP said their test passage was a paragraph.