Hacker News

points by abc42 2 days ago | 0 comments

>If you're an expert on the topic, framework, or library, even the best LLMs are still underwhelming.

How many people on this planet are experts on every topic?

>An LLM is like a low IQ person with vast knowledge.

Are you aware that frontier models have scored in the 135-195 IQ range when they have been tested? How smart are you?

Humans test higher on IQ tests if they had previously taken IQ tests of similar types too tho. And LLMs are trained on IQ tests. IQ tests only really hold their value when the test is fairly novel to everyone taking it.

[–] abc42 1 day ago

Okay. So what does it mean when somebody says that LLMs are "low IQ"?

[–] AngryData 17 hours ago

That if you gave them a test far outside their trained materials that it will do terrible compared to a human.

[–] abc42 17 hours ago

How do humans usually score if you test them far outside their trained material?

[–] solyin 13 hours ago

Poorly, which is why IQ tests are sometimes regarded as racist. Say the test asks a question about a regatta, and your culture has never introduced you to that word, so you fail that question, dropping your score by 20 points. Is that then a fair assessment of your IQ? Another place we see this is math ability. Street kids with no formal training in math may not be able to do traditional word problems, because they've never ridden, eg a train, but when those word problems are phrased in terms they are forced to use to survive, eg the prices of the trinkets they're forced to sell, then they excel at multiplying numbers in their head.

[–] Personability 1 day ago

Said tests are often quite inflated, quite frankly. If you can give me a source that disproves this claim, I'll readily eat my words, but I readily believe that those tests are just as inflated as all other claims about AI.