Hacker News

points by chr15m 1 day ago | 0 comments

Every dot in these graphs is composed of multiple benchmarks smooshed together.

Benchmarks are by definition a curated set of things that can be measured and evaluated.

Things that can be measured and evaluated can be (expensively) trained on.

The progress we are observing comes from figuring out how to measure and evaluate particular aspects of intelligence, and then expensively training on feedback loops made of those exact measurements, and synthesized data representing further measurements (literally benchmaxing).

What is not shown in these graphs are the set of things that intelligence can do, which are difficult, expensive or impossible to measure and evaluate.

That set of things is much larger than the set of benchmarks represented in the graph.

Therefore the post is misleading in the sense that the "superintelligence" being targeted is "jagged" to use Karpathy's term, and we infact already have jagged superintelligence (calculators, compilers, Bayesian spam classifiers, etc.)

We are witnessing a profound explosion of much more jagged superintelligence, restricted to the exact domains that can be (expensively) RL'ed on.

[–] reasonableklout 1 day ago

This isn't really comforting to me for 2 reasons:

1. A key insight of foundation models is that capabilities are correlated. For example, training on code improved general reasoning capabilities: https://arxiv.org/abs/2309.16298

2. A profound capabilities explosion in easily measured domains will still feel very different to today. Like, the moment we come up with a systematic, objective way to verify a task, we become obsolete at it?

[–] chr15m 1 day ago

Yes it will feel very different.

Yes, capabilities are correlated but an open question is how correlated, and how much negative transfer (where the model regresses on other benchmarks) will come into play.

Why are frontier models not chess geniuses (amateur club level elos), despite knowing the rules deeply, having a great deal of chess data in the training set, and having general reasoning training in other domains? Similarly, a beginner can beat a frontier model at Magic The Gathering. Why?

It is because chess and MTG are not priority targets in the (expensive) training pipeline. Chess is already solved by lesser neural networks so there is no wow factor, and nobody cares if they suck at MTG.

Mathematics on the other hand, has a huge wow factor, and a large and highly verifiable dataset.

Related: why did OpenAI give up on video?

I agree it's disorienting and scary but I'd phrase it like this: "the moment we come up with a systematic, objective way to verify a task, *and we can generate or find large datasets, and somebody has a reason to pay for the expensive training*, we become obsolete at it."

The economics side of it is an important constraint, currently ignored because the big labs are swimming in money (debt).

[–] jasomill 1 day ago

"Becoming obsolete" due to increased efficiency could lead to sigificant economic and political unrest, but there's no fundamental reason it must.

[–] senordevnyc 1 day ago

How much do you want to bet that an LLM with the ability to search the web and write python scripts would smoke the best humans at chess and MTG?

How comforting is it really when it’s not the bare LLM beating you, but the LLM using tools it created?

[–] root_axis 23 hours ago

> an LLM with the ability to search the web and write python scripts would smoke the best humans at chess and MTG?

Even if that were true, humans don't need to search the web and write python scripts to play a competitive game of MTG.

[–] pixl97 13 hours ago

I have doubts about your statement.

The particular problem here is it's not addressing the intelligence spikiness between individual humans. Most humans aren't going to play a competitive game of MTG, a small percentage will rise above the rest and dominate.

Also this is being intelligence blind. Most humans could never write a good python script to play a competitive game of MTG, almost any SOTA model can.

This would be like a bee saying humans aren't intelligent because we can't see in ultra violet, where as we're damn good at making tools that can.

[–] senordevnyc 11 hours ago

Again, who cares? How does that help humans compete against LLMs in any domain where the LLM will have web search, python, and other tools (which is basically always)?

I don't understand this argument, it’s not like humans and LLMs are engaging in some kind of competition with strict rules where ONLY the raw LLM responses “count” or something.

[–] chr15m 1 day ago

> and MTG

No. The current frontier models would not beat the best humans at MTG, even with Python scripts and web search. Unlike Chess, there are no existing engines that beat humans.

[–] senordevnyc 11 hours ago

I’m skeptical, but I don’t know enough about MTG.

[–] chr15m 7 hours ago

You can simply look this up to verify it.

[–] senordevnyc 7 hours ago

I’m skeptical of the first part, not the second.

[–] drdexebtjl 1 day ago

I too could smoke the best humans at chess if I was allowed to search the web and write python scripts.

That makes me a good engineer, but not good at chess, and definitely not super intelligent.

[–] pixl97 13 hours ago

Without tools you are nothing, most of the animal kingdom could kill you pretty easily. Engineering is a massive part of human intelligence.

[–] senordevnyc 11 hours ago

Maybe that’s true (though I highly doubt it, tbh), but it would also take you orders of magnitude longer than an LLM.

[–] novok 1 day ago

IMO I think a partial root of why AI writing is so bad is because it's written like code. Like if you really dive into how it constructs things after iterations and ask it to analyze it , without it saying it's like code or without you saying anything about code, you'll notice coding like iterations.

[–] brabel 22 hours ago

How many models have you tried? They write very differently. I found Gemini Argon 4 to be really good at writing like a human. Simpler models like Haiku seem better than more powerful ones like Opus, though even Opus has improved considerably with 5.5. DeepSeek is much more roundabout which can mirror certain people’s styles and some people prefer that. So in summary don’t just say LLMs are bad at writing, you need to be more specific: which model, which style, which context, what instructions were given? Claiming they are bad under all combinations is just ignorant.

[–] BoredomIsFun 21 hours ago

Oh, absolutely. The often voiced statements about "terrible Ai writing", "LLMs cannot joke" they all seem to be fashionably regurgitated by people who had last time used an LLM for said purpose in 2024 if ever.

[–] BoredomIsFun 21 hours ago

> I think a partial root of why AI writing is so bad

I hard disagree with that statement. At shorter outputs, with proper prompting AI produce very decent writing.

[–] pigpop 16 hours ago

Transference from code to reasoning is not surprising, programming languages are formal logical systems and written code is essentially a concretized form of reasoning about a domain and operations in it. Training people in formal logic also improves their general reasoning capabilities.

[–] jamienk 1 day ago

But picture the next 10 years, where all kinds of people, organizations, etc train in a ll different ways with diff philosophies and goals - I feel like there will be some good, unusual, interesting ones in there, even if they all have their own complex set of limitations.

[–] azan_ 1 day ago

Apart from physical world interactions which is obviously lacking, which domains lag behind and do not improve then?