Yes, capabilities are correlated but an open question is how correlated, and how much negative transfer (where the model regresses on other benchmarks) will come into play.
Why are frontier models not chess geniuses (amateur club level elos), despite knowing the rules deeply, having a great deal of chess data in the training set, and having general reasoning training in other domains? Similarly, a beginner can beat a frontier model at Magic The Gathering. Why?
It is because chess and MTG are not priority targets in the (expensive) training pipeline. Chess is already solved by lesser neural networks so there is no wow factor, and nobody cares if they suck at MTG.
Mathematics on the other hand, has a huge wow factor, and a large and highly verifiable dataset.
Related: why did OpenAI give up on video?
I agree it's disorienting and scary but I'd phrase it like this: "the moment we come up with a systematic, objective way to verify a task, *and we can generate or find large datasets, and somebody has a reason to pay for the expensive training*, we become obsolete at it."
The economics side of it is an important constraint, currently ignored because the big labs are swimming in money (debt).
[–] senordevnyc 1 day ago
How comforting is it really when it’s not the bare LLM beating you, but the LLM using tools it created?
[–] root_axis 23 hours ago
Even if that were true, humans don't need to search the web and write python scripts to play a competitive game of MTG.
[–] pixl97 13 hours ago
The particular problem here is it's not addressing the intelligence spikiness between individual humans. Most humans aren't going to play a competitive game of MTG, a small percentage will rise above the rest and dominate.
Also this is being intelligence blind. Most humans could never write a good python script to play a competitive game of MTG, almost any SOTA model can.
This would be like a bee saying humans aren't intelligent because we can't see in ultra violet, where as we're damn good at making tools that can.
[–] senordevnyc 11 hours ago
I don't understand this argument, it’s not like humans and LLMs are engaging in some kind of competition with strict rules where ONLY the raw LLM responses “count” or something.
[–] chr15m 1 day ago
No. The current frontier models would not beat the best humans at MTG, even with Python scripts and web search. Unlike Chess, there are no existing engines that beat humans.
[–] senordevnyc 11 hours ago
[–] chr15m 7 hours ago
[–] senordevnyc 7 hours ago
[–] drdexebtjl 1 day ago
That makes me a good engineer, but not good at chess, and definitely not super intelligent.
[–] jasomill 1 day ago