And you can in practice often see a large network follow an expected training trajectory for loss/entropy.
In practice data is getting more sparse than compute, so you are usually overtraining using more tokens and being suboptimal.
[–] pyrophane 6 hours ago
[–] augment_me 4 hours ago
Are you smart if you see that cognitive scientists had something figured out, connected it to something in your field of chemistry and made a breakthrough treatment for some disease?
Does any of our existing intelligence build on completely unknown information? You could argue that any new solution/emergent intelligence is an association of existing information in new ways.
It also brings us to what is brute forcing? Humans are brute forcing stuff most of the time. You will have 10 stupid thoughts for every good thought. 10 stupid ideas for every good idea, you will discard most of them so fast that you won't think of that you did that. It's very rare, if not near impossible, that a non-mathematician with no prior knowledge, goes into a room on day 1 of deciding to think about a conjecture they don't know anything about, and solve it on the blackboard. They need to use their collect associations, formalize their axioms, think of first principles (known information), and associate it.
So to answer your question you need to answer:
1) What does "smarter" refer to? Is it accumulation of information about the universe?(if yes, a database is smart) Is it an efficient, abstracted representation of accumulated information? Is it prediction of the future state of the universe given a learned abstraction?
Or going further, is it solving tasks useful for the human in the world? Is it acting as an agent with it's own goals and interests in the universe? What if the goals are hard for humans to understand?
Does it count as smart if you make an evolutionary approach that spawn 1000 instances and the most fit one survives and spawns 1000 mutations of itself?
2) What does "genuinely new approaches" refer to? Is it fine to use math axioms to prove things? Does interpolation between two existing solutions count as new? At what granularity of concept does novelty happen (Does using two higher abstraction concepts, like combining some new, modern neural network architecture with a database count as a new thing?), even if the underlying lower abstraction of the same information can be expressed as storage, latency and compute, and is not novel?
---
As you can imagine, this is all not really defined or agreed upon. Here are some examples of projects that are set up to drive these ideas:
The Hutter Prize poses that intelligence is defined by compression, so if you manage to compress the English Wikipedia efficiently, you are closer to better intelligence (http://prize.hutter1.net/)
Compute: Intel CPU
The ARC-AGI foundation poses that being able to be goal-oriented and understand higher level abstractions is intelligence: https://arcprize.org/leaderboard
Compute: NVIDIA T40s
Modded NanoGPT competition poses that intelligence is minimizing entropy/loss over a dataset in the least amount of steps: https://github.com/kellerjordan/modded-nanogpt
Compute: 8x NVIDIA H100
---
All of this also exists in the context of the compute medium. It could be, that you need a certain capacity of memory/computation to perform whatever we have defined as intelligent tasks. Hutter Prize takes the bet of that to make intelligence, you don't need a lot of computation, an Intel CPU laptop and 50GB RAM should be enough.
A lot of the big labs are instead taking the bet on that you just scale everything, make trillion-sized models and things will fall out of that automatically, for this you will need a very different compute medium (data centers) compared to (a laptop).
[–] eli_gottlieb 5 hours ago
There's a mathematical sense in which it cannot be: the data-processing inequality (https://en.wikipedia.org/wiki/Data_processing_inequality). The extent to which that sense is totally counterintuitive to how things normally work in ML nowadays has led to investigating different foundations for information theory: https://arxiv.org/abs/2601.03220.
[–] joe_the_user 9 hours ago
These empirical observations on quantities which aren't "intelligence" since no one has yet defined intelligence.
I wonder about this a lot.
There are all these publications from OpenAI and Anthropic that are solving complex math & science problems.
Are they just "brute forcing" it? Able to try a lot more "known" or reasonable approaches given existing knowledge?
Or are they showing "emergent intelligence" and able to try genuinely new approaches that can't be expected from the knowledge in their dataset?