points by SideQuark 4 hours ago | 0 comments
More parameters without significantly more data fails; it loses the ability to generalize and simply memorizes.
And we’ve consumed pretty much all the data, and that part is getting harder as data is now expensive, Balkanized by owners, litigated and charged for. Couple that with model collapse as models eat their own output (not solved for any models, despite more than a decade of the problem), and add that lots of previously human curated datasets (stack overflow style), and it very well may be the models are not getting much better.
[–] samsartor 1 hour ago
[–] robrenaud 3 hours ago
Very large models trained with sgd variants don't tend to over fit, especially when using few epochs. See double descent.
The coding/math breakthroughs were caused by training models on the subset of their own outputs which pass a verifier, see reinforcement learning from verified rewards.
[–] comicjk 3 hours ago
This is what I thought a year ago, too. But how does it explain what has happened since?
https://youtu.be/lKoJJxjUfdw is a really excellent lecture that changed how I think about it.