points by tehjoker 12 hours ago | 0 comments
I'm not an AI booster (in a "should" not "can" sense), but in terms of "can" we are going to see next generation hardware (i.e. Vera Rubin GPU architecture) and improvements in model density (i.e. loosely improvements in intelligence per millions of weights), so the models will get bigger, the models will get more dense, and we may see more "expert models" i.e. in the sense that for a while there were human generalists that knew all of science and mathematics until the 1800s, after a certain point we had to specialize since there was too much to know. The models will need to partially specialize after hitting the hardware ceiling.
Bioinspired algorithms may eventually lead to vastly improved intelligence per a weight too.
I don't think we should do this, but I feel doomed in that I suspect that we "can".
[–] ACCount39 11 hours ago
[–] tehjoker 8 hours ago
At least for now, those keys haven't been completely unlocked, so parameter scaling is the primary way corporations are achieving increases in ability. If they hit the hardware ceiling, maybe the research direction changes then.
I forget who said it, but a trenchant insight is that current neural networks are generalized function approximators and essentially they are doing through brute force what biology accomplishes elegantly. Our hardware (both in the sense of our brains in terms of elegance and our computer chips in terms of raw throughput) is amazing.
[–] zer00eyz 5 hours ago
> so parameter scaling is the primary way corporations are achieving increases in ability
Thinking tokens have nothing to do with parameter count and where some of the biggest gains have come from.
We're really past the point of scaling parameters being viable. We simply lack the content to make "bigger" better at this point.
I'm incredibly skeptical of anything that goes "oh, we need to split AI into specialist systems".
I suspect that a lot of the delta between "our best models" and "base models they're trained on" lies in relatively low amounts of parameter changes driven by SFT and RL.
Model capacity is not the limit - the limit is our ability to drive those changes efficiently. Even things like "good research taste" could be represented as tiny in relation to total model capacity, if they could be trained into a model efficiently.