points by s_u_d_o 10 hours ago | 0 comments
Companies with billions of $, stopping R&D on the most revolutionizing technology/ breakthrough is something I can’t understand. I do understand there’s an AI Race to win, and that is with transformer and current LLMs architectures, but if research could find a better, more efficient approach they would re-shape this race, lead it, or even dominate it.
[–] ACCount39 8 hours ago
[–] nonethewiser 2 hours ago
And getting the transformer better may make finding the next thing easier
[–] Xenoamorphous 8 hours ago
I think it’s more likely that some future LLM model finds a more efficient architecture than a human does, at this point. So yeah I’d go all in on LLMs too.
[–] move6729 10 hours ago
They are losing tons of money on it. Simple as.
[–] IncreasePosts 10 hours ago
But is it a race to zero? Maybe AI will make very expensive things (eg software engineering) very cheap, but there will be enough competition that the margins are tiny and don't actually justify putting crazy dollar amounts into research
[–] move6729 10 hours ago
That's exactly what's happening. Software -> 0 marginal cost (obviously still need engineers to actually make the architecture). LLMs -> Price of Energy / Amortized Cost of GPUs. "AI Labs" -> Owned by Creditors
[–] s_u_d_o 5 hours ago
From an engineer, who’s devastated by this, i have to admit/raise my concern: why do we still need engineers to make architecture? Why is everyone saying that’s a must? For me the toughest problem was coding a huge repo/pkgs, architecture should be much of an easier task to do. I’ve seen it do sys design, scaled architecture better than 10X engineers. With good planning, context and prospective info -> it can produce extremely good architecture even for complex systems.
[–] nonethewiser 2 hours ago
Im not sure I would say architecture specifically. Just anything on the input and output.
[–] move6729 20 minutes ago
It cant lmao (skill issue)
So far, the attempts to replace the transformer LLM with something else have yielded either "the same transformer LLM but with this one tweak that makes it 10% better" or "a radically different architecture that's a bitch to work with, and performs, best case, about the same as the transformer LLM".
By now, I'm firmly in the camp of "the architecture is good enough". Rifle is fine. Transformer is fine. Get better at the transformer before you look at anything else.