Almost all statements Yann LeCun made back then (LLMs won't be able to do this or that) turned out to be wrong. They were right to fire him.

And we’ve consumed pretty much all the data, and that part is getting harder as data is now expensive, Balkanized by owners, litigated and charged for. Couple that with model collapse as models eat their own output (not solved for any models, despite more than a decade of the problem), and add that lots of previously human curated datasets (stack overflow style), and it very well may be the models are not getting much better.

Is that observed scaling part of some equation or law? Is there a limit to how far that scaling goes before a ceiling or diminishing returns hit? Or is it simply based on more GPUs/parameters/compute plus how much human knowledge is in existence and will scale predictably forever?

But the whole problem is that these things aren't acting like other technological revolutions like the internet or Google where lots of profit comes from relatively little spend. They scale linearly maybe if you are super smart and good about your design.

Meanwhile, citizens all over the world are fighting data centers tooth and nail because we are hitting a scaling limit in terms of the resources people will allow to be put into this highway project and its equally unwise and untethered from reality to pretend this doesn't matter in a strategic sense.

Race down the transformer highway as fast as possible and explore other AI methods.

One need not preclude the other.

More and more, the talk of using generative models (e.g. transformers) is coming up to replace existing models in all concepts. Anything from understanding data, image recognition, labeling, ranking, etc.

As someone who is not an MLE and really doesnt understand these models well, it seems like the transformer approach is a "one size fits all" solution that may not be optimal for all problem categories but is good enough that its not worth investing in specialized models to get incremental improvements.

Which many credible, AI researchers believe is already starting up an increasingly steep scaling wall of diminishing returns that eventually ends at a brick wall.

I think that we're well into diminishing returns already. The models have improved, but to me the far bigger advances have been in feeding them contexts, building them into loops, etc.

I don't even think much more construction is really necessary in data centers. We can do a lot more with the power/HVAC we have, especially once the ASICs start coming online.

It's like we invented the wheel in 2022, the crowds are fighting over capital to build wheel factories, and savvier tech firms are quietly patenting automobile designs while everyone's focused on making slightly rounder wheels.

Feels like this is leaving major breakthroughs on the table for the competition. Sure, transformers might get us much farther. But if someone figures out an approach that unlocks a performance boost or better resolution then they are going to kill the competition.

Feels like the American industry is high on all the capitol being shoveled into the furnace without caring how efficient the furnace is. At some point someone is going to make a breakthrough and burst that bubble. This could crater the value as we are currently betting on the low power efficiency of the models as the barrier of entry and point of profit. Of course that would lead to cheaper AI but is that profitable vs local models once (or if) it becomes affordable to do so?

Already happening: https://news.ycombinator.com/item?id=50397994

Obviously we'll see how it pans out, but other approaches feel a bit like trying to ignore the bitter lesson.

Opus is 5T, not 1T.

> We'll soon start hitting 10's of trillions

Fable is already 10T.

> "Foot down, there's a straight highway ahead don't divert". I have the view this makes sense too.

The roadblock they ignored is politics.

Three days ago the states where most of the new datacenters are being built (Texas, Ohio, Georgia and others) elected governors who campaigned on halting datacenter construction. The highway metaphor implies money in => intelligence out. Turns out there's at least one other ingredient, and we just ran out of it.

Chinchilla Law says no, we wont. We're likely well past the point of diminishing returns but MOE hides a fair bit of it.

We already know that power laws apply to scaling so 10x the size is going to result in minor gains - where costs become a real factor (as if they weren't already).

FTA "The core philosophy there is scaling, and aggressively not research."

This is where the real gains for AI come from - good enough and CHEAP is going to win the day here (brute force, mixture of models). It's the engineers displacing the academics - and that will mean a bloodbath for revenue (see DeepInfer for what the end game looks like).

Where's the money coming for those 10X and 100X ventures?

Anthropic, which is more open about this has shown the numbers, it works.

This is a false dichotomy if I ever saw one

Why not hedge and just keep an ai skunkworks team working? These companies are big enough that they can have these people far away from the main work without distraction or without starving those teams of bandwidth

Is this true?

I barely notice any improvements ever since... 4.6? At least for software development.

Opus makes the same sort of mistakes and has the same sort of strengths ever since that point in my perception. My impressions is that increasing the number of parameters clearly has diminishing returns.

In fact, the best improvement I got from Claude was Haiku 5.5. Good cheap workhorse to implement plans.

Moore's law: Computing power is multiplied by 10 every 6 months? Can't wait for the optimization and rationalization phase, where we'll recover a trillionth of that power to run powerful databases.

I should also say that I use ChatGpt daily and enjoy it for researching algorithms. I tried to set up Claude last week and it/he was so overwhelming as to be unusable. So I think frontier models can be "unwieldy" or something for average use.

Langford has been at MSR a very long time and has done amazing research doing that time. He's bounced around a bit at MSR during his tenure and it's a shame to lose him. AI Frontiers wasn't around that long.

Much of this is Microsoft performing a very large and uncommunicated strategy update. All updates of MSR reorgs are just that "MSR is becoming more lean and focused" when it's obvious this isn't the case. This gets into the conversation that many Microsoft employees tend to have these days which is "Why does culture no longer matter at Microsoft and why did Satya turn into another heartless CEO". I won't speculate there, but the place isn't nearly as fun as it used to be.

To the author: 100% need this kind of thinking. It’s a balancing act. Historically though, dedicated labs haven’t had the best track record at commercializing their tech. The skillset differs vastly from running a product-focused business.

That said, even within this narrow scope - there are areas ripe for research.

Some of my particular interests include:

(1) Latent thinking/reasoning, which could arguably be more efficient but also opaque is one

(2) Separation of “intelligence” from knowledge - ie. keep the ability to reason like a frontier model and make tool calls (calling the ‘oracle’ or searching the web) but remove all parameters associated with niche knowledge/facts.

These are a couple off the top of my head that are very much aligned to scaling.

That statement is broad enough to be unfalsifiable, but if you narrow it down just a bit to scaling LLMs, maybe there is an end in sight. At least financial markets are skeptical of there being no end in sight.

If scaling LLMs really was the road to God in a box, then compute demand would be infinite, and nobody building to meet that demand would have a junk debt rating. Did those money people just not get it?

If so, then yeah, increasing efficiency 100x means we hit that point much faster.

If not, then it just means we make improvements 100x faster.

So far, the attempts to replace the transformer LLM with something else have yielded either "the same transformer LLM but with this one tweak that makes it 10% better" or "a radically different architecture that's a bitch to work with, and performs, best case, about the same as the transformer LLM".

By now, I'm firmly in the camp of "the architecture is good enough". Rifle is fine. Transformer is fine. Get better at the transformer before you look at anything else.

https://en.wikipedia.org/wiki/Digg#2025%E2%80%93present:_The...

Seems like it quickly pivoted quickly to cash in on "AI" hype.

Perhaps it is more free, but my understanding has always been that professional academia still restricts freedom of research. You have to apply for funding, convince the board that your direction will be fruitul.

An archaeologist can't just dig wherever. They compete with their peers for a limitted funding pool. If the school's board is more interested in the bronze age than the paleolithic, you're probably not going to get your dino dig approved.

The same is true for CS, I assume. You can't just "i would like spend university resources developing brainfuck 2.0" because the other teams are more likely to get their funding over you getting yours.

Both corporations and university's have finite budgets, both with bottom lines and margins. The uni wants their research done in a popular field that will attract students because students generate income that funds research that attracts students that generate income that funds research that...

They just think it makes sense to explore things, rather than just rinse and repeat with more cash.

The good news for John is that when 99.99% of the competitors are doing all the same, it’s when the rewards for looking around can be the biggest. Those are the real moments for disruption, and companies disregard them at their peril.

The working environment in academia is often precarious :-( , so such people often have no alternatives to going into industry, even though they still deeply appreciate and are influenced by the values of academia.

I don’t know what industrial research in CS is like in the era of agentic coding. On one hand I suppose building prototypes is faster than ever, but on the other hand I genuinely don’t know how well LLMs do with harder programming tasks. It’s one thing to have an agent work on a CRUD app, but I wonder how effective agents are at handling things like parallelism, distributed computing, load-balancing, and other topics that are harder to write than the average CRUD app. I am also aware of efforts such as Sakana AI’s automated scientific discovery research, and I wonder if many of the incremental, short-term research topics many companies pursue can be automated.

I’m a bit pessimistic about the state of research these days, from funding to publish-or-perish to the effects of generative AI (automation, AI slop publications, AI slop reviews). Then again, I’m a bit of a curmudgeon, and maybe things are better than I think.

Remember Bell labs, XEROX parc? Ask yourself why that existed and why research came out of them.

Monopoly.

So - many people don’t want to admit this but monopolies can be good from an R&D perspective, but without strong oversight by regulators - money will instead spent on R&D projects that specifically raise entry barriers.

That's just patently not true. History is littered with products that were very innovative and useful (and often far superior to competing products), but were canned, bought by competetors, or shelved in other way - often because they threatened the profits being made by some entity (though that's not the only reason).

Some are, some aren't. Some try, but botch it.