RCA developed the first good color TV cameras. They were monsters. Originally, electronic TV (as opposed to mechanical scanning) used Zworykin's iconoscope. This needed huge amounts of light. It didn't have a photo-multiplier stage to boost sensitivity, but it did accumulate charge from light over the whole frame time. There was an alternative approach, Farnsworth's image dissector, which also needed huge amounts of light. It had a photo-multiplier stage but only the light from the dot currently being scanned by the scanning beam mattered.

So RCA threw money at the problem. They developed the image orthicon camera tube. It combined the advantages of both approaches, integration over a full frame and a photo-multiplier stage. It was insanely expensive, costing thousands of dollars in the 1950s. It needed about twenty different voltages and multiple independent power supplies. It worked reasonably well.

Then came color. That took three image orthicons (RGB) inside a box the size of a small refrigerator, along with optics to split the incoming image into three colors. Plus a rack of analog electronics per camera, connected through a large cable. Plus an alignment problem from hell, with a huge number of adjustment knobs.[1]

Scaling worked! From the 1950s to the 1970s, color TV broadcasting was powered by those monsters.

Now, independent of this, other people were working on other approaches. Vidicons, silicon detectors inside camera tubes, and finally fully solid state image sensors. Eventually, in 1983, after thirty years of image orthicon monsters, RCA produced a usable solid state camera. Today, a cell phone camera costs about US$10 in bulk, and it's far better than the monsters.

It took thirty years of low-priority development to get past the giant image orthicon cameras.

Electronic fluorescent lamp ballasts had a different problem. That was a suppressed invention. The magnetic ballast industry was dominated by Magnetek. Someone invented a solid state replacement, which was smaller and cheaper, and licensed their patent to Magnetek for royalties. Magnetek just sat on it, not wanting to gut their core market. The inventor hired Townsend, Townsend and Crew in Palo Alto and sued. (Read this in the law firm's brag book in their office.) Eventually, the inventor got a hundred million or so, solid state ballasts went into production, and Magnetek seems to be gone. (The company called Magnetek today, which makes crane controls, is an offshoot of a Litton unit, and seems to be unrelated.)

There are also parallels from the steam locomotive industry, the Baldwin Locomotive Works being the best known. Baldwin actually built the first Diesel locomotive. They made a few half-hearted tries at Diesels over several decades, then gave up and went back to steam. Baldwin ceased all locomotive production in 1956. The corporation lingered on for a while, not doing much.

[1] https://www.tvcameramuseum.org/rca/tk40a/pix/rcatk40aart.pdf

AGI will require architecture changes not simply more data.

LLMs work well for search problems with the help of a strong verifier.

As it turns out, every problem is ultimately a search problem. But not every domain has an easily available verifier.

The author states "I consider the scaling-only focus to be both misdirected and not fun.". I get the point but also totally get the focus on scale out for now.

Where can I read more about this? I have seen people make this claim for a while, and when I looked deep, it used to turn out to be wrong.

You're being uncharitable. He would have been right if Chain of Thought hadn't been pulled out of a hat, and no one saw it coming, or how u reasonably effective it would be.

Today, no one can know if there will be a future step-change improvement to LLMs at the same level as CoT, and conversely, no one can be absolutely certein there won't be - so it shouldn't be a firable offense because neither position obvious; even though it will look like it in hindsight.

More specifically:

* he said that autoregressive transformer is fundamentally flawed: if it makes one error, all further generated tokens make no sense; that's, obviously, bullshit: there are many ways to implement backtracking, and there are many papers demonstrating backtracking at a time he did that lecture. And, of course, it just doesn't matter - reasoning model can just write "Oh, that's wrong. Let's try a different thing". - it's just not derailed by one mistake.

* he said that GPT-1000 can't possible have understanding of physical world, when GPT-4 already existing at that time could solve a problem he described.

So it's not like he was bad at predicting things, he's intellectually dishonest.

Yann LeCun has the syndrome where someone who sees the value of B (non-text modalities, i.e. physical reality) cannot stop themselves from constantly critiquing A (LLMs), even though clearly A has its own advantages.

I think he is a smart guy. But he constantly deploys the crackpot's PR playbook.

The idea that someone should be fired just because they were wrong seems to be something that would cause issues if one wanted to think long term.

This approach will make people afraid of being wrong. It will lead them to avoid challenging dogmatic assumptions in large organizations and inevitably lead to group think.

People really shouldn’t be punished for saying wrong things. Long term it’s better for a company if they aren’t punished for that. Helps to let people freely criticize dogmas or propose new ones.

Which is superior, the big f*king monstrosity or the elegant on-demand, small batches, reduce waste?

Meta will go in the shitter and Deepseek will make the small batches.

The coding/math breakthroughs were caused by training models on the subset of their own outputs which pass a verifier, see reinforcement learning from verified rewards.

https://youtu.be/lKoJJxjUfdw is a really excellent lecture that changed how I think about it.

And you can in practice often see a large network follow an expected training trajectory for loss/entropy.

In practice data is getting more sparse than compute, so you are usually overtraining using more tokens and being suboptimal.

(This moves the argument to "is being really good at predicting the next token the same as being intelligent?" which has happily diverted many minds for years now)

Going close to as fast as possible while saving a stream for the other approaches makes sense as an approach, but let's not ignore that there's a compromise/tradeoff here. Might not matter in orgs with near infinite resources

Look at what happened to Meta where they squandered a lead, had some mixed results, and reorged to more product certainty and away from fundamental research.

Both of these approaches require quite a bit of institutional support notably.

So yeah, LLMs keep getting more powerful/useful but because of an industry wide pivot not a "straight highway" as imagined in the GPT-4 era, when simply going bigger and bigger seemed like a reasonable roadmap for the foreseeable future.

The one thing that is clear is if it can become more cost effective it’s world changing. Even at its subsidized standards now it’s world changing. And it’s reasonably classified as AI IMO, but it’s not self aware.

I think a lot of your general traditional researchers are fundamentally looking to understand ourselves through machines as proxies in a sense. Something self aware that senses, learns, adapts, so on. The current path isn’t there or necessarily clearly the key to achieving that. That may be the disconnect. I think LLMs may give us some hints at that but not the full picture.

Still incredibly powerful for things we once considered nearly impossible.

I see it that most of legacy research is not obsolete but can be significantly amplified by AI.

Bioinspired algorithms may eventually lead to vastly improved intelligence per a weight too.

I don't think we should do this, but I feel doomed in that I suspect that we "can".

As in your statement to me indicates that you clearly see the scaling working over longer timespans. You chose a relatively recent 2026 model as your baseline after all, not a 2025 model or earlier. You clearly see that the 2026 model is better than ones in the past.

Might i suggest that you perhaps just don't see the improvement as clearly when comparing iterations that are fairly close together? If you're not anecdotally seeing improvement between an early 2026 model and one from ~6 months later perhaps just give it another iteration? Personally i do see 5.5 as much better but i can understand not seeing a lot of difference to 4.6 since they literally aren't that far apart in releases.

It

US rightfully placed a bunch of labor/environmental regulations around manufacturing which moved to China but stupidly didn't place restrictions on goods manufactured with severe labor and environmental violations.

We are repeating the mistake again.