RCA developed the first good color TV cameras. They were monsters. Originally, electronic TV (as opposed to mechanical scanning) used Zworykin's iconoscope. This needed huge amounts of light. It didn't have a photo-multiplier stage to boost sensitivity, but it did accumulate charge from light over the whole frame time. There was an alternative approach, Farnsworth's image dissector, which also needed huge amounts of light. It had a photo-multiplier stage but only the light from the dot currently being scanned by the scanning beam mattered.

So RCA threw money at the problem. They developed the image orthicon camera tube. It combined the advantages of both approaches, integration over a full frame and a photo-multiplier stage. It was insanely expensive, costing thousands of dollars in the 1950s. It needed about twenty different voltages and multiple independent power supplies. It worked reasonably well.

Then came color. That took three image orthicons (RGB) inside a box the size of a small refrigerator, along with optics to split the incoming image into three colors. Plus a rack of analog electronics per camera, connected through a large cable. Plus an alignment problem from hell, with a huge number of adjustment knobs.[1]

Scaling worked! From the 1950s to the 1970s, color TV broadcasting was powered by those monsters.

Now, independent of this, other people were working on other approaches. Vidicons, silicon detectors inside camera tubes, and finally fully solid state image sensors. Eventually, in 1983, after thirty years of image orthicon monsters, RCA produced a usable solid state camera. Today, a cell phone camera costs about US$10 in bulk, and it's far better than the monsters.

It took thirty years of low-priority development to get past the giant image orthicon cameras.

Electronic fluorescent lamp ballasts had a different problem. That was a suppressed invention. The magnetic ballast industry was dominated by Magnetek. Someone invented a solid state replacement, which was smaller and cheaper, and licensed their patent to Magnetek for royalties. Magnetek just sat on it, not wanting to gut their core market. The inventor hired Townsend, Townsend and Crew in Palo Alto and sued. (Read this in the law firm's brag book in their office.) Eventually, the inventor got a hundred million or so, solid state ballasts went into production, and Magnetek seems to be gone. (The company called Magnetek today, which makes crane controls, is an offshoot of a Litton unit, and seems to be unrelated.)

There are also parallels from the steam locomotive industry, the Baldwin Locomotive Works being the best known. Baldwin actually built the first Diesel locomotive. They made a few half-hearted tries at Diesels over several decades, then gave up and went back to steam. Baldwin ceased all locomotive production in 1956. The corporation lingered on for a while, not doing much.

[1] https://www.tvcameramuseum.org/rca/tk40a/pix/rcatk40aart.pdf

AGI will require architecture changes not simply more data.

LLMs work well for search problems with the help of a strong verifier.

As it turns out, every problem is ultimately a search problem. But not every domain has an easily available verifier.

You're being uncharitable. He would have been right if Chain of Thought hadn't been pulled out of a hat, and no one saw it coming, or how u reasonably effective it would be.

Today, no one can know if there will be a future step-change improvement to LLMs at the same level as CoT, and conversely, no one can be absolutely certein there won't be - so it shouldn't be a firable offense because neither position obvious; even though it will look like it in hindsight.

The author states "I consider the scaling-only focus to be both misdirected and not fun.". I get the point but also totally get the focus on scale out for now.

Where can I read more about this? I have seen people make this claim for a while, and when I looked deep, it used to turn out to be wrong.

More specifically:

* he said that autoregressive transformer is fundamentally flawed: if it makes one error, all further generated tokens make no sense; that's, obviously, bullshit: there are many ways to implement backtracking, and there are many papers demonstrating backtracking at a time he did that lecture. And, of course, it just doesn't matter - reasoning model can just write "Oh, that's wrong. Let's try a different thing". - it's just not derailed by one mistake.

* he said that GPT-1000 can't possible have understanding of physical world, when GPT-4 already existing at that time could solve a problem he described.

So it's not like he was bad at predicting things, he's intellectually dishonest.

The idea that someone should be fired just because they were wrong seems to be something that would cause issues if one wanted to think long term.

This approach will make people afraid of being wrong. It will lead them to avoid challenging dogmatic assumptions in large organizations and inevitably lead to group think.

People really shouldn’t be punished for saying wrong things. Long term it’s better for a company if they aren’t punished for that. Helps to let people freely criticize dogmas or propose new ones.

Yann LeCun has the syndrome where someone who sees the value of B (non-text modalities, i.e. physical reality) cannot stop themselves from constantly critiquing A (LLMs), even though clearly A has its own advantages.

I think he is a smart guy. But he constantly deploys the crackpot's PR playbook.

Which is superior, the big f*king monstrosity or the elegant on-demand, small batches, reduce waste?

Meta will go in the shitter and Deepseek will make the small batches.