Hacker News

points by tensor 6 hours ago | 0 comments

Odd. As I see it he was one of the few right about everything. This is also taking his comments about transformers scaling out of context. He specifically said they won’t lead to human level ai and they haven’t, despite exhausting the training data.

AGI will require architecture changes not simply more data.

[–] mr_roboto 5 hours ago

I'm quite confused by what you mean human level AI here. Maybe we haven't hit Einstein but we blew past the average person a long time ago.

[–] haltingproblem 4 hours ago

It is very jagged and some people don't consider that human level AGI - it can solve Maths at the genius level just like Deep Blue could play chess at the grandmaster level. At this point it is simply an opinion and I can see the other side too.

[–] procone 4 hours ago

Sorry, but what? We haven't done anything beyond strapping a large language model to a "harness".

At no point do I ever confuse a human being with a Claude session.

Even if I'm somehow fooled or confused for a minute, the context window / hallucinations / Claudeish will immediately give it away.

Sure, we passed the turing test for chatbots but we are very far away from AGI.

[–] Exoristos 4 hours ago

With this perspective on LLMs, you are presumably often "quite confused" if you interact with them to a significant extent.

[–] godelski 18 minutes ago

  > This is also taking his comments about transformers scaling out of context

There have been significant improvements in architectural changes, but since "pre-training" isn't being openly published people just lazily attribute it to "scale". Scale of what? Scale

  > AGI will require architecture changes not simply more data.

As Sutton claimed, all along[0]. People read what they wanted to read. Unfortunately, they also push against researchers who were trying to explore other architectures and methods. I honestly believe that has held us back. There was always room for people to explore different paths. The problem was, and has always been, preventing or discouraging others from exploring new paths. It's ironic given that one of the most common reasons a paper is rejected is for "lack of novelty"[1]. But if someone does something novel then the reviewer simply compares it to the existing state of the art, which is an unfair comparison[2].

It's silly since people want to move fast, but we're really not good at recognizing what has actually allowed us to move forward in the past.

[0] He speaks about Bitter Lesson directly at 9:57 but the earlier part gives better context: https://www.youtube.com/watch?v=21EYKqUsPfg

[1] An essentially meaningless statement. Near meaningless because it is not objective.

[2] You compare a novel technique against a technique that has had hundreds or thousands of researchers optimizing over several years. That's not even remotely fair.