> This is also taking his comments about transformers scaling out of context

There have been significant improvements in architectural changes, but since "pre-training" isn't being openly published people just lazily attribute it to "scale". Scale of what? Scale

  > AGI will require architecture changes not simply more data.

As Sutton claimed, all along[0]. People read what they wanted to read. Unfortunately, they also push against researchers who were trying to explore other architectures and methods. I honestly believe that has held us back. There was always room for people to explore different paths. The problem was, and has always been, preventing or discouraging others from exploring new paths. It's ironic given that one of the most common reasons a paper is rejected is for "lack of novelty"[1]. But if someone does something novel then the reviewer simply compares it to the existing state of the art, which is an unfair comparison[2].

It's silly since people want to move fast, but we're really not good at recognizing what has actually allowed us to move forward in the past.

[0] He speaks about Bitter Lesson directly at 9:57 but the earlier part gives better context: https://www.youtube.com/watch?v=21EYKqUsPfg

[1] An essentially meaningless statement. Near meaningless because it is not objective.

[2] You compare a novel technique against a technique that has had hundreds or thousands of researchers optimizing over several years. That's not even remotely fair.