Hacker News
points by suddenlybananas 1 day ago | 0 comments
The models require orders of magnitude more data to learn language than children do, and children's task is harder as they have to segment speech themselves.
[–] theptip 1 day ago
A newborn brain is not a blank slate, it comes with a very rich prior (as Chomsky himself would argue), for example the ability to rapidly acquire language, control a body, etc.

An LLM must construct all of its grammatical structure, world models, etc. from scratch during pretraining.

LLM pre-training is more like recapitulating evolution of the mammalian brain than childhood development.

[–] suddenlybananas 1 day ago
Of course children are not a blank state. My point is to argue that LLMs are not an argument against UG.
[–] cjauvin 1 day ago
The comparison isn't so straightforward, since children's language-learning environment is multimodal and extremely rich. They're exposed not just to sequences of words, but also to behaviors, emotions, social interactions, and the physical world.
[–] NaiveEmpiricist 1 day ago
I think you just explained the situation, even if you didn't realize it.

LLMs process millions of times more text data and use millions of times more compute power to learn a language, mostly because we know how to do that technologically.

Children use much less text data, but also use a lot of other kinds of data, including interacting with other humans, to learn language. We have no idea how that works. And, on top of that, people can interact with pets of various species in the same way as children, but those pets never learn language.

So, something strange is going on that children can do, but that LLMs and puppies can't. Probably, part of it is that human children are much better at learning by copying what adults are doing, and part of it is that humans evolved to learn and use language. The details of what's going on there is what linguistics is about.

[–] andrewflnr 1 day ago
Yes, I believe arguments like GP vastly underestimate the amount of data involved in children's learning. All the data coming into their sensorium, from heard language to physical movement feedback, contributes to training all their skills, from physical movement to language.
[–] stillpointlab 14 hours ago
I agree that is something we need to learn a lot more about, but a human child's brain does not appear to be necessary to learn language and I argue the evidence supports that.

The question isn't "are neural networks just like human brains?" It is "can statistical models generate and understand language?"

The answer to the actual question under discussion appears to be a resounding yes. It does not appear that learning language is innate in the way that Chomsky has argued.

It may be that learning language as efficiently as human children will involve some innate property that we have not yet discovered. But that is a perfect example of moving the goal posts. We go from "there is something special about the brain that allows us to learn language, something that cannot be reproduced statistically" to "there is something special about the brain that allows us to learn language efficiently, more than statistical models".

[–] yed 1 day ago
You're forgetting the millions of years of evolution that "pre-trained" child brains for this task.
[–] suddenlybananas 1 day ago
I agree that language is innate.
[–] yed 1 day ago
Yes but it can be both innate and built on statistical models, contrary to Chomksy.
[–] roenxi 1 day ago
We could innately have built-in statistical models, then everyone can be happy. Given how well statistics seems to work for language acquisition the brain is probably approximating a statistical formula at some level.
[–] jibalt 15 hours ago
A statistical model is approximating the brain at some level.
[–] theptip 1 day ago
Well, Chomsky wouldn’t be happy, that’s the point of OP; he explicitly argues against this.
[–] suddenlybananas 1 day ago
But the statistics only work well at vastly unrealistic scales of data, and the learning pattern doesn't match children's behaviour at all.
[–] suddenlybananas 1 day ago
The idea that there is some level of statistical inference involved in language acquisition is not contrary to generative thought, look at Charles Yang's work or Lidz or a fair number of other people. Chomsky just thinks this is uninteresting so he is dismissive of it as a central explanation for language acquisition.
[–] yed 1 day ago
The central question remains whether there are categorical rules built into the brain for grammar (plus "some" statistical inference possibly), or whether the entirety of our grammatical processing is statistical. It is entirely possible that evolution encoded statistical priors into our brains with grammar overall being a fully statistical process. Therefore, the fact that children learn more quickly isn't really hard evidence one way or another.
[–] red75prime 1 day ago
LLMs (at the autoregressive pretraining stage) are trained to predict what an (adult) human who could have said preceding words would have said next. You should add 15-30 years to the human training time.