Hacker News

points by AlienRobot 4 days ago | 0 comments

To be fair, most people aren't writers, so the "average" writing the statistical model can produce is mediocre writing.

I don't think this is the issue. Most people are mediocre mathematicians, too, yet the companies have selected well enough for math literacy in training data (presumably). I think its a matter of training - they are often post trained by other agents (which read everything), human reviewers not paid enough to care (easily wowed by 10 paragraph length essay with correct punctuation), and deterministic scripts (checking for presence of a specific point, so agents are incentivised to write as much as possible to have more chances of hitting the unknown thing the script is looking for)

As far as I know labs try to train the other way, else we'd have monstrously long and useless outputs, but its hard to overcome these natural biases for length

[–] teiferer 4 days ago

I don't think that's a suitable mental model of how LLMs work. Their output is not just the statistical average of all the training data. If it was then the output would be quite useless.

[–] Loquebantur 4 days ago

An LLM trained on the entire internet incorporates essentially all (human-known) aspects of things.

Like a superposition of all kinds of writing styles, all perspectives on every topic and all kinds of ways to do things right and wrong.

In post-training (and of course already by sensibly curating the input data in the first place), people try to select useful submanifolds from that.

Ideally, you get a combination of all the extant strengths of people combined into one model. Self-improvement then leaves the sphere of human capability for good.