Hacker News
points by stingraycharles 3 days ago | 0 comments
So, what exactly can a base model do? Is this like the pre-reasoning models that just autocompleted, without any alignment, reasoning, etc optimizations?
[–] londons_explore 2 days ago
A large base model is ~90% of the effort/cost to making a frontier model, and by itself is not useful for much. There aren't many controversial decisions to make when training a base model - you mostly just train a model as big as possible on as much human-generated data as possible.

But it makes it far easier for other parties to do that last 10% to train it to be useful.

However, there are many choices one makes in the last 10% of training which will impact the models behaviour when it comes to AI safety, political opinions, willingness to help make a bomb, etc etc.

[–] stymaar 2 days ago
> A large base model is ~90% of the effort/cost to making a frontier model

This kind of figures is often quoted but where does that estimation comes from? Isn't that legacy number from the era when all the post training WS just instruction following and RLHF? I'd be surprised if current generation if reinforcement learning was still a few percent of the training cost.

[–] zozbot234 2 days ago
AIUI more recent estimates have about half of the total training compute, or more, being used for post-training/fine-tuning. Post training a base model (even for simple chat, let alone thinking/reasoning or performance on verifiable tasks) requires reinforcement learning, which is quite hard.
[–] stymaar 2 days ago
A base model is what you get just from next token prediction training. Think GPT-2/3 but bigger.

Withtout RLHF, instruction-following training, or RLVR to develop reasoning.

[–] otabdeveloper4 2 days ago
All LLMs are next-token predictors. The post-training is to get it to predict tokens from a structured, meaningful dialogue exchange and not from a random reddit and 4chan data dump.
[–] stymaar 2 days ago
Nah, that's only one part of the post training, and it has been so since OpenAI's o1 and Deepseek-R1 in fall 2024-winter 2025.

Now a good fraction of the post training (Most maybe at this point? Who knows?) is spent on RLVR (reinforcement learning on verifiable reward), in this kind of training you don't judge the token generated themselves, but the answer as a whole ("does this program compile and pass the tests", "is the proof valid"). That's where all of LLMs progress over the past two years in software and math come from.

So yes, they are still next token predictors at their core, but they stopped being mere stochastic parrots since they are now being trained on much more than just imitating their training data in a statistically plausible way.