Hopefully there will be more releases of open BASE LLMs, i.e. with the weights that are the results of pre-training, because those are much more suitable for being trained for whatever you want, than when you try to retrain fully-trained models and you have to undo whatever was done to them that is inappropriate for you.

The BASE LLMs have already done the part that requires a lot of resources, i.e collecting a huge data set, then running training on a big number of GPUs with a lot of memory and fast interconnections, and thus compressing by several orders of magnitude the training data set into the LLM weights.

Starting from there, training can be finalized in any desired direction with much less resources.

There is another on-going HN thread about such a BASE model that has just been released, baidu/ERNIE, which seems quite good. Several other companies, e.g. Google, NVIDIA, DeepSeek, Alibaba, have released in the past such BASE models suitable for training your own LLM.