As a fellow machine learning practitioner, I don't think the OP's issues are specific to machine learning as much as they are more generally about system design and context. Much of my job as a data scientist is gathering more context about what business value a model is actually trying to provide, and it's difficult if not impossible to encode all of that context into a text format that's digestible by LLMs.

For example, there's a model we have in production whose sensitivity has to be balanced based on which of our customers / partners is consuming its inference. It's a cat-and-mouse game where we are increasingly taking into account additional features to use in the model based on feedback directly from these customers and partners. Adjusting the model based on this feedback is much more art than science because there's really no "correct" adjustment to make, just an adjustment that produces inferences that convince our customers and partners that their concerns have been addressed.

Sure, we could encode every conversation that we've had both internally and across the internal<->external interface, both formal and informal, as text that an LLM could consume. But we find it much easier to just have a human (me, or one of my teammates) do the same thing and then guide an LLM to design a system that satisfies the requirements collected from those conversations.

--

That being said, I'm equally amazed at the quality of suggestions that LLMs sometimes come up with for improvements to systems as I am dismayed by the complete dead-ends it sometimes hits when it doesn't have a good solution in its knowledge base. It's not limited to just ML topics, but also very esoteric things that have very little documentation or even contradictory documentation!

So yes, you're right that if the frontier model providers aim their reinforcement learning laser beams at improving a specific topic they will probably succeed. But, no, I don't think there's a panacea.

It's very possible I'm wrong. I think that what we're seeing in maths is that RL is training LLMs to solved well-posed problems, especially when they can recruit existing ideas from across the literature. I don't want to discount that - it's clearly powerful as we've seen in maths. But in fuzzier research fields, the right question is often more important than the solution - I'm not convinced LLMs are there yet, and I don't think this problem is as amenable to RL as problem-solving.

This recent work which tried to get LLMs to reinvent methods they haven't seen before (given the same problem setting) also raised some of the same issues I've noticed in my own usage: https://epoch.ai/publications/innovationeval

Because "miss the forest for the trees" does not have a large and easily machine-verifiable dataset.