As a fellow machine learning practitioner, I don't think the OP's issues are specific to machine learning as much as they are more generally about system design and context. Much of my job as a data scientist is gathering more context about what business value a model is actually trying to provide, and it's difficult if not impossible to encode all of that context into a text format that's digestible by LLMs.

For example, there's a model we have in production whose sensitivity has to be balanced based on which of our customers / partners is consuming its inference. It's a cat-and-mouse game where we are increasingly taking into account additional features to use in the model based on feedback directly from these customers and partners. Adjusting the model based on this feedback is much more art than science because there's really no "correct" adjustment to make, just an adjustment that produces inferences that convince our customers and partners that their concerns have been addressed.

Sure, we could encode every conversation that we've had both internally and across the internal<->external interface, both formal and informal, as text that an LLM could consume. But we find it much easier to just have a human (me, or one of my teammates) do the same thing and then guide an LLM to design a system that satisfies the requirements collected from those conversations.

--

That being said, I'm equally amazed at the quality of suggestions that LLMs sometimes come up with for improvements to systems as I am dismayed by the complete dead-ends it sometimes hits when it doesn't have a good solution in its knowledge base. It's not limited to just ML topics, but also very esoteric things that have very little documentation or even contradictory documentation!

So yes, you're right that if the frontier model providers aim their reinforcement learning laser beams at improving a specific topic they will probably succeed. But, no, I don't think there's a panacea.