The problem is the "if". There is no guarantee that there's a "better, more efficient approach" to be found.

So far, the attempts to replace the transformer LLM with something else have yielded either "the same transformer LLM but with this one tweak that makes it 10% better" or "a radically different architecture that's a bitch to work with, and performs, best case, about the same as the transformer LLM".

By now, I'm firmly in the camp of "the architecture is good enough". Rifle is fine. Transformer is fine. Get better at the transformer before you look at anything else.