Could having a model modifying its own weights in like this be a way to achieve learning for LLMs?

To achieve statefull behavior from an LLM regardless of context (and ignoring whether this is something you would ever want), would require changing the weights in some way. It could be an external supervisor system, but self-modification would have some cool implications, e.g. the potential for arbitrarily deep meta-cognition.