I was thinking along these lines a couple of weeks ago[0] - specifically whether we could treat weight manipulation as a sort of programming/compilation task. I expected it to require a lot more training with synthetic datasets. It's surprising that existing models are already somewhat decent at it.

[0] https://x.com/o_lalonde/status/2100389409564643518