The entire AI sector been on the cusp of building “superintelligence” any day now since 2023. Yeah, we’ve seen dramatic improvements since not even those three years ago, but the “I” in AI is a misnomer; they are no more intelligent than any other probability engine. I’m even sure they can obtain primitive forms of RSI, given what we’ve seen of Claude improving smaller models via weight-space programming. The problem is that I don’t see how an LLM could improve itself if it doesn’t already know the thing it wants to know, and so an “RSI-capable superintelligence” would be restricted by the procurement of new training data to solve what it can’t currently solve, which naturally brings in issues of how it can independently create that data.

Yeah great point.