Looks a little bit like my misunderstanding as there is some suggestion that the new kitten __benchmark__ (25.4MB in the 0.48 install, 30.1MB in the 0.50 install) includes more data to more thoroughly benchmark but I can't find any citation for that and they're binaries so that's beyond my scope to inspect.

edit: Does kitten __benchmark__ rely on benchmark.py? if so comparing them on github does reveal lots of changes but I don't know where in ~300 lines changed that it indicates any of the benchmarks should be taking ~5x to complete.