points by IanCal 3 days ago | 0 comments
21s on a single core for 1024 tokens, that’s ~50t/s to verify and a 6.5 meg download. I’m pretty sure you won’t hit that for a 400B model at 4 bits, but then maybe you would I don’t track that performance so much. You’d need an enormous amount of ram to handle it though.
Yes, the generation is slower but I think here the verification itself is more efficient than regenerating the output.
I think there is a place for both. Deterministic inference has other benefits, like "compressing" an output into input+model, as well as more esoteric applications like solving coordination problems. But if we get fast ZKPs for real frontier models, I'll be overjoyed!