That's slower than I would have expected for the RTX 6000, do you have figures for say Qwen 3.8 27B? That model fits completely on the card, that should push it a lot further.
I'm pretty certain I can get it to go faster by caching experts and whatnot (with experimental llama.cpp branches), but that would consume a lot more VRAM and it's fast enough as it is for playing around with, and I can leave it always loaded. It's like 50% CPU inference, with a few layers + router + kv cache on the GPU.
[–] barrkel 2 days ago
[–] jacquesm 2 days ago