points by jimbokun 2 days ago | 0 comments
Except it’s very difficult for home users to afford the GPUs needed to run them with acceptable performance.
[–] layoric 19 hours ago
[–] notenchange 2 days ago
I have real world data here.
On a 5060 Ti (16GB) with 64GB RAM, I’m getting ~65 tok/s on decoding and ~450 tok/s prefil for Qwen 3.8 Flash Next. 256K context.
This is totally suitable (speed wise) for coding.
This is not some top of the line hardware (might be now, but it should be run of the mill for people that built their system <1.5 years ago).
[–] ozgrakkurt 2 days ago
How do you even fit that model in there? What quant is it? Are you using llama.cpp?
[–] notenchange 2 days ago
No, I'm using strata. This is a MoE model, so it keeps the last used model in VRAM, and the rest in RAM. If it needs, it swaps the less used model in VRAM for the model in RAM for the next token.
Seems very slow, but it's actually fine.
[–] ozgrakkurt 1 day ago
Thanks, the information in their docs are really promising. I have been meaning to try it but wasn't able to yet.
[–] notenchange 2 days ago
P.S. Sorry, where you read “models” I meant experts.
Yes, hardware has gotten insanely expensive, but software is moving so quickly and a lot of effort is pointed at eking out every bit of performance, and we are seeing a lot of success. Really looking forward to Qwen 4+. Yes Opus 5.5 is better than what I can use at home, but the way I use these models, I don't feel I need that. I prefer to be in the drivers seat, and these delegate the simple tedious parts to models which they are getting really good at.