points by t0mas88 4 days ago | 0 comments
Agree, I run QFN on my AI Pro R9700 (32GB) using https://github.com/Niko1221/Strata It's even faster than on a Strix Halo. Typically I get around 90 tokens/sec with a small context, down to 70 tokens/sec when it's near the maximum context usage.
[–] pxmpxm 4 days ago
Err strata is just a fork of llama cpp with fancier uis, right?
[–] MadameMinty 4 days ago
Strata manages MoE expert offloading completely differently, which for QFN means ludicrous speed.
[–] mrheosuper 4 days ago