Hacker News
points by t0mas88 4 days ago | 0 comments
Agree, I run QFN on my AI Pro R9700 (32GB) using https://github.com/Niko1221/Strata It's even faster than on a Strix Halo. Typically I get around 90 tokens/sec with a small context, down to 70 tokens/sec when it's near the maximum context usage.
[–] mrheosuper 4 days ago
I run QFN I2 XS on my 3060ti + 48GB RAM for it. Get around 2x token with 200k Context window. Good enough for me.
[–] pxmpxm 4 days ago
Err strata is just a fork of llama cpp with fancier uis, right?
[–] MadameMinty 4 days ago
Strata manages MoE expert offloading completely differently, which for QFN means ludicrous speed.