points by rvz 1 day ago | 0 comments
If you were wondering:
> Up to 7.4 tokens/s.
It's not really "Flash" if it is 7 tokens per second.
Also don't bother with any GPU lower than an RTX 5080 or even a DGX Spark (it is a scam) if you are seriously considering running frontier local models.
[–] Ancapistani 1 day ago
Oh, I don't know - it could be useful for long-running and/or background tasks in some cases.
The problem is that this particular model is so cheap, compared to others. Optimizing to host it locally with relatively poor performance just doesn't make sense when yesterday I burned ~180M tokens for $2.20.
[–] cassianoleal 1 day ago
Surely local hosting is not so much about cost savings as it is about data ownership and risk management. e.g. can't leak a secret if you're not calling out.
Strata runs considerably faster than that on similar hardware, though this one doesn't seem to be a fork.