points by esafak 3 days ago | 0 comments
Not cost competitive. The performance difference is not commensurate with the price.
Model | Cached Input | Input (Miss) | Output
---------------|--------------|--------------|----------
Tencent Hy4 | $0.042 | $0.83 | $2.50
DeepSeek (peak)| $0.006 | $0.30 | $1.20
DeepSeek (off) | $0.003 | $0.15 | $0.60[–] ritz_labringue 3 days ago
[–] bscphil 3 days ago
The AA benchmarks seem to be mostly "complete this small intelligence-proving task from scratch". That consistently puts models with high intelligence to output tokens on the frontier, especially the OpenAI models, on the frontier. But a lot of software engineering work is many-turn work on established code bases. There's a lot of reading and a relatively large context unless it's carefully controlled. Writing only a few output tokens doesn't really cut costs because most of the spend is read tokens that get recycled through dozens of turns. That's basically the ideal case for DeepSeek because of how cheap cached tokens are.
[–] joshheitzman 3 days ago
It highly depends on the task. So far, on the tasks I personally benchmark, DS v4.1 Flash @ deepinfra.com is always cheaper because their cached input price for it is so very low and their cache hit rate is so very high.
[–] nextaccountic 2 days ago
do you generally favor deepinfra or you are mentioning them because it's 50% off on openrouter?
I really want to see some comparison on deepseek 4.1 flash 1017 providers, like, what's more trustworthy to not quantize without disclosing it, etc
also I can't find the official DeepSeek provider on openrouter https://openrouter.ai/deepseek/deepseek-v4.1-flash-1017
[–] joshheitzman 22 hours ago
I use deepinfra directly. They have had a deep discount listed on their site for as long as I've been using them for that model and for v4.0 before that. I've used the DS v4.x Flash model from a number of provides and the inference quality at deepinfra seems to be as good as anyones.
[–] gregwebs 3 days ago
That analysis is unfortunately running DeepSeek on Max. DeepSeek recommends against that since it guarantees higher cost and rarely improves the outcome- it should be ran on "high".
[–] jambutters 3 days ago
where does deepseek say they recommend against high?
[–] bscphil 3 days ago
They have "high" on there now too, just search for it.
[–] gregwebs 2 days ago
Found it on that page, but they don't have it on their coding page.
Their coding page shows DeepSeek Max as Pareto optimal: https://artificialanalysis.ai/agents/coding-agents Costing 1/2 of Terra Xhigh. It would also be great to see Sol and Opus not on max there.
Not a useful benchmarking site for coding unless you only want to run your models on Max.
[–] Rodmine 2 days ago
This benchmark is total bs and does not represent real world usage. None of the models that “beat” Deepseek in the benchmark are any good.
[–] ActionHank 3 days ago
Wild that this is the first question I ask at this point. After that it’s a question of how it stacks intelligence wise vs glm or ds.
OAI / Ant are in for a reckoning.
https://artificialanalysis.ai/#intelligence-efficiency-tabs