The AA benchmarks seem to be mostly "complete this small intelligence-proving task from scratch". That consistently puts models with high intelligence to output tokens on the frontier, especially the OpenAI models, on the frontier. But a lot of software engineering work is many-turn work on established code bases. There's a lot of reading and a relatively large context unless it's carefully controlled. Writing only a few output tokens doesn't really cut costs because most of the spend is read tokens that get recycled through dozens of turns. That's basically the ideal case for DeepSeek because of how cheap cached tokens are.

It highly depends on the task. So far, on the tasks I personally benchmark, DS v4.1 Flash @ deepinfra.com is always cheaper because their cached input price for it is so very low and their cache hit rate is so very high.

That analysis is unfortunately running DeepSeek on Max. DeepSeek recommends against that since it guarantees higher cost and rarely improves the outcome- it should be ran on "high".

This benchmark is total bs and does not represent real world usage. None of the models that “beat” Deepseek in the benchmark are any good.