Hacker News

points by saejox 11 hours ago | 0 comments

There is always a huge discrepancy between benchmarks and real world capabilities.

For my use case deepseek 4.1 flash is almost as dumb as haiku 5.5. Both are fast garbage producers.

I'm curious what quantization and version you're using. I found DeepSeek v4 to be practically useless when I was using frob/deepseek-v4-flash-0731 via Ollama, but later I switched to using unsloth/DeepSeek-V4.1-Flash-1017-GGUF, and I've been extremely happy with using it for coding tasks since then.

[–] efromvt 3 hours ago

deepseek does pretty well on my personal benchmarks, but I had a ton of problem with any openrouter providers; had to resort to always using it directly.

[–] rapind 8 hours ago

Different expectations maybe. In my experience, DSF 4.1 high/max is basically what Terra should be for coding, and is very acceptable (Terra is hot garbage, yes even the latest 6.1). Luna at xhigh+ stands out as a cheap implementor/reviewer of very very well defined tasks, but goes off the rails if you're not very specific, Sol 6.1 and Sonnet 5.5 at medium are basically powerhouses for implementation with a half decent plan (slightly prefer sonnet), and Opus 5.5 high is an excellent planner (and implementor). And Astra is amazing at burning money.

[–] jeffbee 11 hours ago

I was beginning to worry I had overlooked the inner beauty of DeepSeek. When I have tried it, it was totally useless. Never understood HN cheering for it.

[–] AlotOfReading 11 hours ago

Don't know what you're doing, but it's good enough for the reasonably sized tasks I give LLMs. It's also a much better price. My browser stopped rendering the UI frame recently because I messed up permissions on /usr/share/icons. Took DS 4.1 flash about 3 minutes and a few cents to diagnose it. Same harness with opus 5 took twice as long and several dollars.

Moreover, the months I've used anthropic/OAI exclusively my API token costs are in the hundreds (or sometimes thousands). With DS flash as the first agent I reach for, I rarely exceed $40/mo.

[–] jeffbee 11 hours ago

I meant for generative coding. I'm sure it can do a few things here and there.

[–] AlotOfReading 10 hours ago

I figured it was assumed I was including that? It's been fine as long as I break tasks up into reasonably sized chunks, which is fine because I think you should only be generating as much as you can comfortably review at a time.

[–] jameson 6 hours ago

I'm a heavy DS v4.1 user and been satisfied with its performance. I use it just like you described -- "... break tasks up into reasonably sized chunks...".

Every once in a while I come across comments that just hates DS. I wonder if it's a matter of how people use it or the prompt. I admit pre-v4.1 was pretty bad, hallucinating too much and not getting the abstractions right so I had to correct it often.

I also have $200 Claude subscription (now called "Ultra"). Fable gets abstractions right on highly complex problems (e.g. defining protocol that spans multiple services) that DS simply cannot do. However, DS is comparable to Opus 5.5, although less "intelligent", but DS excels at speed which is what I prefer than waiting minutes on Opus 5.5.

[–] andreypopp 10 hours ago

How did you use it? I was using it as "implementor" subagents orchestrated by opus 5.5 and it was pretty good and fast.

[–] lisplist 1 hour ago

I wasn’t a huge fan of DeepSeek until their very latest 4.1 flash model. The original 4.1 flash, v4 flash, v4 pro, all of it wasn’t very useful for me, but their latest release has been extremely useful.

[–] pixelesque 10 hours ago

I've had significant success using DS 4 Pro and now 4.1 flash for fairly complex coding tasks on C++, Rust and Python projects at home.

I have access to Claude Code at work (and use it quite a bit), so I feel I can honestly compare them.

You do have to plan things first a bit more with DS (than say Opus can (Opus will have more chance of one-shotting something, although I generally don't think that's a good idea), although it often over-engineers and tries to be too helpful in my experience) and be a bit more explicit at times, but I feel it's quite close to Sonnet in performance for implementing plans, debugging issues and explaining how codebases work.

[–] arjie 10 hours ago

This has always been a problem with the Internet. When the iPad was released, anti-Apple subreddits talked about what a garbage device it was. Wishful thinking dominates a lot of human content generators. Extracting information from humanslop is harder than avoiding it entirely.

[–] mtrovo 10 hours ago

Hard to say if you don't share what kind of project, workflow and prompt you're trying tbh. I've been driving it mainly to write a quite big golang and android app and most of the tasks are all handled fine by DeepSeek.

[–] howunfortunate 10 hours ago

Assume about 50% of internet writing has zero tether to reality and is written out of pure motivation to shape a preferred narrative (and usually the same one).

When you see this your worldview will instantly become much clearer.

[–] tyzoid 10 hours ago

Not everyone is operating with the same constraints. As for me, once the models got good enough (~opus 4.5 levels of intelligence) that they could solve most problems that I was dealing with, I stopped following the frontier intelligence trajectory for daily use and instead followed constant intelligence and decreasing cost. Nowadays I primarily use GPT 6.1 luna, and DeepSeek 4.1 flash.

I've also been very happy playing around with qwen 3.8 27b running locally for personal projects. It sometimes can't quite get there, and that's when I'll reach for GPT.

[–] sjbzbeiks 7 hours ago

Benchmarks they share are on the much older v4 of DSF. DSF V4.1 1017 is a big step up in benchmarks and is also actually twice as many parameters.

Anyway, I use DSF v4.1 on max as much as I use Opus 5.5 on xhigh, Opus is definitely a more versatile model, but I’m not getting any value using it to implement well defined tasks that DSF v4.1 does just as well at for a fraction of the cost.

You see where GLM-5.3 is in this test down but respectable, but this model benches below DSF 4.1. TLDR basically nothing in this benchmark invalidates the Deepseek that everyone is using today.

[–] embedding-shape 11 hours ago

> There is always a huge discrepancy between benchmarks and real world capabilities.

I mean of course, the real world is messy, the models are only good for what they've been trained on, and post-training frequently takes benchmark scores into account too, leading to this sort of "Great score, sucks in the real world" syndrome that seemingly most models are affected by.