Hacker News
points by gregwebs 2 days ago | 0 comments
These general intelligence/capability benchmarks don't apply to your specific task. If you are coding, you want just benchmarks for coding. I don't see performance+price at this benchmark for coding. When you look at just coding, DeepSeek is Pareto optimal for a good portion of coding benchmarks. For example here, although even this is very flawed because it only benchmarks max/xhigh effort: https://artificialanalysis.ai/agents/coding-agents

Closed models might be better generalists because they don't have that much constraint on size and come from well funded labs that can afford to better train for generalization. If a model's actual goal is to be open then smaller is better.

[–] CyLith 2 days ago
I see this take frequently; that coding does not require general intelligence. Perhaps I am in the minority, but rarely do I find myself in coding situations that are completely divorced from other subject areas. The code is modeling or interacting with the world, either through business processes, or (in my case) simulating the physical world, and I need the agent to have some working knowledge of not just the code, but also physics, materials, or social conventions. Are you all really able to just "write code" without dealing with all the externalities?
[–] true_religion 2 days ago
The externalities have to be dealt with using human judgment. Ai can help organize data to make better judgements but that use case is taken by chat based AI.

By the time, we want to code, the scope has been limited pretty greatly to the point that my rule to the Ai is “don’t make any decisions outside of the plan, if something in the plan turns out to be untrue or cannot be implemented after 2 differing attempts stop and ask for clarification”.

Sub frontier level Ai can circle endlessly thanks to a bad assumption, but a human can cut through that by coming at the problem sideways.

[–] gregwebs 2 days ago
Its a good point that there is benefit to using general intelligence models to figure out what code to write and to write specs. Implementation plans as well as long as the general intelligence is also great at coding. But you would actually prefer here the model that performs best in your business domain rather than best at general intelligence.

The Opus frontier model will spin up a cheaper Sonnet subagent to explore the code base. Similarly we can have a Pareto optimal subagent write most code and verify it.

[–] simion314 2 days ago
How often you needed the coding AI know all the trivia about some metal band? Like who are the members, songs , years etc. You can't even be sure if the facts are correct . This are good for a generalist model where you say remember some scene from music video and the AI will find the video for you.

Sure some world knowledge like things are affected by gravity, how people function, how a society function I think is needed for a coding AI but not trivia.

[–] mik1998 2 days ago
Are you approaching it like programming? I first design the program I want it to write, and then let the LLM implement it. All the important decisions it has to make are contained in the design. The LLM only has to know programming and a little bit of algorithms to implement it.
[–] Perenti 2 days ago
How often do the cousins of Louis the 5th of France impact on coding decisions? I can't see the argument for a completely generalist model being best for _coding_ in all situations.
[–] dofm 2 days ago
They probably don't. But some words acquire their full sense meaning from historical and-non-programming context.

Words that are or were fairly common in tech circles — "bailiwick", "gatekeeper", "daemon", "boundary", "infer", "method".

It's true to say that most of these words have some specific meanings in tech, but you don't necessarily want the LLM to have only those narrow meanings when you are talking to it. A broader grasp of the word means the LLM won't go down a specific pathway before understanding you by analogy. That broader grasp comes from non-programming understanding.

[–] samtp 2 days ago
Yes, 100% agree. Without more details on how they got their data, it's very hard to understand why they don't value DS Flash 4.1 as much as pretty much every other analysis I've seen (and my own experience).
[–] dofm 2 days ago
They do, to the extent that general intelligence training is what gives you the ability to talk in broad natural language to a coding agent.

Words only have their meanings through general context.