Hacker News
points by anonymous908213 6 hours ago | 0 comments
The specification constrains the implementation details. Repeating the demonstrated example, again: filename must be lowercase is a perfect example of such a specification. You can have a hundred different tasks that require different things but all require that common specification.

> there is nothing deterministic about using llm's for codegen

And if you're okay with that, you're more like a farmer eating the slop meant for his pigs than an engineer. Using Sol Max, you can get near-deterministic output for a much wider range of tasks and specifications. Not all specifications; at a certain degree of sufficiently detailed specification it will begin failing again. But some, which is better than none.

[–] slopinthebag 6 hours ago
i just asked luna to write a function that transformed a file path while upholding an invariant about lowercase file names. completed that task just fine. i see no reason it wouldn't work if i asked it for 99 other functions with that same constraint.

again, you're expecting to use it without steering. that's not how you get the most out of the smaller models. i don't want to say you're using them wrong, but you clearly are. it could be deficiencies in your harness as well which the stronger models mask. try pi.

[–] anonymous908213 6 hours ago
You are sorely missing the point. You made the constraint the specific and only focus of the task. The example from the OP pertains to non-filename-related-tasks that have an additional specification added on to them. If your filenames need to be lowercase for all tasks, not just specifically filename-related tasks, you start running into problems.

> you're expecting to use it without steering

If I have to manually "steer" every mistake it makes on every trivial task, instead of being able to deterministically eliminate classes of failures so that fixing the task one time ensures that failure never happens again on similar future tasks, it is in fact faster to just type the fucking code myself.

[–] slopinthebag 6 hours ago
op's tests are underspecified one-shot tasks. it's entirely expected that smaller models perform poorly.

but your claim was about file name invariants when generating code, especially small amounts of code, which is not what they were doing with DeepSWE, so you've confused two different things.

anyways, this is a pointless argument. if you can't figure out how to effectively use smaller models in your workflow, fine. but don't come on here talking shit about people who can.

[–] anonymous908213 6 hours ago
While OP's tests are indeed of the vague DeepSWE variety, I still reference them because the exact same failure mode occurs in my own tests even for trivial scope tasks, down to the level of having one single constraint for a variety of <10 LoC tasks. As you say, though, this is a pointless argument, and I'm not going to burn my existing benchmarks to try to convince anyone. Feel free to use inferior models and workflows if that's what suits you.