points by anonymous908213 8 hours ago | 0 comments
You are sorely missing the point. You made the constraint the specific and only focus of the task. The example from the OP pertains to non-filename-related-tasks that have an additional specification added on to them. If your filenames need to be lowercase for all tasks, not just specifically filename-related tasks, you start running into problems.
> you're expecting to use it without steering
If I have to manually "steer" every mistake it makes on every trivial task, instead of being able to deterministically eliminate classes of failures so that fixing the task one time ensures that failure never happens again on similar future tasks, it is in fact faster to just type the fucking code myself.
[–] slopinthebag 8 hours ago
[–] anonymous908213 8 hours ago
While OP's tests are indeed of the vague DeepSWE variety, I still reference them because the exact same failure mode occurs in my own tests even for trivial scope tasks, down to the level of having one single constraint for a variety of <10 LoC tasks. As you say, though, this is a pointless argument, and I'm not going to burn my existing benchmarks to try to convince anyone. Feel free to use inferior models and workflows if that's what suits you.
but your claim was about file name invariants when generating code, especially small amounts of code, which is not what they were doing with DeepSWE, so you've confused two different things.
anyways, this is a pointless argument. if you can't figure out how to effectively use smaller models in your workflow, fine. but don't come on here talking shit about people who can.