Hacker News

points by Dzugaru 4 days ago | 0 comments

There is no substitute for reading the code in many many cases. I spend more than 3k/mo on tokens. I find incorrect assumptions baked into the code (the ultimate spec) and missed by the agents and tests daily. This is more pronounced in data analysis and complex systems.

As a side effect having to read the code at least once inevitably makes it way shorter and usually much better.

[–] argee 4 days ago

That's right. And you have to read what the agent is doing as well. Once I said (Opus 5.5), given this list, find issues that are both critical and related to {X feature}.

Guess what: because I said "both", Opus decided I was looking to pick out exactly two of the issues. If I hadn't read that reasoning log, I would've remained oblivious and thought there were only two it could find.

[–] zahlman 4 days ago

> Guess what: because I said "both", Opus decided I was looking to pick out exactly two of the issues.

That seems like a misinterpretation of English that I'd expect from a much less capable model. Like, a complete lack of understanding of grammar. To have that meaning, you would have to have put "both" before "issues" instead.

[–] argee 3 days ago

It is, and it was surprising in an unsurprising sort of way. This only happened last week, I'm not "misremembering" or anything either. It was a long context session, though.

[–] nhecker 4 days ago

You get a reasoning trace from Opus 5.5?

[–] argee 4 days ago

Yes, I use the pi harness and an API key.

[–] MisterMunchkin 4 days ago

They don't give you the actual reasoning, they use a smaller model to produce a slop summary. (They don't want you seeing all the MITM stuff anthropic injects into the chain of thought)

[–] DrammBA 4 days ago

> Opus decided I was looking to pick out exactly two of the issues.

In this specific case, does it matter if you get the raw reasoning vs a "slop summary"? I bet both would tell you that Opus locked into two issues no matter how bad the slop summary is.

[–] epihelix 4 days ago

To be fair, that's a pebkac problem. You got exactly what you requested (and the language was unambiguous).

This is like writing `if (cond_x && cond_y) {}` and then blaming the compiler because it didn't realise that actually meant to write `if (cond_x || cond_y) {}`.

[–] Feathercrown 4 days ago

That's not what they said, they said it limited its output to 2 issues because it saw the "both" in the sentence, which is clearly an incorrect reading of their instructions.

[–] epihelix 4 days ago

Yes, I think I misread -- looks like the pebkac problem was at my desk instead :/

But I don't think their prompt as written would have limited the responses to two, unless there really were only two matching both conditionals. I've not seen an LLM misinterpret language that way before.

[–] retsibsi 4 days ago

You're right, but I do wonder if they're misremembering their prompt a bit. The AI's response would be much more likely to arise if the prompt was something closer to "find both issues that are critical and related to X". Based on my own experience, it would be a very surprising mistake for the AI to have made if the prompt really was as straightforward and precise as "find issues that are both critical and related to X". I know it's a bit annoying to question someone's account of their own experience, and obviously I'm not saying this couldn't have happened, but my guess would be that the actual prompt was at least somewhat ambiguous.

[–] echion 4 days ago

The language is not unambiguous; it's exactly ambiguous (you and your first reply demonstrate how hard it can be to see the other reading):

0. "find issues that are both critical and related to X"

...could be read as "issues that are X and are Y":

1. [valid, intended] find [all] issues that are[, at the same time,] 1) critical; and 2) related to X

...or as "issues, both of which issues are X and are Y":

2. [valid, your reading] find [an to-be-specified number of] issues that are, both [so I want two issues], possessing X and Y attributes.

[–] AlexCoventry 4 days ago

> I spend more than 3k/mo on tokens.

I'd be curious to learn how you're using them.

[–] prmph 4 days ago

Exactly, I see insidious errors made by Claude Opus daily, all the time. Errors that would be very hard to catch with tests, because usually they are architectural level issues.

[–] geocar 4 days ago

> I spend more than 3k/mo on tokens.

Dude you should buy tin. I can get you a banger box for $30k.