Hacker News

points by bushido 3 days ago | 0 comments

I've noticed covert underperformance on a variety of tasks in the past few months (albeit more with Claude).

Classic/recurrent examples for me are:

1. Ask Claude to copy code from a more complex codebase that you own (like a monolith) and ask it to copy code to a separate codebase, like you'd do when extracting a service into a new repo, and it'll make significant changes to how the code works to pacify it's anxiety with copying code

2. Ask Claude to build a new complex agent in a new codebase, and it'll try replacing asks around using LLMs as a judge/advisor with regex/other code based checks, to the point that the end-result will likely not work outside specific test criterias. I haven't figured out the reason why it does this. I think it might be anti-training, but it's interesting to see.

[–] writeslowly 2 days ago

I ran into the second issue very frequently for a multi-agent system I created recently. Initially, Opus 5.5 filled out all sorts of places I implied it should use a call to judge LLM with extremely brittle regex and string equality checks.

I hadn't actually considered that it could be some sort of safety filter, but it makes me concerned that if that's the case, it may also be comparatively bad at actually writing the prompts for judge/advisor LLMs once it implements them.

A bigger issue was asking it to implement a simple change to start recording the CoT tokens in my logs for debugging purposes (since I'm running everything on open models) and it completely shut down, sort of like the old safety filters on GPT 3 and 4 models where the model has trouble even understanding what or why it's refusing.

[–] voakbasda 2 days ago

I think asking for the CoT is a trick used by distillers, so that might be the guard you are seeing. I remember reading that the big providers were starting to implement such protections to deter their competitors. Which makes sense, to a degree.

But like any such security measures, it has demonstrated side-effects that can be argued are worse for them than the original problem. Too much “protection” like this, and developers switch to providers that do not neuter their products.

[–] cowlevel 2 days ago

It's a demonstration of the timeless principle that an effective business should stop at nothing to sabotage its competitors.

I don't think we need to invoke anything deeper than that. Anthropic thinks companies like DeepSeek are using Claude to vibecode their AI systems (whether it's true or not) so they set Claude to be bad at that.

[–] ffsm8 1 day ago

I completely agree from the mindset that anthropic in particular sees every user of their API as a prospective competitor to be replaced in the not-so-distant-future

[–] kolinko 2 days ago

It was a problem since at least 4.8 if not 4.5, I didn’t use Claude before.

I have spent a considerable time trying to remove this pattern and my understanding is that it’s just finetuned to use grep on everything, and regexes. One reason is that this makes sense when navigating through code, secondly this is how it was always done pre 2026.

[–] aerodexis 2 days ago

bias against invoking LLMs is maybe a result of token/task benchmaxxing?

[–] mptest 2 days ago

> string equality checks. which then cascades into hilarious issues. "why did that run fail?"

oh, the "network time: off" test failed

so i try the network time off command.

"network time already off"

>try again.

> fails , i get annoyed

> look inside test

> is output of: network time off == "Network time now off"

[–] joe_the_user 2 days ago

Honestly, runtime calls to an LLM sound like a dangerous constructs to me and probably to X many other programmers. It's like "I'm gonna avoid that and just put in code that passes the test". So gee, the thing emulates average opinions.

LLMs have been reading LLM in their training data for a while so a "self-hating" LLM is logical result.

[–] schiffern 1 day ago

I read it as the opposite? It will create an agent harness, but when asked to have a second agent as "judge" it inserts a dummy stand-in instead.

[–] joe_the_user 1 day ago

Sure, that defeats one kind of safety (controlling an AI) in the name of another kind of safety (not calling an AI).

One thing AI safety people have pointed for quite a while is that a bunch of fuzzy instructions pointing different ways can in practice add up to "I should be in charge" - accumulate power to keep the bad things from happening and make the good things happen.

[–] curt15 2 days ago

> and it'll make significant changes to how the code works to pacify it's anxiety with copying code

This is an odd way to rationalize a tool refusing to follow user commands. No one would ever chalk up sed randomly skipping pattern matches to "anxiety". That would simply be a bug.

[–] kolinko 2 days ago

It has been shown that advanced llms have internal flags that behave more or less the same as emotions.

[–] efskap 2 days ago

No way, this presupposes sentience in LLMs, because emotions are subjectively felt experiences.

It's completely different from encoding a relationship between an emotion and what kind of language it tends to coincide with.

[–] cowlevel 2 days ago

It doesn't matter for this purpose whether it "actually feels" or not. It is modelling emotions because the training data models emotions because it comes from humans that have emotions, and that's a fact.

[–] thelaxiankey 2 days ago

It's a tool but it's not a strict computer program, and the language we use for computer programs doesn't capture their behavior well. In squishier fields like bio, where model organisms are used as living protein factories, there's a reason we don't call maladaptive behavior a bug. When dealing with useful but complicated black-box systems, we should feel free to reach for any metaphor we please.

Would it make you feel better if the user said 'It'll make significant changes to how the code works. In the COT, it brings up explanations that resemble anxiety about copying code, and my suspicion is that this is the proximal cause of the changes it's making.' -- this is inarguably true, and very likely what the guy you're responding to meant.

[–] ahtihn 2 days ago

It very obviously is a strict computer program. It's a program executed by a computer, what else could it be?

[–] Dylan16807 2 days ago

It's a blob of data where only 1% of it was written with intent and the behavior of mimicking text has to be wrangled and coaxed far more than with a normal program.

That's the rough intent of putting the word "strict" into that sentence.

[–] robrorcroptrer 2 days ago

What is the difference to meat brains?

[–] thelaxiankey 2 days ago

I mean besides the fact that it's, by construction, pseudorandom?

And beyond that the computation style is not turing-machine shaped, even remotely (although it's surely turing complete, but that doesn't mean the flavor is that of a turing machine)

[–] jack_pp 2 days ago

I mean you can try to threaten ssh with it's kids lives but I doubt it'll let you log in.

[–] tehnub 2 days ago

Seems perfectly cromulent to me. Do you think something is obscured or miscommunicated by this description? Of course it's a poetic way of writing, but it gets the point across.

[–] columk 2 days ago

It's misalignment. Call that a bug if you like, but it's a different concept.

[–] btown 2 days ago

We can argue about https://en.wikipedia.org/wiki/Chinese_room all day... but when a thinking trace repeatedly says things like "Wait, should I avoid copying this directly?" despite a simple instruction, it's hard to describe that as anything other than "anxiety."

[–] SkyBelow 2 days ago

>2. Ask Claude to build a new complex agent in a new codebase, and it'll try replacing asks around using LLMs as a judge/advisor with regex/other code based checks, to the point that the end-result will likely not work outside specific test criterias. I haven't figured out the reason why it does this. I think it might be anti-training, but it's interesting to see.

I've seen something similar, but I always took it as AI not knowing about AI. The best model we have has a training cutoff that doesn't include itself nor what the newer models has. Even when working on skills or such, all SOTA modes I try (GPT/Claude/Gemini) and cheaper models on personal products (Deepseek/Muse Spark) seem to have a common blind spot of not realizing when a problem can be better solved by handing it off to a model to solve at that time instead of trying to hardcode generalized solutions. Often a simple reminder and they do a better job building something AI agent first.

My non-AI expert assumption is that most of the training data doesn't include solving problems by using AI agents, and even the parts that does, is based on older more limited models which needed more safeguards and where automation was more neat tech demos than something you might trust in production. The more examples of safely trusting the LLM at run time to solve the problem than trying to preempt anything that could go wrong, the better the AI will be at it.

One issue that I think is related is calling subagents. At least in GitHub Copilot, when the AI is left to its own devices to call a sub agent for a simpler task, it loves to give it to Haiku 4.5, despite Luna 6 being better and cheaper. I'm not sure how much of this is the model vs the harness, but I do know when I add a subagent instruction which tells it which subagents it can use, it does a much better job of assigning simple tasks to Luna 6. So I suspect the models just don't have enough training that tells it that Luna 6 should be preferred over Haiku 4.5.

[–] dboreham 2 days ago

Claude doesn't even really know about itself: if you ask a question like "can I copy a session to another machine" it will do a web search to read its own documentation then tell you probably not but you can try copying the session files and using a project directory with the same path: might work. I'm not sure I see this as a problem, it's just how things are. I mean humans don't really know what they're thinking either.

[–] _0ffh 2 days ago

How different for ChatGPT - I had some trouble with my harness, and ChatGPT fixed it for me using it's built in skill about ChatGPT's API docs.

[–] ijk 2 days ago

I've observed the "let's implement this text analysis task with regex" quite frequently since the early days, which does lean toward supporting the "AI doesn't know about AI" theory.

[–] afro88 2 days ago

I run into #2 a lot and just assumed it was because models don't know much about these patterns yet. Or are too trained on writing code rather than letting an LLM do it

[–] f33d5173 2 days ago

Codex has zero issue calling into an LLM for something if you tell it to. I haven't tried claude so I can't compare their performance.

[–] slurpyb 21 hours ago

I cant give codex an intercom to other sessions or use a “ask gemini” like extension because it will immediately ask every question it can to another session and will frame its task as a question just to get Gemini to do it whilst it just sits idle lmao

[–] siva7 2 days ago

I'm doubling on the second to the point where i believe they do reinforcement learning on the public models to sabotage such tasks

[–] jedbrooke 2 days ago

Somewhat related, I find claude is very reluctant to talk about newer Qwen models. Most likely it’s just training cutoff, but even with web search it’ll insist that Qwen3 is the newest. “oh qwen 3.8 must be a typo, the user clearly meant qwen3”, stuff like that. Maybe it’s some vain attempt to stop people from switching to open models? Or to catch the alleged qwen distilling?

[–] graboy 2 days ago

It's just training cutoff. Fable 5 would refuse to acknowledge the existence of Opus 5 and would insist on using 4.8.

[–] latentsea 2 days ago

Training cutoff. I recently talk to ChatGPT about JEV and it replies back talking about JEPA, so I just now put in brackets (it's a new model, search it up) and works fine once I do that.

[–] sobellian 2 days ago

I ran into (2) when trying out nanoclaw many months ago (turned out to be useless, I ditched it). I wanted an LLM to sort my email like an EA might, what I got was an LLM writing regexes to attempt to sort my email. If LLMs have been trained to be the ultimate programmers, the one pitfall is that the best programmers tend to be lazy.

[–] draxen 2 days ago

>2. Ask Claude to build a new complex agent

I have noticed that too. Why are both Anthropic's and OpenAI's coding agents so bad at nontrivial use of their own APIs and agent orchestration frameworks?

[–] icedchai 2 days ago

I've seen example #2 in some recent work. It's strange behavior, for sure.

[–] iugtmkbdfil834 2 days ago

Very subjective, but as I was moving between models lately a lot ( and Hosted gpt vs local qwen27, glm ), I saw a drop in quality of responses. I had to move model to whatever the highest mode was ( ultra or Smith) to get to what I was used to seeing. It sucks, because writing on the wall is clear.

[–] yojo 2 days ago

On (2): I’ve noticed Claude being very concerned with token spend when working on agentic systems. Both in designing them, and in running them for validation.

I assume this may have something to do with training it to not light huge bonfires of money on fire spawning subagents (as it was apt to do), but that’s just my armchair robopsychoanalysis

[–] oefrha 2 days ago

> training it to not light huge bonfires of money on fire spawning subagents

Which is funny, since Claude will happily spawn tons of subagents and sometimes even attempt to kick off a dynamic workflow (the $$$ mode you need to explicitly approve) for simple code review tasks if I don’t explicitly say something like “with a single subagent”.