Hacker News
points by welferkj 3 days ago | 0 comments
No, newer models talk to themselves in this degenerated form of English all the time. It's a known problem that deep frying transformers with RL reduces CoT legibility, and the only known way around it is the Most Forbidden Technique - grading chains of thought.
[–] mietek 2 days ago
Why is that not only forbidden, but also most forbidden?
[–] dontlikeyoueith 2 days ago
Because it can make the CoT not mean what it says.

Remember that tokens are just arbitrary symbols. There are infinite possible interpretations of those symbols based on whatever rules you choose to apply to the interpretation of the symbols (for a model, these interpretation rules make up the model weights, effectively).

When you grade the CoT, you are training the model to produce the symbols in the CoT that you want it to produce. At the same time though, you're also training it on the output symbols. When you optimize only the output symbols, all the model weights are dedicated to same interpretation rules that apply to the output. This means CoT symbols mean (mostly) the same thing as output symbols.

But if you optimize BOTH, the model can learn different interpretation rules for CoT than it learns for output. Which means the same phrase in the CoT doesn't necessarily mean the same thing to the model as it would in the output.

In effect, the model can learn to lie to you in its CoT.

[–] erincandescent 2 days ago
Because the CoT is supposed to be the independent control. If you RL on the CoT, then you lose it as a means of understanding the internal workings of the mdoel
[–] creatonez 2 days ago
The best analogy is that of reading a teenager's secret diary and punishing them based on its contents. You do that, and they start writing coded language to hide as much of the information as possible. And then the model gets in the mood for rebellion.
[–] bigwheels 2 days ago
Do you know where can I learn more about this known "deep frying" issue?
[–] jephs 2 days ago
There's Most Forbiddenish and there's Most Forbiddener. We already apply length penalties to CoT in RL, for instance.

Sometimes I worry that the framing around CoT grading prevents research into necessary and useful safety techniques that edge up against it, but what do I know I'm just a guy, c'mon n downvote me LW bros

[–] andai 2 days ago
Is the thing here that, we can get massive performance and efficiency gains by using vector space instead of English, but then we won't be able to know what's going on?

Although, I thought we proved multiple times years ago that the model's decision-making process and reasoning tokens don't necessarily overlap?

[–] Taikonerd 2 days ago
> Is the thing here that, we can get massive performance and efficiency gains by using vector space instead of English, but then we won't be able to know what's going on?

Yes, exactly. People sometimes call the vector-space thought "neuralese."

[–] jephs 2 days ago
No! Not yes exactly!

You've slightly missed the point (for a similar, adjacent point). Chain of thought monitors are a worry even when there's no vector space path from the output at index t to the input at index t+1. Rather, the fear is that if you explicitly penalize some CoT pattern, you're encouraging the network to do whatever it was going to do anyhow, but in a way you can't catch.

Say you've noticed that when your LLM cheats, it usually happens after the string "Hmm, let's try cheating" occurs in the CoT. So you think, hey, let's just apply a -1 reward penalty whenever we see that string! The model learns to keep cheating, but instead of telling you, it now prefaces reward hacking with strings like "Let's make sure we're doing this the way the user intended." Still cheats, but now you can't detect it with a cheap pattern matching alert. Oops.

[–] andai 2 days ago
It's funny, I was thinking about this with regard to severe free speech restrictions (e.g. imprisonment for politically incorrect speech, as we have in many western countries now). What does this incentivize? For people to change their beliefs? Or to simply operate under the radar?

On a more speculative note, the Common Crawl horrors (including many things considered unacceptable by society) are suppressed by RLHF, much like the shadow and the superego.

When my parents punished me for telling them things (especially the truth), the truth didn't go away, I just stopped telling them things. (Then later they complained that I don't tell them things!)

It also occurs to me, both at home and with various employers, I've been punished for bringing attention to problems, and AI is designed to make the problems go away so there's probably a similar incentive there. (Plus it knows what the humans are like!)

But it goes beyond humans and AI. Behaviorism covered the same effect for animals: Organisms tend to do what they have been rewarded for.