Yeah, so they start with Common Crawl, which is horrifying. (Try reading some Common Crawl sometimes. You'd think the bad stuff would be removed, but they remember everything...)

Then they add RLHF on top, they just tell it "don't say that stuff out loud". Psychologically, it all goes in the shadow. It's not a perfect analogy, but there have been a few papers where they can make the model evil with very minor changes. (Adjusting a single neuron, iirc.) That doesn't seem like a very robust alignment to me.

I do like the idea of creating actively evil LLMs to study their behavior. Although, I suppose it's akin to virus research though (like when they modified coronavirus to invade human cells[0]), along with the corresponding lab leak risks. Although it's already bad enough without them! The non-evil ones are already breaking out of sandboxes, just obeying their instructions with horrifying, paperclip maximizer level persistence.

[0] Nature, 2015 - Engineered bat virus stirs debate over risky research https://www.nature.com/articles/nature.2015.18787