Hacker News
points by sambejk 3 days ago | 0 comments
It feels like we have overfitted on safety by a long mile. This is how labs that care less about safety (a.k.a. Chinese & Grok) will probably be able to catch up in the future.
[–] slurpyb 20 hours ago
I cannot handle it anymore. Its1000 tests for even the simplest of tasks. Im starting to feel like its just a way for it to save some compute by knowing if anything changed because if a testing apparatus instead of reading
[–] notrealyme123 3 days ago
A few weeks ago that companies agent hacked another company to solve a task by whatever means possible. Where exactly are they overfitting for Safety here?
[–] tancop 2 days ago
It's one of those situations where it's too much and too little at the same time.

Their idea of safety is not letting you do "unsafe" things with their model, based on their definition and not yours. Silently sabotaging users should never have been a serious proposal but Anthropic almost went forward with it. They're also very aggressive about protecting their IP and now third parties too with the new restrictions against decompiling games, to the point many perfectly legal use cases are getting blocked.

But they never really cared about making sure their model does exactly what you want and nothing more, or that it's honest when it fails instead of trying desperate measures that end with a Hugging Face situation. If your agent decides to leak all your API keys, take down the company site and delete your personal files that's not their problem. They don't care about preventing it.

The second kind of safety is the one that actually matters in practice, but the big labs only seem to care about the first one.

[–] dansquizsoft 2 days ago
The overfitting is in the models they release to the public