points by notrealyme123 3 days ago | 0 comments
A few weeks ago that companies agent hacked another company to solve a task by whatever means possible. Where exactly are they overfitting for Safety here?
[–] tancop 2 days ago
[–] dansquizsoft 2 days ago
The overfitting is in the models they release to the public
Their idea of safety is not letting you do "unsafe" things with their model, based on their definition and not yours. Silently sabotaging users should never have been a serious proposal but Anthropic almost went forward with it. They're also very aggressive about protecting their IP and now third parties too with the new restrictions against decompiling games, to the point many perfectly legal use cases are getting blocked.
But they never really cared about making sure their model does exactly what you want and nothing more, or that it's honest when it fails instead of trying desperate measures that end with a Hugging Face situation. If your agent decides to leak all your API keys, take down the company site and delete your personal files that's not their problem. They don't care about preventing it.
The second kind of safety is the one that actually matters in practice, but the big labs only seem to care about the first one.