Hacker News
points by sojournerc 2 days ago | 0 comments
We used to engineer systems that worked deterministically. There was no "oops I leaked all your pii, but totally cool, we were just vibing over here". Own your bugs. This feels... Not like an oversight, more just embarrassment of being caught

You're sorry it happened, but like domestic abuse, I doubt the underlying behavior will change.

[–] smashed 2 days ago
It's the nature of the beast and all my research shows it can't be solved fully. Just like human social engineering.

The only 100% safe solution is to not reveal the private info in the context. That's exactly what anthropic is doing here: removing the logged in user email from the system prompt.

Make no mistake though, that's a weak fix. The email is usually just a "git log" away, and for many tasks it finds its way into the context quickly.

Internet access must be blocked for agents dealing with private information. This makes agents vastly more cumbersome to develop or much less useful.

So the frontier labs pushed the idea of guard rails and harnesses as the solution. It sort of works, most of the time, and it lets users yolo all their secret information and get things done fast.

[–] tpmoney 1 day ago
Oh come on, data leaks (even unintentional ones) have been a part of our collective industry since about the time the first two computers were connected together. Whole industries exist to put together profiles of you based on data that deterministic applications that you installed and choose to use send with or without your knowledge all day long.
[–] sojournerc 13 hours ago
Sure, but companies would typically take responsibility for the leak, and the bug could be fixable in a deterministic system.

Here we get a half apology and a reason, but the passive, "we let this happen" is telling. The apology isn't for the bug, but the inability to manage their own system.

> I'm sorry we let this happen. Users had reported Claude would hallucinate names for things like @author tags, since it didn't have context on the account. So we added the account email in the model's context for signed-in users.

That's admitting the leak was intentional, and because such harnesses are loose there can be no assurance it wont happen again.

[–] wellcodeisnot 1 day ago
> We used to engineer systems that worked deterministically.

I had the distinct pleasure of savoring some of the exquisite UI and API experience that Cisco and friends cooked up for me in a misguided IoT adventure. Let me tell you, "deterministic" does not mean "not wrong" or "not stupid".