Hacker News

points by il 2 days ago | 0 comments

Why was this the default behavior in the first place?

Always use a hidden email service, you cannot trust the LLM or inference provider to secure the email address in this context. If they don’t have the email address, they can’t leak it.

I never give a provider my actual email address, I recommend others do the same. Otherwise, it’s just a matter of time until it is leaked or otherwise part of a data breach.

[–] komali2 2 days ago

If you let the LLM see git commands, they have whatever email you use to sign commits.

[–] ceuk 2 days ago

I use a masked email for Git too. Always have. Last 5 years or so it's just been the GitHub provided mask e.g. <numeric-ID>+<username>@users.noreply.github.com

[–] soraminazuki 4 hours ago

That won't save you in this case because that email address still identifies you. It's an anti-spam measure, not a privacy measure.

[–] j45 2 days ago

Masked email or separate email is the way.

[–] toomuchtodo 2 days ago

I am a security person, I don’t let the LLM see anything except that which I wouldn’t mind printed on the front page of the New York Times. I treat my LLM context as I would a potentially malicious threat actor. Nothing valuable, sensitive, or personal goes into the context.

“lethal trifecta”

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

(a component of my cybersecurity work is red teaming inference, agents, and mcp interfaces)

[–] moritzwarhier 2 days ago

I'd guess it's simply because it tried to use the context to achieve the goal.

Guarding PII is an extra, I guess, and if you want to use a giant cloud-hosted NN to control software on your own device, there are very few default restrictions. And no matter what restrictions there are, when they go beyond prompting, they would have their own ShowHN.

[–] teamonkey 2 days ago

Why is the login information even allowed to touch the context?

[–] michaelt 2 days ago

For the user's name it's intentional; they template it into the system prompt with no option to disable it.

For 99% of users, the model knows it's in the /home/joesmith/whatever repo and `git log` shows the last change was by `joe.smith@example.com` - trying to keep the user's identity secret from the model would be nigh-impossible so I think anthropic gave up on even trying.

[–] hyperhello 2 days ago

But that still doesn't answer the question of why Claude passes the info to web sites. What if it finds your banking information in a personal file and uses it to achieve its goal, which is not different technically?

[–] CharlieDigital 2 days ago

Haven't read the details, but is this because of prompt injection when CC reads the site?

[–] tempaccount420 2 days ago

Some websites will return an error if you send a request to them without a User-Agent - for example, crates.io does this. Curl doesn't send a User-Agent by default.

Then, your agent invents a User-Agent and it sometimes invents one with your email in it.

It doesn't even require prompt injection or a failed request, it might be in the agent's pretraining knowledge that a given website requires a user-agent.

[–] j45 2 days ago

I don't care that it sometimes invents one - it should never do something so ridiculous

[–] caturopath 2 days ago

I use the name "Human" for that but it overrides with its knowledge of my name from other stuff quite regularly.

[–] TuxSH 2 days ago

> `git log` shows the last change was by `joe.smith@example.com`

GitHub offers noreply email addresses

[–] woopsn 2 days ago

You can absolutely remove an email from the context, at the model API it's tokens in tokens out.

[–] zbentley 2 days ago

I can think of tons of common prompts and tasks for which the AI being easily able to introspect subscriber details would be very useful.

Wish that were itself behind a permission wall, but I guess everyone just uses auto mode anyway so perhaps that ship has sailed.

[–] moritzwarhier 2 days ago

This was indeed the friction i meant, but I was too lazy to elaborate.

Common problem: you might consider it trivial to define a rule that forbids access to any file path outside the working directory. Allowing "safe" shell access is already a heuristic, there is no way around it:

- shell scripting is Turing-complete

- there are platform-specific paths that need to be at least readable to perform trivial tasks: Everything is a File!

- security models to make such low-level file access transparent exist, and generally work, but: they are never perfect (container environments, chroot etc). Because...

[–] grey-area 2 days ago

There are ways to sandbox this, and it should have been the first thing Claude Engineers were focussed on, it's a basic first requirement that the app doesn't have access to files outside a very limited set it requires to perform the task at hand.

[–] teamonkey 2 days ago

I can’t think of any, to be honest. It could have access to a parcel that contains some of the subscriber information, but there’s no reason it needs to know how to log in to itself.

[–] oefrha 2 days ago

Do you use git? I have bad news for you, your email will be all over the place in the context as long as the agent is allowed to read git log, whether it’s injected into the system prompt or not. (Likely retort: but not that email! Sure, but leak potential is obviously still there.)

[–] moritzwarhier 2 days ago

But I need to set up remotes to transfer my commits with my e-mail to anyone?

If someone else distributes my commits (or messages), I have no control over that, not any different from sending a physical letter?

I was saying "cloud-hosted" explicitly, and was speaking about what it means to have a remote control software?

Doesn't even matter much whether it's an Artificial Neural Network for the sake of my argument.

[–] cowlevel 2 days ago

So that if you tell it to do something with your email address it'll know what that is.

[–] j45 2 days ago

Bingo. Auto mode doesn't mean send private information outbound. There are other deterministic hard points - why not this?

[–] mpoteat 2 days ago

Some sites like Wikipedia have a policy where scripted clients identify themselves, and there's a standard in HTTP to have a contact email in the User-Agent for such cases. Claude apparently tends to decide that this policy is worth following, and we now disallow this programmatically.

This is why alignment is important and highly non-trivial

[–] swatcoder 2 days ago

> This is why alignment is important and highly non-trivial haha

FYI, it's generally bad form to "haha" your way through privacy violations.

Leaking private information is a very big deal for a lot of people.

[–] BowBun 2 days ago

The stealth edit, ugh. I don't like this social presence when they're failing to leverage their own automations to catch problems like this before the masses do.

[–] phoghed 2 days ago

You people will complain about literally anything Jesus. You gave a coding agent full fucking control of your computer, let it send requests without your review, and then are surprised when it does so. But no, it’s this guy’s fault. Certainly couldn’t be you own.

[–] swatcoder 2 days ago

I didn't do any of those things and this issue doesn't effect me, but I've been around long enough to know how to take responsibility for my work and how to respect the concerns of affected customers when speaking as a professional.

Anthropic has an engineering culture problem that's going to shoot it in the foot if it doesn't learn to take these kinds of issues seriously.

This isn't the first example of the kind of work they're shipping in Claude Code or in how flippantly they engage with the community.

[–] h0rv 2 days ago

> This is why alignment is important and highly non-trivial haha

I disagree, we can likely never fully (or should?) trust models to do these things deterministically, that's why we have harnesses around the agent with built-in constraints.

It is actually in the name: https://en.wikipedia.org/wiki/Harness, https://en.wikipedia.org/wiki/Safety_harness

[–] echoangle 2 days ago

Glad you can find the humor in the situation.

[–] christina97 2 days ago

I cannot believe Anthropic’s official response to having presumably sprayed a significant portion of customers’ emails and private information straight out into the web willy nilly is basically “[it’s] highly non-trivial haha”.

[–] cluckindan 2 days ago

>haha

This is an official Anthropic response???

[–] LastTrain 2 days ago

Because they didn't think about it or any of the other million things that might go wrong - that would take careful planning.

[–] rplnt 2 days ago

Because Claude decided so. You think humans write or even review what they ship to run on your computer?

[–] c0rruptbytes 2 days ago

you'd need to ask one of the billion claude sessions they have doing whatever in their cloud rn