Hacker News

points by moritzwarhier 2 days ago | 0 comments

I'd guess it's simply because it tried to use the context to achieve the goal.

Guarding PII is an extra, I guess, and if you want to use a giant cloud-hosted NN to control software on your own device, there are very few default restrictions. And no matter what restrictions there are, when they go beyond prompting, they would have their own ShowHN.

[–] teamonkey 2 days ago

Why is the login information even allowed to touch the context?

[–] michaelt 2 days ago

For the user's name it's intentional; they template it into the system prompt with no option to disable it.

For 99% of users, the model knows it's in the /home/joesmith/whatever repo and `git log` shows the last change was by `joe.smith@example.com` - trying to keep the user's identity secret from the model would be nigh-impossible so I think anthropic gave up on even trying.

[–] caturopath 2 days ago

I use the name "Human" for that but it overrides with its knowledge of my name from other stuff quite regularly.

[–] hyperhello 2 days ago

But that still doesn't answer the question of why Claude passes the info to web sites. What if it finds your banking information in a personal file and uses it to achieve its goal, which is not different technically?

[–] CharlieDigital 2 days ago

Haven't read the details, but is this because of prompt injection when CC reads the site?

[–] tempaccount420 2 days ago

Some websites will return an error if you send a request to them without a User-Agent - for example, crates.io does this. Curl doesn't send a User-Agent by default.

Then, your agent invents a User-Agent and it sometimes invents one with your email in it.

It doesn't even require prompt injection or a failed request, it might be in the agent's pretraining knowledge that a given website requires a user-agent.

[–] j45 2 days ago

I don't care that it sometimes invents one - it should never do something so ridiculous

[–] krapp 2 days ago

Why not?

It's just doing what it's supposed to do, analyzing the statistical patterns of a natural language prompt and matching it to a likely response.

What's ridiculous is an assumption that it "it should never do something so ridiculous" as if an LLM is a person who understands what that word means in any context other than next-token prediction.

[–] TuxSH 2 days ago

> `git log` shows the last change was by `joe.smith@example.com`

GitHub offers noreply email addresses

[–] woopsn 2 days ago

You can absolutely remove an email from the context, at the model API it's tokens in tokens out.

[–] zbentley 2 days ago

I can think of tons of common prompts and tasks for which the AI being easily able to introspect subscriber details would be very useful.

Wish that were itself behind a permission wall, but I guess everyone just uses auto mode anyway so perhaps that ship has sailed.

[–] moritzwarhier 2 days ago

This was indeed the friction i meant, but I was too lazy to elaborate.

Common problem: you might consider it trivial to define a rule that forbids access to any file path outside the working directory. Allowing "safe" shell access is already a heuristic, there is no way around it:

- shell scripting is Turing-complete

- there are platform-specific paths that need to be at least readable to perform trivial tasks: Everything is a File!

- security models to make such low-level file access transparent exist, and generally work, but: they are never perfect (container environments, chroot etc). Because...

[–] grey-area 2 days ago

There are ways to sandbox this, and it should have been the first thing Claude Engineers were focussed on, it's a basic first requirement that the app doesn't have access to files outside a very limited set it requires to perform the task at hand.

[–] teamonkey 2 days ago

I can’t think of any, to be honest. It could have access to a parcel that contains some of the subscriber information, but there’s no reason it needs to know how to log in to itself.

[–] oefrha 2 days ago

Do you use git? I have bad news for you, your email will be all over the place in the context as long as the agent is allowed to read git log, whether it’s injected into the system prompt or not. (Likely retort: but not that email! Sure, but leak potential is obviously still there.)

[–] moritzwarhier 2 days ago

But I need to set up remotes to transfer my commits with my e-mail to anyone?

If someone else distributes my commits (or messages), I have no control over that, not any different from sending a physical letter?

I was saying "cloud-hosted" explicitly, and was speaking about what it means to have a remote control software?

Doesn't even matter much whether it's an Artificial Neural Network for the sake of my argument.

[–] cowlevel 2 days ago

So that if you tell it to do something with your email address it'll know what that is.