Hacker News

points by grey-area 3 days ago | 0 comments

At first I thought this was work on the far more important problem of proving that LLM output is provable and correct given some source of truth. But it's about the trivial problem of 'proving' which model is being used.

Who will find this useful? I can't really think of a use for it, because if you don't trust the provider of your models, it's game over really - you're already giving them all sorts of private data, you're relying on their service, you have to trust them to some extent and this doesn't really help because unless you are running the service yourself. Seems like more of an internal problem for Anthropic than something that would interest others.

[–] OneDeuxTriSeiGo 3 days ago

The main benefits are:

1. it proves your models are running correctly. FP ops tend to deviate in small amounts between hardware providers and tensor ops even more so due to the relatively new and unstable standardisation environment atm. So it proves your HW configuration is delivering outputs correctly. i.e. it's a good test for what is currently a very non-trivial validation environment.

2. it proves your external provider is running the inference they said they are. i.e. it eliminates the concerns of model providers running quantized versions silently behind the scenes since quantization breaks the proof and you need to use a new, updated proof signature for the quantized version.

3. likewise for fine tuned or tampered models. Or outright substitution for another model (ex the accusation by OAI and Anthropic that Kimi and DS and co are periodically forwarding requests to their platforms to harvest responses for knowledge distillation).

All in all it's a nice chain of trust utility if it can evolve out of the toy stage and reach production viability (since rn it's limited to int-only weights and 1k context which is practically useless atm).

[–] grey-area 3 days ago

1. That sounds like an Anthropic concern, not a concern for most customers who don't run their own model. 2. This still requires trust in the provider, I imagine these proofs would change weekly as they tweak their models, and you have zero knowledge of what the changes are, you just know you're getting version 11.12.1999 - does that really help you if the provider is maliciously downgrading after release? All you know is you got the same version as everyone else, not what that version is. 3. Again, if you don't trust the provider, it's game over - you're sending them all of your data. Knowing a version number is not going to help you much.

I'm not convinced this is useful for problems most people face using models, it seems a very niche academic concern frankly and an incredibly complex system to prove not very much.

[–] OneDeuxTriSeiGo 3 days ago

1. It's a provider concern and a developer concern. If your infra is serving broken proofs that might be justification for compensation for breaking SLA.

2. Models only change weekly for closed frontier models and even then they should be reporting model "sub-versions" if only for observability purposes.

3. This doesn't really hold true. Kimi and DS release their models on a monthly or bi-weekly cycle and while it's still DS-v4.1-Flash or whatever, under the hood it's DS-v4.1-Flash-1017 or some other date code identifier of which specific model it is and that is both exposed to the user and that model version can be requested explicitly by the user.

[–] imtringued 3 days ago

>1. That sounds like an Anthropic concern, not a concern for most customers who don't run their own model.

You're talking about this as if a bug or defect in their inference stack has no impact on you as the customer.

[–] OneDeuxTriSeiGo 3 days ago

Yeah a provider serving botched inference probably would count against SLA if it was at all detectable.

[–] manugo4 3 days ago

Also useful to audit in case we pause the development of AI and we need to check inference is being done with known models.

[–] skybrian 3 days ago

For example, there are Chinese resellers that promise to provide access to Claude (which isn't officially available in China) and defraud users by substituting something else. They're obviously sketchy in all sorts of ways, but in theory, this would prevent one kind of fraud.

It might also provide an additional guarantee for users of Open Router and Fireworks.

[–] grey-area 3 days ago

But if you’re using them they’re probably already stealing your data, fake-models are the least of your problems.

[–] nater5000 3 days ago

>Who will find this useful?

It's worth reading past the opening paragraph before asking these kinds of questions:

>When a user sends a request to an AI model, today they have no way of knowing for certain that the model they selected is the one that generated the response. For everyday uses, in which users mostly care about the output and not necessarily which specific model produced it, this is a less pressing concern. But there are several cases where such assurance would be useful. For example, independent evaluators who test AI models before and after release need to know that the model they are evaluating is the model being served. Clients who use a model through a third-party routing service want assurance that the tokens came from the model they paid for. And if a cyberattacker modifies the token stream while pretending to be a given model, a user has no way to detect it from the output alone.

This is from the first paragraph in the Introduction.

[–] grey-area 3 days ago

I didn't find any of it compelling, perhaps you do. That's what I was asking really, which of these use-cases is valuable for you?

For me if you use a routing service you have to trust it, it sees all your inputs and outputs.

If a cyberattacker can modify the token stream (i.e. man in the middle your traffic), you should be worried about a lot more than which model you're using!

[–] hobofan 3 days ago

If one were to listen to /r/claude or /r/codex where half the posts are complaining about models being "lobotomized", one could think that this is the most valuable thing there is!

In all seriousness, this is mostly just one of those defense in depth kind of things. Data exfiltration via a compromised provider is also a very different class of vulnerabilities than prompt injection via malicious tool calls.

[–] grey-area 3 days ago

But it still requires trust in whoever is running the server that tags version 'x' as x. All you have proof of is that they return the hash for a given set of weights which is the same as someone else's hash, and those tags/hashes will be changing all the time as the models evolve, so people are not going to be able to keep track of whether Anthropic say is dumbing down their model or not after release, because this is a zero knowledge proof, all it tells is that a certain set of weights was used (you have no idea what weights, how many weights etc etc).

So I don't really see this as useful, except in the very limited use case of using a third party proxy who is trying to cheat you (in which case why are you sending all your data to someone you can't trust?).

[–] hobofan 3 days ago

> and those tags/hashes will be changing all the time as the models evolve

I think that's where your assumption differs from stated claims.

According to Anthropics own docs[0], they don't alter the models at all after release. So for each model release you should be able to prove that a response was created with that exact model without dumbing it down.

Zero knowledge proofs also still allow you to have disclosed inputs. You can make one of the inputs be a publicly commited hash of the model weights, and have the ZKP validate that the inference was done with that committed set of model weights, in addition to all the inferences in the batch all coming from the same model.

[0]: https://platform.claude.com/docs/en/about-claude/models/mode...

[–] grey-area 3 days ago

If the releases change frequently though, this would be of little use to anyone save those who feel sticking with one model is best. The weights are not exposed so you have to trust Anthropic that changes they make are in your best interest.

[–] nater5000 3 days ago

>which of these use-cases is valuable for you?

None of these are directly valuable to me. But you asked "Who will find this useful?" and I gave you the answer to that question (provided by Anthropic in this post). Not everything being worked on will be useful for you. It's just bizarre that you'd be trying to make these arguments.

>If a cyberattacker can modify the token stream (i.e. man in the middle your traffic), you should be worried about a lot more than which model you're using!

Sure, so it'd be great if you could know if someone is modifying your token stream. If only there was some advancement in provable inference that could allow you to verify that what you're getting from a provider is what you expect to get from them.

[–] grey-area 3 days ago

If only the internet had some way to encrypt data in transit between two parties.

This is a solved problem if you trust your counterparty and if you don’t it’s the least of your problems.

All this provides is a tag saying 'x' version (which your counterparty defines anyway) is being used.

[–] porridgeraisin 3 days ago

Off the top of my head, it is useful when there are content-level proxies between the client and anthropic's API server, of which there are many these days.

[–] pxmpxm 3 days ago

Customers care about model output being correct, Anthropic cares about model output being generated by Anthropic model.

This solves for the latter with a bunch of handwaving for the former. The end goal is gonna be FUD for cheaper model routing because that obviously is at odds with their biz model.

[–] elzbardico 3 days ago

You developed your agent implementation and more importantly, evaluated it, against a specific model.

You don't want all you eval work to go to waste just because the discount inference provider you're using decided AcmeModel4.23 is as good as the Anthropics GilgameshEpic7.2 that you used to validate your solution.

We already have trouble enough with model retirement.