1. it proves your models are running correctly. FP ops tend to deviate in small amounts between hardware providers and tensor ops even more so due to the relatively new and unstable standardisation environment atm. So it proves your HW configuration is delivering outputs correctly. i.e. it's a good test for what is currently a very non-trivial validation environment.
2. it proves your external provider is running the inference they said they are. i.e. it eliminates the concerns of model providers running quantized versions silently behind the scenes since quantization breaks the proof and you need to use a new, updated proof signature for the quantized version.
3. likewise for fine tuned or tampered models. Or outright substitution for another model (ex the accusation by OAI and Anthropic that Kimi and DS and co are periodically forwarding requests to their platforms to harvest responses for knowledge distillation).
All in all it's a nice chain of trust utility if it can evolve out of the toy stage and reach production viability (since rn it's limited to int-only weights and 1k context which is practically useless atm).
[–] grey-area 3 days ago
[–] OneDeuxTriSeiGo 3 days ago
2. Models only change weekly for closed frontier models and even then they should be reporting model "sub-versions" if only for observability purposes.
3. This doesn't really hold true. Kimi and DS release their models on a monthly or bi-weekly cycle and while it's still DS-v4.1-Flash or whatever, under the hood it's DS-v4.1-Flash-1017 or some other date code identifier of which specific model it is and that is both exposed to the user and that model version can be requested explicitly by the user.
[–] imtringued 3 days ago
You're talking about this as if a bug or defect in their inference stack has no impact on you as the customer.
I'm not convinced this is useful for problems most people face using models, it seems a very niche academic concern frankly and an incredibly complex system to prove not very much.