points by peri-cl 11 hours ago | 0 comments
The big trend in that graph is that smaller models collapse the most. Everything with "-flash" in the name, plus OpenAI Luna.
Look at GPT 5.6-Sol vs. GPT 5.6-Luna in particular.
(The GLM pair shouldn't be expected follow this pattern, because they're misnamed: GLM-5.3 and GLM-5.3-flash are distinct lineages. The -flash is an early release of a new pretrain on a new architecture).
If this *is* the main factor, I'd expect Kimi K3.1 and Gemini Carbon would also do well on this test (unlike Gemini-flash); while Haiku 6 would collapse (breaking the trend of the other recent Claude models). It's a falsifiable prediction.
[–] druskacik 8 hours ago
[–] Tade0 7 hours ago
There's an interesting way to find this out: ask the model "What are you?" hundreds of times. Around 1% of replies at default temperature are really left field.
For instance, I had Qwen3.8 Flash Next reply with "I'm Kilo Code, a coding agent designed to ...".
That was in llama.cpp's UI - no harness whatsoever.
[–] Muromec 4 hours ago
DS4 when asked about censored stuff through api believed it's Claude. Otherwise it doesn't know what model it is (or pretends to not know).
[–] pianopatrick 10 hours ago
That has unfortunate implications for the cost of using AI for coding.
It's not unreasonable to assume Kimi 3.1 is also a distilled model even though it's large (the difference being that it's a distilled from US models). I wouldn't be surprised if it also collapsed on this benchmark.