points by wmf 3 days ago | 0 comments
Congratulations I guess but it's a pretty bad look to not even mention those other open weight models.
[–] flatline 3 days ago
Right? These are not likely to be meaningful points of comparison to anyone but the authors.
[–] ghc 2 days ago
I disagree, my company is restricted from doing on-prem non-US model deployment. I appreciated not having to look up where each model was from since I don’t pay attention to the releases of all the models I can’t use and don’t know their names very well.
[–] wmf 3 days ago
Maybe Palantir will use it or something. There probably is a market for US models but I don't know how large it will be.
[–] Onawa 3 days ago
US Gov won't let Chinese models be hosted on any internal infra, not even supposed to use them for local AI. So there is absolutely a market for US-made or non-Chinese models there.
[–] DaSHacka 3 days ago
Do you have a source for this? Genuinely curious, I've seen reports online of MIC contractors already using it extensively on-prem. In fact, one of the issues they bring up is how there is no official guidance right now, so no one really knows what to do.
[–] ghc 2 days ago
General guidance we’ve received for deployment on protected gov’t networks the past few years has been any open source, whether software libraries or models, won’t pass security review if there are substantial contributions from Chinese/Russian/etc. nationals. There’s even an exceptions list for “well known software” where there are vetted versions for deployment (nginx).
[–] Onawa 3 days ago
I don't have a source unfortunately, this is mostly internal directives from my work within NIH. I've had many conversations with security officers and IT about my plans to use local models for various projects, and hackles were always raised in regards to Chinese models.
I also did a short stint with an IT operations company, and we were told Chinese models were a no-go as well from our deployments on DoD hardware.
[–] schleck8 3 days ago
Because Chinese models heavily rely on distillation and regularly behave strangely in public benchmarks (where they score much better than in private evaluations), even if many world class researchers are Chinese. It's difficult to quantify to what extend labs like Moonshot would be competitive if they could not distill from OpenAI, now that Anthropic mostly blocks it
https://www.theinformation.com/articles/openais-astra-model-...
[–] kevmo314 3 days ago
It would also be pretty difficult to quantify to what extent labs like Anthropic would be competitive if they could not distill from human knowledge.
[–] petcat 3 days ago
Do those other models usually mention all the competition? I haven't noticed
[–] karmakaze 3 days ago
What's funnier is that after cherry picking models for comparison, they further cherry picked Inking for size comparison instead of better and smaller Beam 1.0
The bar chart also shows Nemotron 3 Ultra instead of smaller and better Nemotron 3.5 Super.
[–] Neywiny 3 days ago
3 ultra vs 3.5 super is a big discussion. But in mine and others' experiences, 3.5 super is a lot worse than 3 ultra.
[–] tedivm 3 days ago
We've always been at war with eurasia.
Seriously though, it's just easier to pretend that China doesn't exist when they're stomping us this badly.
[–] stewartpark 3 days ago