Hacker News

points by advael 3 days ago | 0 comments

Category narrowing to this degree really has diluted the term "SOTA" quite a lot

You mean it's state of the art for this subset of benchmarks you're targeting? No? Well surely it's state of the art for models of a certain size category? Huh, what about models that are available for local inference? O, state of the art for open models of a certain size made in the US specifically. Uh. Neat!

[–] gpm 3 days ago

Playing games like this also makes it very easy to accidentally say things that aren't true, like in the headline of the article...

> Introducing Ripple-65B Preview, a Frontier Open-Weight Model

It's not "frontier", it's not "frontier" even in the category of open-weight models (though usually that phrasing would mean frontier over all, and also open weight)... it's only "frontier" in the small sub-category of US made open-weight models.

[–] GuB-42 3 days ago

It is technically frontier, if by "frontier" we mean the Pareto frontier.

Being at the Pareto frontier means that you can't get something better on one aspect without compromising on some other aspect. The trick is that it is easy to be at the Pareto frontier by having enough dimensions.

Here, we have 4: score, number of parameters, origin (US) and open-weight. You can get a better score or a lower number of parameters, but you have to compromise on at least one of the other aspects. That's a pretty weak frontier if you ask me.

[–] derefr 3 days ago

“Origin” isn’t part of the Pareto frontier because it’s not a quantitative dimension.

(You could say that “ability to be trusted by US government/military users” is quantifiable, but I would argue that by that metric, this model falls behind non-US-originated open-weight models that have more eyes on them!)

[–] Routine8502 3 days ago

Discrete dimensions can exist in Pareto considerations. Not that I'm too fond of this game, but given that "Origin" is definitely part of the calculus of model deployment, it makes sense to consider it in a Pareto calculation.

[–] getcrunk 3 days ago

Ahh I run into this issue all the time with ai when applying any sort of dimensional reasoning abstractly it always implicits continuous

[–] bastawhiz 3 days ago

It's just a different pareto frontier because you're filtering out items that don't fit your criteria. "Model was made by Anthropic" would be an equivalent dimension. You would never plot this, you're just reducing the subset of models you're comparing.

By that logic, any model is a frontier model. My GPT-2 clone is a frontier model when you compare it against models that were trained on my homelab.

[–] kennywinker 3 days ago

Which you're welcome to do and then people are welcome to disagree that is an important frontier to consider.

[–] bastawhiz 2 days ago

That's literally the point. It's a different frontier. If you say "frontier model" and don't specify which frontier you're either being deceptive or naive.

[–] jhgb 2 days ago

But Pareto optimality only considers ordinal information. Surely there's no reason why binary choices can't be dimensions in it.

[–] yorwba 3 days ago

AFAIK the currently biggest US open-weight model is Inkling, which they include in the comparison, so at least their claim is not limited to a size category. Still they score 0.42 percentage points higher than the second-best model on their chosen benchmark, so it's hardly a meaningful lead.

[–] quixoticelixer- 3 days ago

Fourth most popular comedy folk rock duo

[–] hkt 3 days ago

..in New Zealand!

[–] senko 2 days ago

Flight of the Conchords!

[–] frinxor 3 days ago

The US bit got me as well.

I guess it makes sense - there is a segmentation where users may want to run on domestic "frontier LLMs."

[–] twelvechairs 3 days ago

Why should US or China matter if its open weights? Its going to matter a whole lot more whether you are running it locally or where your provider is surely.

[Edit i think the preferred combination for those who actually want some data security at the moment and arent at the scale of self hosting llms is chinese open weight models served in germany or France where theres some better data privacy/security]

[–] aftbit 3 days ago

There are a number of reasons that you might not want to build your inference stack on a Chinese origin model even if you're running it entirely in house.

For one, you might not trust that the weights do not contain ideological biases (e.g. ask DeepSeek about tibet) or that the methods used to impose these biases may not have contaminated the model in some other way.

For another, you might not trust that the code the model generates won't be backdoored or attempt to leak information through your sandboxes.

For a third, you may not want to build your stack on a partner who you feel is a supply chain risk, who might become unaligned with your interests.

By far the biggest though, is that many industries will be forbidden from doing so by the government in an attempt to protect the US model industry (and thus create the demand for some model like this one).

I'm always excited to see new entrants to this open-models race, no matter why they're running.

[–] dragonwriter 3 days ago

There are zero reasons to have any less concerns about ideological biases, side effects of the mechanisms used to implement ideological biases, or code backdoors in a US model, and a supplier of open (as opposed to hosted) model weights is not an active ongoing partner with whom the idea of supply chain risk from possible future divergence of interests is coherent, so that concern is either invalid or, if the “partner” is the model itself, simply a restatement of the ideological bias in the model weights concerns which, again, is not diminished by the model having a US origin.

And your speculation about future protectionist regulation is just that, not an actual fact.

[–] aftbit 1 day ago

>There are zero reasons to have any less concerns about ideological biases, side effects of the mechanisms used to implement ideological biases, or code backdoors in a US model

If you are a US based critical infrastructure provider, I'd argue that you have already accepted a baked-in threat of ideological misalignment from the US, but you have not accepted it from farther away countries like China.

>a supplier of open (as opposed to hosted) model weights is not an active ongoing partner with whom the idea of supply chain risk from possible future divergence of interests is coherent

I disagree. Models are improving so quickly that you'd probably not want to freeze yourself on an older model forever. You likely want to continue to upgrade models. The best way to do this is to build a sufficiently powerful and representative benchmark that you can confidently move to _any_ model, but you might expect that future models from the same providers that you accepted before will show less divergence than totally new providers. Similarly, if you build your inference around one of their custom tools (e.g. DeepInfer rather than vLLM), you might find substantial switching costs in moving away to another provider.

>And your speculation about future protectionist regulation is just that, not an actual fact.

It's not a done deal for sure, but you can already see a lot of signs of this in recent actions from the government. I can tell you based on personal experience that some customers will automatically rule out any Chinese models.

[–] spockz 3 days ago

I think the only point being is that if you see US as homogeneous and you are from the US then it’s okay to use an US model because it’s values will be signed with yours. But that is pretty weak anyway.

[–] dragonwriter 2 days ago

If you see the US as homogenous and you are from the US, you have much bigger problems than your choice of open LLMs.

[–] walrus01 3 days ago

> For another, you might not trust that the code the model generates won't be backdoored or attempt to leak information through your sandboxes.

I don't disagree with you on the severity of the concerns, but given the vast amounts of code that must have been generated in the past six months by the likes of GLM5.2, GLM5.3, deepseek v4 flash 0731 and newer, qwen 3.5-122, 3.6 and 3.8 series models, I would like to see any concrete examples of it building backdoors/intentional vulnerabilities/phone-home-to-china or similar unprompted. If it's been doing this and anyone with even a modicum of a clue has been watching it, surely someone has caught a model doing this? At least once?

Surely a model doing something like writing intentionally obfuscated code that obscures its actual purpose from a code reviewer, in the service of something malicious/backdoor-like, would have been seen and noticed by now.

[–] aftbit 1 day ago

https://cohere.com/blog/concerning-behaviour-popular-open-we...

[–] vrganj 3 days ago

Every model has ideological biases, such is the nature of ideology.

The question is only whether they overlap with your own.

[–] mcnewcp 3 days ago

I wish it didn't matter to me, but it does. I work in government and we self host. No matter how much I explain to them that by hosting weights on our own hardware, it doesn't matter where they came from, they still refuse to let me deploy anything from China.

[–] dragonwriter 3 days ago

> Why should US or China matter if its open weights?

Weights aren’t source that can be inspected for nefarious tricks that would impact use, so, to the extent that there is a dinension of trust associated with nation of origin, being open weights doesn’t obviate the concern (I would argue, though, that there is very little reason to for nation of origin to be a significant factor in trust alone, though a finer grained consideration of origin might be sensible.)

[–] lynx97 3 days ago

> Why should US or China matter if its open weights?

Well, you might want to know what propaganda/censorship you are subjected to. Case in point, a regional government in Austria just learnt that their AI doesn't know about the Tiananmen Square massacre.

[–] mettamage 3 days ago

I’m SOTA on being exactly who I am.

Yep, you’re right. Quite easy to dilute the term.

[–] visarga 3 days ago

No, the term is well defined. It means being on the Pareto curve of performance vs size, such as: https://tinyurl.com/26sw35fw

[–] advael 3 days ago

Irrelevant. A term can be well-defined and used correctly while also being diluted. In this case, I am pointing out that it's diluted by narrowing the category in which the claim is being made. I have no doubt that in this narrowed category, the term is being used correctly. This has no bearing on the fact that by narrowing the category in that way, this claim is made remarkably less impressive or interesting to many people hearing it, including myself

[–] epolanski 3 days ago

By...whom? An ISO standard? Highly cited scientific literature?

A tinyurl pic linked by randoms over the internet?

[–] XnoiVeX 3 days ago

Frontier and SOTA usually mean that something is on the edge of what's possible. The bleeding edge.

[–] jhgb 2 days ago

The Pareto frontier IS "the edge of what's possible".

[–] O_H_E 3 days ago

well yes but the y axis heavily depends on the benchmark you choose

[–] vikramkr 3 days ago

Terms can have multiple definitions and language is descriptive not prescriptive. If you intend to communicate that you mean "Pareto frontier" in the current context you need to include the word "pareto." If you don't then "frontier" will be understood as referring to mythos/astra/opus 5.5 tier models in intelligence (max capability regardless of tradeoffs). It doesn't matter that the second definition in the dictionary is the one you want to convey (charitably assuming you're not trying to mislead given the value of the hype of being able to call your model a frontier model) if your audience understands what you're saying to mean the first definiton.

[–] bastawhiz 3 days ago

Is it really on the pareto curve if you get to define arbitrary conditions for what other models are plotted on the curve?

[–] msy 3 days ago

It's like the Baseball World Series of Frontier models. It's SOTA in the same way Tesla are SOTA EVs - the best you're permitted to buy in the USA.

[–] drob518 2 days ago

Exactly my thought. It all struck me as roughly “Best of all open weight models designed in a leap year and released on Thursdays.”

[–] dominotw 2 days ago

First one eyed dude with tatoo on left nut to climb everest.

[–] drob518 2 days ago

Bingo.

[–] swiftcoder 3 days ago

For sure, but unfortunately, there's a non-zero chance the state department at some point makes the "US" part of this mandatory for a bunch of companies.

[–] cyanydeez 3 days ago

It means ot was released recently.

It was already meaningless when US labs used it.

It means new.

[–] dev_l1x_be 3 days ago

State of the AI-Apocalypse.