Hacker News

points by theptip 1 day ago | 0 comments

This is the Model Router architecture, from another angle. The problem is always, how does a dumb model know it needs to escalate to a smart model?

Some problems are obviously hard known-unknowns (eg “solve Navier Stokes”). The problem is unknown-unknowns or knowledge gaps, where the model doesn’t know they need help.

[–] bastawhiz 1 day ago

It's not a problem you can get right, because you can't know if you need to escalate until after you escalate (and what you escalated to disagrees with what your first choice would have been). But that's the point: how cautious you make the model is the knob that lets your trade cost for correctness.

I don't believe the answer is ever "how certain is the model?" It's "how much is at stake?" Or "how hard of a problem does this appear to be?" It's not about teaching the model to have uncertainty, but rather to be avoidant of things that are sufficiently important to get right.

[–] theptip 1 day ago

Yeah, it’s a precision/recall tradeoff. And as such you calibrate by the cost of each FP vs FN.