Update: let me elaborate more, I read the HN entry about the team of 5 Qwens being close to fable quality... so I embarked to add to my harness (https://github.com/alainnothere/privibe because we all have one right?) and once I had added the orchestrate tool I had to now measure somehow it it's able to improve quality, so I claude and then Qwen to run a experiment to call my harness passing the model to use and ask them to do something, 3 rounds I think and see what quality came out of it... and as per the grading... qwen and claude agreed that the quality of the response comming out the harness using the orchestrate was VERY CLOSE, with opus and fable providing more "in my opinion you should" which matches my experience on why they are better.

One more update: I can't get copilot models to work decently, it may be my way of working... but I can get Qwen to follow what I try to do... if Qwen get's me he can then fire copilot cli and get amazing results by using copilot... bonus points seems like copilot cli is cheaper with the AIC (cost of the requests) so you can make the copilot subscription longer

People, please try to understand what you are building. Don't just build it as asked!

With an assemblage of models like Jev + Opus or Jev + Qwen, Jev would often do the right thing when deciding to escalate to another model - I was impressed that it would usually make the right call, especially if you gave it good awareness of what the other model capabilities were. "ESCALATE to opus if you need good vision awareness".

When Haiku 5.5 came out though, for my tasks, the cost differences disappeared almost immediately. Haiku was good enough / fast enough for so many things that introducing a decider layer didn't offer huge wins. Jev's "millisecond" level performance was nice but would get dwarfed by other latencies in my chain ( web browsing ), that it didn't make a measurable difference in my tasks.

Despite the title, the article is not making claims of, or demonstrating general usage improvements or general coding agent improvements

Why give all the context to an agent, when the context can also be curated, updated and made concise automatically for each new problem.

They propose:

Valentino → Female Child

Yet this is obviously a boy name, else it would be Valentina. Is this model going to return confidence values? No, they want decisions. This is an inherent problem in AI with reductive logic for these tasks.

Likewise, dates. Is 17/08/22:

- 20170822 (Chinese write the year first, and in speech and chat they will say '17' not '2017'.)

- 20220817 (Everywhere But North America.)

Is 04/06/2009:

- 20090604 (Non-North American)

- 20090406 (American)

Without knowing the cultural context, the location, the document type, the document author, etc, you could be getting it wrong.

Jev's confidence score was more reliable. Which means that they were able to use Jev despite it having lower accuracy, because of the confidence was below a cut off, they would escalate to an expensive model (Opus)?

Whereas Luna, even though it was more accurate than Jev, its confidence was way off, so they couldn't reliably know when to fall back?

I don't have the actual data but based on the graphs I estimate the expected calibration error to be 2-3% lower for Luna on both the tasks.

Its true that Luna seems to give lots of high confidence values but it's also correct more. Higher number of mistakes at higher confidence isn't itself undesirable if its pass rate is also high. It'd be good if they reported actual ECE.

>Most of Jev's mistakes are closer to 50% confidence, which is what you'd expect out of a well-calibrated model.

This statement is also incorrect

I don't usually get excited about the high-level, "whoooa man" marketing-speak, but the parallel is striking.

I have haiku hooked up for guardrails in my llm proxy. With the prompts routed to opus or sonnet after they pass the check.

I'll replace haiku with Jev and my costs should definitely drop.

[1] https://github.com/codealong/codealong

Give me a decision model that can do both, more expensive than today’s decision models but still cheaper than full on frontier model, and it’d open up so many more use cases, IMO

I agree. What I’ve encountered though is even building enough cases for a calibration data set is hard. And so we farm out to smarter expensive models like Astra to label cases and use that as a calibration data set for the now cheaper LLM judge actually running as our judge

Is this standard ? Or am I just too lazy to manually label enough data to calibrate ?

That isn't obvious, unless you've been doing ML for a while.

Some problems are obviously hard known-unknowns (eg “solve Navier Stokes”). The problem is unknown-unknowns or knowledge gaps, where the model doesn’t know they need help.

GPT-6 Luna is a small LLM, not a decision model :P

It's what I use locally. For my workload, it was right up there with Jev.

Hope my sarcasm is making it across.

And yes, that saves money since less tokens are sent to a remote LLM.

If we compare coding agents with Jev on vs off we get those numbers (metered with Memdoor): - read-heavy tasks (coding questions , logs tail etc..): -63% to -65% input tokens, 40-45% fewer model calls, on GLM 5.3 half the wall time. Reproduced a month apart on two models (DeepSeek V4.1 Flash, GLM 5.3). - a file edit: -23%

Two things match the post. Calibration is what makes it safe: I apply a decision at confidence >= 0.75 and fail below this threshold or on timeout or on an unknown tool family. The dollar saving scales with the price of the model being saved: on a $0.04/M model Jev costs what it saves so on a $0.38/M model it's negligible but on Claude model for example it can be outstanding.

Where it did not help: giving Jev more context about the repo. I found same thing as the post.

If you are interested i have put figures here : https://github.com/guregodevo/memdoor/blob/main/docs/feature...