points by mannyv 1 day ago | 0 comments
What's interesting and annoying is that the results only matter for these specific releases of the models, with the specific parameters used.
That isn't obvious, unless you've been doing ML for a while.
That isn't obvious, unless you've been doing ML for a while.
These are the results of a little performance tuning/permutation testing run against context format and question wording we did. Obviously there is a lot of noise here but I was actually kinda surprised that there was a clear "best way to ask the question" even for relatively low N and low complexity/ambiguity of the task.
(This is a ~30s Ai-driven gif presentation of the results, the full grid shows up about 15s in): https://imgur.com/a/wGFZvGw
These models in particular have some pretty dubiously generalized and diverse training datasets. You can see this for yourself BTW by looking at the datasets SOTA Jev-like models are being trained against and evaluated on. No knock against the teams btw, I just think it's rather demystifying and makes me quite optimistic at how much better these models are going to get over the next few years: https://labs.baseten.co/articles/mammoth-1-and-elephant-1-fr...