points by Spivak 4 days ago | 0 comments
This seems to be the inherent trade-off with smaller models. They can be just as "bright"
as their larger contemporaries but they necessarily can't encode as much knowledge.
I think over time we'll realize this is the way forward, you want the smartest model with the least amount of embedded knowledge and you provide the knowledge externally or merge it into the weights for the domain you're interested in. Then you don't have to train a ginormous model every 6 months to pull in new information.
[–] dcrazy 4 days ago
[–] lnkdinsuxs 4 days ago
> A RAG solution would make time to first token awful.
in your specific workflows, where specifically is the time to first token a dealbreaker? what's your typical target and what's a "no go"?
[–] dcrazy 4 days ago
I tend to use LLMs conversationally, often as search-and-summarize but also as a pair designer. I find it very difficult to operate in this mode if prefill takes more than, say, a dozen seconds. And that only gets to the reasoning output; depending of effort level it can be multiple minutes to get a reply in a freshly compacted context.
A RAG solution would make time to first token awful.