Hacker News
points by lnkdinsuxs 4 days ago | 0 comments
> A RAG solution would make time to first token awful.

in your specific workflows, where specifically is the time to first token a dealbreaker? what's your typical target and what's a "no go"?

[–] dcrazy 4 days ago
I tend to use LLMs conversationally, often as search-and-summarize but also as a pair designer. I find it very difficult to operate in this mode if prefill takes more than, say, a dozen seconds. And that only gets to the reasoning output; depending of effort level it can be multiple minutes to get a reply in a freshly compacted context.
Hacker News