points by keybrd-intrrpt 2 days ago | 0 comments
> but will still benefit from the context having higher information density.
That's an assumption.
And is that what we really care about anymore? Maybe we are ok accepting some "inefficiencies" in the code for the LLM but we get better CPU utilization or cache/RAM usage
So is that ;) But yeah, the link between context length and performance reduction is well researched. Automatic context management has been getting better, but it’s not a panacea, especially in very large codebases and long sessions.
> Maybe we are ok accepting some "inefficiencies" in the code for the LLM but we get better CPU utilization or cache/RAM usage
If by ‘inefficiencies’ you mean choosing hardware-friendly architectures over those that correspond to a human-friendly world model, then I would agree with you (this was already required in my domain pre-LLMs). Even still that does not stop one from adhering to DRY, KISS, etc as the abstractions required are often either free or close to free at runtime.