points by embedding-shape 4 days ago | 0 comments
> Unless you are trying to prove some hard math, or writing extremely complex code, for general coding tasks (which represent 90% of code out there) 27B-Q6_K is more than enough. Q4 does it well too, but might need a few more iterations.
99% of the time, I'm not. But if it cannot reasonably write a basic CLI for transforming data without properly following a ~1K token long system prompt to the letter, then waste my time on it?
"Ah yes, I forgot about part X" or "I misunderstood the intent of Y" are so frequent compared to SOTA models, that it's hard to justify relying on it, even though I'm able to run it pretty much 24/7 at home, if it just was intelligent enough to actually make sense to do so.
[–] mycall 4 days ago
> Ah yes, I forgot about part X
Would something like LMCache help here?
I did had it forget some stuff over long runs that went through several context compactions and i'd expect a larger model to be able to handle more vague instructions or have wider breadth of knowledge (if you're trying to make Qwen 3.8 27B do stuff outside the mainstream you need to provide it with some reference material to use) but i never had a problem with it not following instructions.
The harness might matter though but i'm using plain Pi (which is a very barebones harness with a very small system message) without any addons.