Hacker News
points by ActorNightly 4 days ago | 0 comments
Gemma4:31b is better. Qwen is really good at speed (especially with megakernels), but if its going in a wrong direction, it will go in the wrong direction hard. Gemma can self correct with its thinking.

The only reason Qwen is more popular is because it can run decently fast on MLX, while Gemma needs a discrete GPU with 24gb of ram to run at usable speeds.

Also, theoretically speaking, any small model can be as good as a big model if all they have to do is summarize text. Back before agents were a thing, we had prototype coders running of local inference models from Mistral for ai dev. The workflow was to generate llm summaries of the codebase, and a summary of changes to be made. Then we would grab search results from the web, feed it to an llm to summarize, and then finally combine all of that context into a final inference run which would generate code, then the codebase would be run automatically, any errors would be llm analyzed, and an llm would generate the next change to be made in the above cycle.

Nowdays, we can do that with pi coding agent and Gemma4:31b. But we still have plenty of custom harnesses for things.

[–] beachsidebunnie 4 days ago
The context window left on my 24GB card with Gemma is to low to do any major task though. Maybe small diffs or a single direct task. But Qwen has enough context for agentic tasks.
[–] ActorNightly 4 days ago
32k context on my 3090 runs fine.
[–] DavidSchor 4 days ago
32k context is quite limited in agentic context. Qwen 3.8 27b is much better in agentic and coding workflows. Gemma4 31b is much better in general assistant and language focused workflows.
[–] t0mas88 4 days ago
Realistically a 32GB card is what you need to run Gemma4:31b well. Doing it in 24GB leaves almost no room for context / KV cache.
[–] ActorNightly 4 days ago
32k context is plenty fine. thats 50 pages of single spaced text
[–] leansensei 1 day ago
Not if you preserve reasoning in the KV cache, 32k ctx is nothing.
[–] clusterhacks 4 days ago
...especially with megakernels...

I don't understand why megakernels haven't become the default choice. The speed advantage is crazy. I made a few changes to the openjet megakernel to run in a dual gpu (older gen, not blackwell) box and doubled my token generation speed on qwen3.8-27B (4-bit though).

I keep debating adding a third matching gpu to see if I can push the speed a bit more and move back to the 8-bit quant, but I'm stuck on the idea of just building a Blackwell gpu system despite the cost factor. Plus a third gpu is going to going to be weird for the tensor split and P2P data syncs and my workstation is old with 3rd gen PCIe etc etc.

[–] ActorNightly 4 days ago
Architectures (like Gemma4) have things like sliding window attention that make it pretty impractical to do on a GPU, you would have to do additional things to make it work, and the talent to do that, even with LLM assistance is very small.
[–] clusterhacks 4 days ago
Ah, that makes sense. I haven't bothered to look at current low-level NN architecture details. I'm a little surprised that there isn't a bumper crop of relatively well-informed talent to do gpgpu work. Although, when if I think about it, it has always been a little bit of a niche area.