Hacker News

points by andai 2 days ago | 0 comments

Well done. $20k? That's insane. I remember tinyllama back in the day (2023) was trained on $40k and could barely form coherent sentences.

Are pictures easier than English, or did the methods advance so much?

Edit: I guess I should have said, are pictures plus English easier than English!

>The text representation comes from a frozen Qwen3.5-4B model, which we read at layers 7, 15 and 27 (its three full-attention layers). We concatenate and project these representations into the text refiner.

Still, as a layperson I would have expected text to be simpler than images, so I'm still amazed by this.

[–] wongarsu 1 day ago

Over the last 3 years we made a lot of progress in efficient training methods. On text models, you can now train GPT-2 levels of coherent English from scratch for about $50 (e.g. using Karpathy's nanochat [1])

1: https://github.com/karpathy/nanochat

[–] andai 1 day ago

That's insane. How does that work? Does it take so much less data now, or smaller models, or what? Apparently GPT-2 cost ~$40,000 to train when it came out.

I'm sure Moore's Law for GPUs helped a little, but I'm really curious about the savings on the methodology side. (The repo mentions "hyperparameters")