Hacker News

points by layer8 4 days ago | 0 comments

They did test for correct interpretation, so what point are you making?

These are famous games tested here though, and I tried to feed one problem to DeepSeek 4.1 Flash 1017 and it comes back at 66% confidence it is a Pac-Man (incorrect). That smells to me the problem setup heavily hint towards these 2D games. Might be resolution related to be honest.

[–] wohoef 4 days ago

I tried the same experiment with DeepSeek 4 Flash as well, on more than 4 games. And I think you might be right, the performance is much worse for other games: https://news.ycombinator.com/submitted?id=wohoef

[–] liuliu 3 days ago

Great post. I looked into the reasoning trace, and it seems DeepSeek 4.1 Flash 1017 does know all the games you listed in the post, so there is no knowledge gap. Not sure about Qwen though. I will not be surprised if it doesn't know all these games.

[–] rich_sasha 4 days ago

To be honest I think the experiment would be insanely impressive even if you prompted explicitly that the game is one of 4 options.

[–] nightski 4 days ago

From what I saw in the paper, no they did not. They tested that the specific change introduced increased performance on a single task (the one mentioned in the prompt). They did not study the impact of that change on overall model performance across a variety of benchmarks. But maybe I missed something.

[–] lnenad 4 days ago

But increasing performance on a single task was the point of the change?

[–] garg 4 days ago

His point is that he has read Shakespeare but not the linked article.

[–] 3form 4 days ago

He can call on comments from the vasty deep, but shalt we find them relevant?