Hacker News

points by liuliu 4 days ago | 0 comments

These are famous games tested here though, and I tried to feed one problem to DeepSeek 4.1 Flash 1017 and it comes back at 66% confidence it is a Pac-Man (incorrect). That smells to me the problem setup heavily hint towards these 2D games. Might be resolution related to be honest.

[–] wohoef 4 days ago

I tried the same experiment with DeepSeek 4 Flash as well, on more than 4 games. And I think you might be right, the performance is much worse for other games: https://news.ycombinator.com/submitted?id=wohoef

[–] liuliu 3 days ago

Great post. I looked into the reasoning trace, and it seems DeepSeek 4.1 Flash 1017 does know all the games you listed in the post, so there is no knowledge gap. Not sure about Qwen though. I will not be surprised if it doesn't know all these games.

[–] rich_sasha 4 days ago

To be honest I think the experiment would be insanely impressive even if you prompted explicitly that the game is one of 4 options.