Hacker News
points by entrope 3 days ago | 0 comments
For most purposes, this seems reasonable? This looks like a single-height card, so fitting four of these in a workstation or server would give 512 GB RAM and 2 TB/s total bandwidth. That's pretty attractive for a lot of current LLMs, even if one wishes for 1 TB capacity at 5 TB/s bandwidth.
[–] throwawayffffas 3 days ago
If you are going down this path, MI350P and RTX PRO 6000 Blackwell is what you would want. A lot more compute a lot more vram bandwidth.
[–] zozbot234 3 days ago
> A lot more compute a lot more vram bandwidth.

But this gives you significantly more memory capacity for a compute/vram bandwidth tradeoff (i.e. arithmetic intensity) that's ultimately comparable to Blackwell, perhaps even more skewed towards compute. This seems good if you care about being able to store larger sparse models and more KV caches because that ultimately bottlenecks your ability to do inference.

[–] entrope 3 days ago
Maybe. I don't know power for the MI350, but the RTX PRO 6000 is normally a 200-300W card, so you pay 2x the power for 3/4th the RAM and 3x the bandwidth. The performance frontier looks very different in terms of power to hold a mid-size model and throughout per user or across users. Price and inference stack would be big factors in what I would choose.
[–] throwawayffffas 3 days ago
MI350P, H200 NVL and Nvidia RTX PRO 6000 are all 600W but compute wise all of them should be 3 to 4 times more performant than A100 Ultra according to their specifications.

https://www.servethehome.com/the-amd-instinct-mi350p-is-a-hb...

[–] eyegor 3 days ago
600w is way off the vf curve though... Tiny bit of tuning and you'll get ~80-90% of the performance at ~150w nominal. At least that's what I got out of some pro 6000s when I spent 15 mins to underclock it.
[–] varispeed 3 days ago
But there are already plenty of products for most purposes.