Hacker News
points by Silagi 3 days ago | 0 comments
Huh. I wrote a whole thing here about serving a niche in serving local LLMs, but turns out, any niche it could have served is probably better served (per dollar) by either more RAM or modern SSDs.

There's an argument to be made for the persistent memory being a cheaper substitute for more RAM, but imo the gap isn't big enough to justify it when considered as a % of the rest of the system you're building around it and the performance tradeoff.

Important reminder to check your biases. I was sure Optane had missed its mark by dropping out just before LLMs became a primary use case for high $ systems.

[–] AlotOfReading 3 days ago
There was a version of optane (optane DC?) that plugged into DIMM slots and cost less than RAM, with possible higher capacities. That would be a useful tradeoff had it kept up the advantage.
[–] amluto 3 days ago
Intel had multiple false starts on getting their platform ready for these devices to be used in a sensible manner. I have one of these false starts in a closet somewhere, and I did some kernel dev work for it. It was an incredible mess.

(I didn't actually have an Intel DIMM module -- I had a different vendor's. But fundamentally there needed to be a clean way to get data from software into the durable memory domain, and there wasn't. There also needed to be a clean way to talk to the module to get health information and to configure it to the extent it was configurable, and that didn't exist either. I wrote the kernel driver for the latter and submitted it upstream, and it probably could have gotten merged if Intel had assisted with the gnarly bits involving coordination with platform thermal firmware that shared the same SMBUS lines, but they didn't.)

[–] Silagi 3 days ago
That's the persistent memory modules; they were roughly half the price per GB compared to DDR4 from my quick research, but substantially higher memory latency, which would significantly affect usage as a KV cache, which would probably be their primary use case since SSDs beat it in raw data throughput. Who knows how it would have developed over the years since it was discontinued, but my gut reaction to the numbers said you'd rather just invest in more RAM based on the numbers I saw.