Hacker News

points by FuriouslyAdrift 3 days ago | 0 comments

We're running Kimi 2.8 on a 4x AMD MI300A single server on-premise that cost about $110k.

We will break even from our old Claude spend in about 2 weeks.

It services 30-40 analysts and staff concurrently and is faster than any of the SaaS out there for us.

Wow... I'd like to know more. 40 analysts not spending 100USD/month is 4k/month saved. That means you break even in 27 months. But you probably used Claude much more than that.

Also what harness do you use? Any problems you encountered on the way?

[–] FuriouslyAdrift 3 days ago

Just one of our analysts was burning around 30-40 million tokens per project.

EDIT: per project, not per month

EDIT 2: digging out an old bill from 8 months ago when we bought the server, $14k per month is what we were running and it was going up, a lot.

[–] Moto7451 3 days ago

Anthropic is very good at either convincing or forcing (I’m not sure which) companies to pay for tokens rather than for subscriptions. My old job moved from subs to tokens and everyone started running out of budget mid month. It was a bad move for a company that laid off all of QA and 2/3 of Engineering and committed to everyone making up for it with LLMs.

[–] mosselman 3 days ago

I don't get this, why do people pay for API tokens at all? What is the reasoning?

[–] ghshephard 3 days ago

I don't think Anthropic or OpenAI offer subscriptions to enterprises - it's a hobbyist/solo-entrepreneur kind of thing. I seem to recall SpaceXAI had some kind of hack going where they were trying to use subs with Anthropic and then got cut off.

Enterprises also need very stringent IP protections that you don't get through the subscriptions. Even the Anthropic-API per token license isn't great - sometimes you need to go to 3rd parties like AWS Bedrock to get guarantees regarding no multi-tenant GPUs.

[–] darkwater 3 days ago

You can have subscriptions up to 150 seats with Anthropic.

[–] ghshephard 2 days ago

Ah - so I was somewhat, but not totally wrong. They do offer subs to SMB (I did not know this!) - but for the Enterprise (typically in thousands to tens of thousands) directed through other channels.

[–] mattnewton 3 days ago

Zero Data Retention is the big one I think (and also the ToS don't allow multiple personal accounts for businesses anyways)

[–] dools 3 days ago

I have only ever paid for tokens and never used subscriptions because I need to know that I can generate sufficient real world value at the true cost of inference.

[–] vidarh 2 days ago

This seems like setting money on fire. Nothing stops you from tracking token use on your subscriptions and tracking how much you would spend via APIs for if/when you can't use a sub any more, and taking the savings in the meantime.

[–] dools 16 hours ago

If the stakes aren't real your behaviour won't change

[–] kay_o 3 days ago

You are forbidden from the subscriptions on enterprise

[–] bonoboTP 3 days ago

Claude Teams exists.

[–] KptMarchewa 3 days ago

No Zero Data Retention there.

[–] rat9988 3 days ago

They don't have the choice to begin with

[–] cute_boi 3 days ago

I think you should compare with api cost? I am using $200 plan and it is probably around $12k in token cost....

[–] codedokode 3 days ago

Are $12K worth of token guaranteed or the company may adjust token generation speeds?

[–] trentor 3 days ago

Oh boy... we are talking about insane token spends in corps at the moment. 100$ is a blip.

[–] StilesCrisis 3 days ago

Are they happy with Kimi 2.8's reasoning ability? I haven't tried it.

[–] komali2 3 days ago

Kimi is also a lot more willing to do whatever tf you ask it such as vulnerability scanning or PoC development. Or reverse engineering.

Plus on prem can go find abliterated models.

It's an advantage that keeps me coming back for local models when I can't get around Claude or Codex's little blockers.

[–] FuriouslyAdrift 3 days ago

It's worked out. Definitely not frontier but it's the largest model that works reliably on our hardware. We also try some smaller ones when they come out.

[–] LugosFergus 3 days ago

How bad did the electric bill get hit?

[–] FuriouslyAdrift 3 days ago

We have a data center already and we're in heavy industrial manufacturing. It didn't even register in our usage.

[–] edg5000 3 days ago

Very cool

[–] segmondy 3 days ago

There's no Kimi 2.8 as far as open weight is concerned. WTF. This must be a bot and other bots.

[–] nxdmum 3 days ago

If you take the kimi plan you would have seen kimi 2.8 preview alongside kimi 2.7

[–] FuriouslyAdrift 2 days ago

There was about 6 months ago... lol. We can't run 3.x, it's too big.

The 4x MI300A server has 128GB per card.

[–] gessha 3 days ago

Maybe they're talking about parameter count?