points by penagwin 3 days ago | 0 comments
> Aggregators like OpenCode Zen and OpenCode Go can shop around providers on these. OpenRouter shows you the costs of a bunch of the providers, and you can route to the cheapest ones.
Unfortunately due to their licensing requirements for modal-as-a-service providers, the pricing can’t go lower than Moonshot’s from what I can tell.
This is still great if like your example you trust together.ai more, or if different providers have better speeds/cache rates (though usually the original lab has the best cache hit rate).
And obviously if you’re a large enough service to afford to self host or finetune it then this is good.
[–] athrowaway3z 3 days ago
[–] penagwin 2 days ago
In my own experience the vast majority of llm consumers would prefer to pay by token, especially for large (300B+) as either their usage is below the point a dedicated cluster is viable or it’s just a lot more hands on.
With Kimi k3 specifically you need something like 1.6TB of memory which... Well you can’t just use 8x H100’s, and per token pricing becomes extremely attractive at this size for most users.
[–] knollimar 3 days ago
Aren’t these only financially viable by sharing weights in some big setup?
[–] joecot 3 days ago
Maybe for the API rate, but for OpenCode Go they're regularly making deals to get users a lot more than their money's worth on open source models.
[–] pigeons 3 days ago
Sometimes it can be so hard to tell and track because of the cache differences.
All the service provider needs to do is provide the option to rent out VM's with access to the inference cards.
Have the customer provide the model.
Maybe provide a bit of a speedup, by letting the customer mount the model by hash.
Sure, that all sounded way too technical 10 years ago, and somebody would have said that's already too much annoyance to deal with for average customers.
Reluctantly i'd agree.
That was 10 years ago. Nowadays, with some proper docs, this entire set up can be up and running in 5 minutes.