The fastest possible group by is in sequential memory on that box, fetched from an nvme.
So, when we do group by partitioning, we're really saying, put the groups themselves on different boxes, and group the group by. If it were some utter megaset, the place to put it is on a mainframe, which is basically created for purposes of group by on ridiculous datasets.
[–] ozgrakkurt 1 day ago
[–] eru 1 day ago
I'm not sure ergonomics matter that much these days, when your agent can write the tedious bits?
[–] ozgrakkurt 16 hours ago
It is not just ergonomics but the point of using rust is to have solid abstraction and a lot of high quality libraries to use.
If you are doing static allocation + arenas then you can't use nice abstraction, you can't use libraries, you have to write a ton of unsafe code because of these reasons. Then using rust is actually working against you so there is no point in using it.
[–] sagarm 1 day ago
However horizontal scaling has advantages in that it's easier to scale up and down incrementally. It gives you more aggregate network bandwidth for accessing disaggregated data. Disaggregation allows for independent scaling of compute and storage. Losing a single machine has a marginal impact on the cluster, which can enable you to use cheaper spot resources.
Basically, it's often easier to operate a system designed around horizontal scaling if you need resiliency.
Measurements and context is so situational that it would be useless anyway imo.
It is not like any DB company is writing blog posts with benchmarks compared to hardware limits. DuckDB had blog posts that has sentences like "You are doing it wrong if you are not using HTTP api on your database in 2026", last time I saw their blog.
I also think object storage is bad in this case. And rust is no a no-no for a database project like this, because it is not ergonomic to do static allocation and arena allocators in rust. And parquet is trash for performance because of many reasons like supported compression formats etc.