Hacker News
points by weitendorf 3 days ago | 0 comments
TANGENT WARNING: What bothers me is that unless you are directly plumbing data on disk to the wire, there is literally no benefit to serializing data this way. (I originally realized all this when trying to add varint and byte types to postgres to make storage more efficient, and then finding it was almost impossible to actually serialize them in a way that saved space)

At runtime, if you've allocated a boolean or want to store a byte somewhere, you've either done it on the stack (it's temporary and you know it will need to be accessed at least once as an independent unit of data, and packing it together with other small data structures will make you use an extra instruction to extract it into its own register anyway, so there is not really any benefit to packing them tighter) or heap (you might actually gain something by reducing memory usage here, but packing data together binds the lifecycle and structure of everything at the same address and either requires additional lookups to check the interior structure, or a fancy compiler/build process to bake in the instruction to extract packed data without knowing what other data is packed with it).

When you read serialized data for a record from the disk, while running a typed database, the lifecycle/structure within the record is already shared. Most times you load the record you probably won't be accessing any particular packed data structure (eg if you have 5 bool columns in a table, most queries won't use all or any of them). You already have a global way to handle internal types/optimized reads, because that's what databases do.

So it would actually be feasible to load packed data and efficiently access or calculate with it eg XOR against 0x00..<single bit flipped to match bool index in word>00 to use a bool, or bit shift to load a subword into a register. It would be at worst 1 instruction per read slower if implemented efficiently (the query planner can handle the type lookup/deciding how to do the optimized read, which amortizes and isn't the query bottleneck or even on the critical path).

BUT denser data = more cache coherency and higher read throughput (more rows/s). And if you're doing fancy bit operations and arithmetic to extract and perform calculations on the packed data, it might actually be easier to batch operations with SIMD. So you could get arguably higher performance + much less passive disk usage (saving actual money) by packing data better probably. I'm about 30% of the way towards convincing myself to try this

Aún no hay comentarios.