Hacker News
points by ajb 1 day ago | 0 comments
Yeah, that's an interesting use case. The problem for you with a compiler would be how to get it to work with all of the efficient python modules like pandas, script, numpy, pytorch etc that all implement their backend in C with intrinsics/GPU usage.
[–] MOZGIII 12 hours ago
Well, this is something we've been, kind of, getting ready to tackle with some of the aspects of the internal design of the VM. In short, it would be pretty easy for us to just add an "instruction (sub)set" to work with GPU kernels or pytorch or etc - and then to implement it via whatever means fir best (a third-party crate, a GPU offload via vulkan or sth, heck - even spawning a python interpreter to handle some big operation where the overhead of python bringup is negligible in on the table).

The main issue so far is that our VM that does the workflow orchestration and the actions runtime are living in separate processes - but we even then, we could do tricks to share memory between them for more efficient / zero-copy data passing... That's not something we have right now though.

[–] msteffen 9 hours ago
I guess I could add:

With Pachyderm, part of our model was that the "actions" were arbitrary, user-provided containers[^1], and the inputs/outputs were mounted filesystems (`/pfs/my_first_input` was a directory containing the first input dataset, or one shard of it, etc, and `/pfs/out` was a writeable directory where the output for that shard would go[^2]). We built the (versioned) filesystem that data in /pfs came from/went to, but you could probably use ceph or something if you're not trying to do versioning. That's how we dealt with big inputs/outputs.

We never tried to optimize the single-machine case, for better or worse.

[^1]: Using arbitrary containers was a good choice for us, though, because we were marketing to spark users and a lot of them were frustrated with how hard worker-dependency management is in spark. So we got to say "put perl, weird sequence analysis libraries, whatever you want, in a container, and if it can read and write files, we'll mapreduce it over your data".

[^2]: `/pfs` was a terrible choice because it's nonstandard. Really wish we'd used `/mnt` or `/mnt/pfs` instead)