With Pachyderm, part of our model was that the "actions" were arbitrary, user-provided containers[^1], and the inputs/outputs were mounted filesystems (`/pfs/my_first_input` was a directory containing the first input dataset, or one shard of it, etc, and `/pfs/out` was a writeable directory where the output for that shard would go[^2]). We built the (versioned) filesystem that data in /pfs came from/went to, but you could probably use ceph or something if you're not trying to do versioning. That's how we dealt with big inputs/outputs.

We never tried to optimize the single-machine case, for better or worse.

[^1]: Using arbitrary containers was a good choice for us, though, because we were marketing to spark users and a lot of them were frustrated with how hard worker-dependency management is in spark. So we got to say "put perl, weird sequence analysis libraries, whatever you want, in a container, and if it can read and write files, we'll mapreduce it over your data".

[^2]: `/pfs` was a terrible choice because it's nonstandard. Really wish we'd used `/mnt` or `/mnt/pfs` instead)