[–] MOZGIII 11 hours ago
[–] icyfox 1 day ago
As for async - background jobs are better modeled as async because of their natural parallelism across nodes imo. Logic in actions themselves are free to be as synchronous as you want. Why the preference?
[–] dangistan202611 1 day ago
Also, it is okay to run something in the foreground. Not everything needs to run in the background. Sometimes it's better to wait and have assurance of completion.
There is no natural parallelism of async code in Python. The coroutines run in a single thread.
[–] 9dev 1 day ago
This is a platform for running workflows distributed on multiple machines, sometimes long-running and with lots of IO. Every single contender in the space advertises the same code, from temporal to hatchet to dagster. Because it makes sense to write it this way.
[–] dangistan202611 1 day ago
Also, the claim made that something is "better modeled as async" is wholly unqualified.
[–] whilenot-dev 1 day ago
You'd use asyncio.to_thread[0].
I also don't quite get your complaint... asyncio provides a high-level concurrency primitive with coroutines, and that's a good enough primitive for workflow engines. Even with asyncio you could fork/spawn processes easily with aiomultiprocess[1], if you'd ever want or need to.
What's your use case for having more low-level concurrency primitives (threads/processes) available, having to deal with their synchronization, and running workflows sync?
[0]: https://docs.python.org/3/library/asyncio-task.html#asyncio....
[–] dangistan202611 15 hours ago
No such thing is typically necessary. If modeled correctly, the need for synchronization is minimal.
With sync code, tasks run on different threads/interpreters/processes. Message passing via queues is used to communicate which tasks need to run. Many asyncio users seem to wholly unaware of how well it works. Go-lang doesn't even have or need async.
I actually like asyncio; I am just not a religious zealot about it as many here seem to be, also without awareness of the struggles they face with loop-hogging coroutines and multicore parallelism.
With the way our current Python workers are implemented you'd have we'd have to place the sync code onto threads manually.
We could make it happen though, if there is a demand for sync code; we can allow actions to be non-async, and from that run them in sync mode - either with an sdk-level offload to threads off-of async reactor, or on a separate worker pools that run fork-per-action. From the perspective of our VM, we don't really have a difference there...