Introduction to mizu

mizu provides parallel computation and data exchange between R processes on the same machine: lock-free channels and work-stealing task pools over POSIX shared memory (Linux, macOS) or Win32 file mappings (Windows).

A channel is a two-way message link between an R session and a helper process that it spawns. A pool is a set of worker processes that divide submitted tasks among themselves.

In both, one process writes data into shared memory and the other reads it in place, with no copy through a socket, pipe, or file.

library(mizu)

p <- mizu_pool(n_workers = 4L)
t <- mizu_submit(p, sum(x) + y, x = 1:10, y = 100)
mizu_collect(t)
#> [1] 155
m <- mizu_map(p, 1:5, \(i) i * 2L, .template = integer(1))
mizu_collect(m)
#> [1]  2  4  6  8 10
mizu_pool_stop(p)

The articles

• Benchmarks — measurements against mirai.
• Channels — two-way message links between R processes: send and receive, batched forms, sentinel values, remote errors.
• Task pools — work-stealing worker pools: submit and collect, nested tasks, waiting on several tasks, sizing and observing a pool.
• Parallel map — mizu_map(): mapping over a pool with the function and data staged once, templates, reproducible random numbers, prepared maps.
• Python interop — channels and pools shared with Python processes through pymizu, including the cross-language map.
• Operations — sizing /dev/shm on Linux, the Linux memory allocator, and what happens when a process crashes.

mirror server hosted at Truenetwork, Russian Federation.