Materialize is replacing Linux kernel swap with an application-managed buffer pool to handle out-of-core data storage, exploiting the asymmetry between random access latency and sequential scan bandwidth between DRAM and NVMe. This shift from kernel-managed paging to explicit buffer pool management will enable the system to optimize for data transfer rather than CPU cycles while maintaining control over eviction policy.
RapidsMPF is an out-of-core shuffling system that enables data shuffling at 1.8 TiB/s by spilling to disk instead of causing out-of-memory errors. Shuffling is a critical but expensive operation in distributed data analytics, particularly for joins, groupby, and sort operations, as it requires moving all data between processes while managing memory constraints and synchronization barriers.