Engram extends token embeddings with learned multi-token lookups to reduce computational overhead, enabling efficient DRAM/SSD offloading for large language models. The technique allows model architects to work within HBM constraints by prefetching embedding rows from slower memory tiers while computation proceeds. Benchmark results across NVIDIA and AMD GPUs show offloading embeddings to DRAM can improve performance compared to keeping them in HBM.
Engram extends token embeddings with learned multi-token lookups to reduce computational overhead in large language models. The architecture enables efficient memory offloading from HBM to DRAM and SSD, allowing larger models to run on constrained hardware. Researchers validated Engram across multiple GPU architectures and inference frameworks, finding that offloading embeddings to DRAM can improve performance even on high-capacity systems.