32-bit virtual memory was added to the x86 platform in 1986 with the Intel 80386 processor. It took Microsoft almost ten years to ship a volume operating system that utilized this feature, but several operating systems that supported isolated, flat 32-bit address spaces, such as UNIX, were ported in short order. And one striking thing about virtual address spaces is their sheer pervasiveness: Every memory access by a program running under a modern operating system specifies a virtual address, and that virtual address must be translated to a physical one. This address translation operation is expensive: on x86-64, a four-level radix tree of page tables is rooted at the physical address in the CR3 register, with a different nine bits of the virtual address indexing each level; the entry found at each level supplies the physical address of the next, and the entry at the last level supplies the physical page frame, along with the permission bits that determine whether the access is allowed at all. Computing a translation with a full page walk requires four memory references (five, with the five-level paging that supports virtual address spaces larger than 256 TB), and they are dependent: each reference supplies the address for the next, so no amount of memory-level parallelism can overlap them.
The translation lookaside buffer (TLB) caches the output of that function. A virtual page number goes in; a physical frame number and its permissions come out. It is the oldest of the covert caches, and in the x86 lineage it is older than the data cache itself: Intel’s 80386, introduced in 1985, translated addresses through an on-chip TLB of 32 entries, but the first x86 processor with a cache on the chip was the 80486, announced in 1989.1 (Many 80386 systems did have a data cache, on the motherboard, often managed by Intel’s 82385 cache controller, which its datasheet advertised as “software transparent,” with “bus watching” to maintain coherency.) Intel’s 1986 manual for the 80386 called the TLB a “page-translation cache,” and stated the division of labor it has imposed ever since: “The existence of the page-translation cache is invisible to applications programmers but not to systems programmers; operating-system programmers must flush the cache whenever the page tables are changed.”2 The TLB is a covert cache in the plainest sense, since the thing it stores appears nowhere in memory. The page tables contain the ingredients of a translation, scattered across four cache lines in four different pages; the TLB contains the translation.
Modern implementations also memoize intermediate results: Intel’s documentation describes paging-structure caches that hold entries from the upper levels of the tree, so that a TLB miss on an address near one that was recently translated can skip the first two or three levels of the walk and begin partway down3. The walk itself is performed by a hardware state machine (Intel calls it the page miss handler) that issues its references through the ordinary data cache hierarchy, so the cost of a TLB miss ranges from a few dozen cycles, if every page table entry it needs is in L1, to several trips to DRAM if none are resident.
Virtualization makes the function even more expensive, so memoization via the TLB becomes a commensurately more profitable optimization. Under nested paging (EPT on Intel, NPT on AMD), the guest’s page tables contain guest-physical addresses, each of which must itself be translated through the host’s page tables, so a two-dimensional walk of four-level tables on both sides can take up to 24 memory references. A TLB hit elides all of them.
None of this means the cache waits for the TLB. A first-level data cache is indexed with bits that translation does not change: AMD specifies Zen 4’s as “bits [11:6] of the linear address,” which lie below the 4 KB page offset and are therefore identical in the virtual address and the physical one4. The cache reads its set while the TLB translates, and the translation is wanted only at the end, to check against the physical tag stored with the line. That arrangement puts the TLB inside the latency of every load that hits, and not only of the ones that miss, and it bounds the cache’s shape: with 64-byte lines, six bits of page offset remain to index with, which allows 64 sets and no more, so a first-level cache grows by adding ways rather than sets. Zen 4’s 32-Kbyte data cache is eight-way for that reason, and reading eight tags to find one match costs enough power that AMD guesses the way first, from a hash of the linear address, before the physical tag exists to check the guess.
The other difference between the TLB and a data cache is who is responsible for keeping it correct. x86 keeps its data caches coherent in hardware, across cores and with respect to device DMA, so a program never has to tell the processor that a cached line has gone stale5. The TLB enjoys no such guarantee. When the operating system modifies a page table entry (to unmap a page, change its permissions, or migrate it to a different physical frame), the TLB entries derived from the old value remain valid in the hardware’s estimation until software explicitly invalidates them. On the core that made the change, that takes one instruction, invlpg. On every other core that might be caching the translation, it performs a TLB shootdown: the kernel sends an inter-processor interrupt to each core, the interrupted core invalidates its entry and acknowledges, and the initiating core waits for every acknowledgment before it can safely reuse the physical page. On a large machine running a multithreaded process, shootdowns are a noticeable cost of munmap and mprotect, and one of the reasons memory allocators are reluctant to return memory to the operating system.
ARM took the opposite position. Its TLB invalidation instructions can be marked Inner Shareable, in which case the hardware broadcasts the invalidation to the other cores in the coherence domain, and no interrupt is needed. x86 has since acquired broadcast invalidation from both vendors, in incompatible forms: AMD’s invlpgb instruction, starting with Zen 3, and Intel’s Remote Action Request, starting with Sapphire Rapids. Linux began using AMD’s in 2025, and patches to use Intel’s have been proposed.
Context switches present the same problem in bulk. Loading a new value into CR3 historically flushed every TLB entry not marked global, since those translations belonged to the address space being switched out. Process-context identifiers (PCIDs on x86, ASIDs on ARM) tag each entry with the address space that created it, so entries can survive a switch and be reused when the process is scheduled again. PCIDs sat mostly unused in Linux for years, until the mitigation for Meltdown (kernel page-table isolation) began switching page tables on every system call and interrupt, and made the cost of an untagged TLB impossible to ignore.
x86 has another translation cache of the same kind, and this one is openly called a cache by Intel. Every segment register has a visible part, holding the selector the program loaded, and a hidden part that the manual calls “a descriptor cache or a shadow register.”6 Loading a selector copies the segment’s base address, limit, and access information out of the descriptor table into that hidden part, which is there to avoid “extra bus cycles to read the base address and limit from the segment descriptor” on every access. Nothing refreshes it afterward, and Intel states the consequence in the same terms the 80386 manual used for page tables: “it is the responsibility of software to reload the segment registers when the descriptor tables are modified,” failing which “an old segment descriptor cached in a segment register might be used after its memory-resident version has been modified.”
The TLB’s capacity problem is also different in character from that of a data cache. A Skylake core’s first-level data TLB holds 64 entries for 4 KB pages, which cover 256 KB of memory, and its 1,536-entry second-level TLB covers 6 MB, less than the L3 cache of many of the chips it shipped in. A program whose working set fits in cache therefore can still miss in the TLB on every access, which is why huge pages matter so much to programs with large working sets: a 2 MB page, mapped one level higher in the page table tree, covers 512 times as much memory per TLB entry as a 4 KB page7. The data cache hierarchy has no analogue for TLB reach, which depends on how the operating system chose to map memory, not merely on how much memory the program touches.
Huge pages are not the default, because they are not free. A 2 MB page must be backed by 2 MB of physically contiguous, 2 MB-aligned memory, and a long-running system fragments its physical memory into smaller pieces. Linux’s transparent huge pages (THP), which since 2011 have let the kernel back ordinary memory with huge pages without the program’s asking, therefore must either find such a block when a page fault occurs, compact memory to create one, or fall back to small pages and let a background thread (khugepaged) collapse them into a huge page later. Each choice has a cost: stalls while memory is compacted, memory wasted when a 2 MB page is committed to a region the program barely touches, and the latency of zeroing 2 MB on a single fault. Several databases have recommended disabling THP for exactly those reasons. Virtualization adds a further condition, since a huge page in the guest yields a huge TLB entry only if the host also backs it with a huge page. When it does not, AMD’s documentation describes the TLB entry as smashed, broken down to the smaller of the two page sizes8.
Between the 4 KB page and the 2 MB page lies a middle ground, which hardware and operating systems have been approaching from opposite directions. A TLB can exploit physical contiguity without any change to the architecture, by noticing that several consecutive virtual pages happen to map to consecutive physical pages, and storing them as a single entry. Academic researchers proposed that in 2012, as coalesced large-reach TLBs, observing that ordinary operating system allocators produce such runs of contiguous pages more often than one might expect9, and AMD’s Zen cores implement a version of it: “If a 16-Kbyte aligned block of four consecutive 4-Kbyte pages are also consecutive and 16-Kbyte aligned in physical address space and have identical page attributes, the processor may opportunistically store them in a single TLB entry.”10 Zen 4’s first-level data TLB accordingly lists 16 KB entries alongside its 4 KB, 2 MB, and 1 GB entries, although no x86 page table can describe a 16 KB page. ARM made the idea architectural instead. Its page table entries have a contiguous bit, with which the operating system asserts that an aligned run of entries (sixteen of them, with 4 KB pages) maps physically contiguous memory with identical attributes, so that the TLB may cache the whole 64 KB run as one entry.
Either mechanism pays off only if the operating system arranges for virtually contiguous pages to be aligned and physically contiguous, and that has been the focus of recent work. In 2024, Linux gained multi-size THP (mTHP), which backs memory with naturally aligned, physically contiguous blocks of 16 KB, 32 KB, 64 KB, and so on, while still mapping them with ordinary 4 KB page table entries; the kernel’s documentation notes that “some architectures also employ TLB compression mechanisms to squeeze more entries in when a set of PTEs are virtually and physically contiguous and appropriately aligned.” On ARM, Linux began setting the contiguous bit automatically for such blocks the same year, with the stated aim that a kernel using 4 KB pages “approach the performance of the 16K kernel, but without breaking compatibility.”11 On AMD, the same blocks are exactly what the coalescing hardware looks for, and it needs no cooperation from the kernel beyond the allocation itself. In this respect the TLB is a cache whose hit rate is controlled more by the operating system than by the program.
How to Detect It
Names are Intel’s for Skylake and Ice Lake, and AMD’s for Zen 4.
- Page walks for loads and stores, and the cycles spent walking. - Intel: - DTLB_LOAD_MISSES.WALK_COMPLETED,- DTLB_STORE_MISSES.WALK_COMPLETED,- DTLB_LOAD_MISSES.WALK_ACTIVE
- AMD: - ls_l1_d_tlb_miss.all_l2_miss
- Page walks for instruction fetch. - Intel: - ITLB_MISSES.WALK_COMPLETED,- FRONTEND_RETIRED.ITLB_MISS
- AMD: - bp_l1_tlb_miss_l2_tlb_miss.all
AMD’s cores have no event for cycles spent walking page tables, so DTLB_LOAD_MISSES.WALK_ACTIVE has no counterpart. The two AMD events above are Zen 4’s, and neither reads back unchanged on earlier cores:
- The instruction-side event is renamed. - Zen 2 and Zen 3: - bp_l1_tlb_miss_l2_tlb_miss, with no suffix
- Zen 4: - bp_l1_tlb_miss_l2_tlb_miss.all
- The data-side event does not exist before Zen 4. Zen 4’s - ls_l1_d_tlb_miss.all_l2_missis an aggregate that Zen 2 and Zen 3 lack, so on those cores the same quantity is the sum of four events,- ls_l1_d_tlb_miss.tlb_reload_4k_l2_missand its 2 MB, 1 GB, and coalesced-page counterparts.
Shootdowns are better observed without the performance counters. They appear on x86 Linux as the “TLB” row of /proc/interrupts, which counts the number of shootdown interrupts received by each CPU.
Working Around It
Increase TLB coverage. When page walks dominate, the most effective remedy is usually huge pages, which multiply the TLB’s reach by 512 without changing a single line of the program’s logic. On Linux, transparent huge pages can be requested for a region with madvise(MADV_HUGEPAGE), or reserved explicitly through hugetlbfs, and 1 GB pages are suitable for large, long-lived tables. Where 2 MB pages cost too much in fragmentation or latency, the intermediate mTHP sizes (enabled per size, through /sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled) give coalescing TLBs the contiguity they need at a fraction of the cost. Code can benefit too, since an instruction TLB covers a large server binary poorly, and some large deployments remap their hot code onto huge pages at startup.
TLB shootdowns call for a different remedy, which is to change page tables less often: batch calls to munmap, avoid changing page permissions in a hot path (a JIT compiler that flips a page between writable and executable for every function it emits pays for shootdowns as it goes), and tune memory allocators to return memory to the operating system less eagerly.
Further Reading
- Ryan Roberts, Transparent Contiguous PTEs for User Mappings — The kernel side of TLB coalescing, and a clear account of why physical contiguity is hard to come by.
- Ben Gras and coauthors, Translation Leak-aside Buffer — TLBleed, which recovers a cryptographic key from TLB timing alone, on a machine whose caches are isolated.
The 80386's TLB was four-way set-associative, with eight entries in each way; its manual (section 10.6.1) describes "four sets of eight entries each." The pattern is older than the x86. IBM's System/360 Model 67, which shipped in 1966, performed dynamic address translation with the aid of eight associative registers that held recently used translations, several years before the Model 85 shipped its cache.
Intel 80386 Programmer's Reference Manual, 1986, section 5.2.5, "Page Translation Cache." The 82385 quotations are from Intel's advance datasheet, 82385 High Performance 32-Bit Cache Controller, July 1987.
Intel 64 and IA-32 Architectures Software Developer's Manual, Volume 3A, section 4.10.3, "Paging-Structure Caches."
Software Optimization Guide for the AMD Zen4 Microarchitecture, publication 57647, section 2.6.2.3 for the index bits and the microtag, and section 2.6.2 for the cache's size, associativity, and its four to five cycle integer load-to-use latency. The arrangement has a name, virtually indexed, physically tagged, and its awkward consequence is that one physical line reached through two virtual addresses can be indexed twice, which the hardware must then prevent or detect.
A few unprivileged instructions can flush cache lines: clflush, and later clflushopt and clwb, take a memory operand and evict that line from every level of the hierarchy. What a program never has to do is flush for the sake of coherence, which is maintained whether or not it asks. The per-line instructions were added for participants not accounted for by the coherence protocol, such as persistent memory and devices performing non-coherent DMA. These unprivileged instructions also enabled the Flush+Reload side channel attack, as described by Yarom and Falkner at USENIX Security 2014.
Intel 64 and IA-32 Architectures Software Developer's Manual, Volume 3A, section 3.4.3, "Segment Registers," and Figure 3-7, which draws the visible and hidden parts of each segment register side by side. The reload requirement is stated in the same section.
x86-64 also supports 1 GB pages, mapped one level higher still, each covering 262,144 times as much memory as a 4 KB page. They are provisioned differently and cached differently. Linux does not create them transparently; they are reserved through hugetlbfs, most reliably with the hugepagesz=1G hugepages=N parameters on the kernel command line, because at boot, in the words of the kernel's documentation, "memory has not yet become fragmented." And TLBs cannot hold many of them: a Skylake core can hold four first-level and 16 second-level data TLB entries for 1 GB pages, and Zen 4 can hold them only in its 72-entry first-level data TLB, never in its 3,072-entry second level. They suit large, long-lived allocations whose size is known in advance, such as database buffer pools, network packet buffers, and the memory that backs virtual machines.
Software Optimization Guide for AMD Family 17h Processors, publication 55723, revision 3.00, June 2017, section 1.2: "when the Family 17h processor encounters a larger page size in the guest page tables which is backed by a smaller page in the host page tables, it will smash translations of the larger page size into the smaller page size found in the host."
Binh Pham, Viswanathan Vaidyanathan, Aamer Jaleel, and Abhishek Bhattacharjee, "CoLT: Coalesced Large-Reach TLBs," MICRO 2012.
Software Optimization Guide for the AMD Zen4 Microarchitecture, publication 57647, revision 1.01, April 2023, section 2.7.1. The feature predates Zen 4: the Linux perf event list for Zen 2 includes an event that counts L1 TLB misses satisfied by "a coalesced page."
Ryan Roberts, cover letter for the patch series "Transparent Contiguous PTEs for User Mappings," December 2023. The mTHP quotation is from the kernel's documentation, Documentation/admin-guide/mm/transhuge.rst.