1 .. SPDX-License-Identifier: GPL-2.0
7 The mmu (in arch/x86/kvm, files mmu.[ch] and paging_tmpl.h) is responsible
8 for presenting a standard x86 mmu to the guest, while translating guest
9 physical addresses to host physical addresses.
11 The mmu code attempts to satisfy the following requirements:
14 the guest should not be able to determine that it is running
15 on an emulated mmu except for timing (we attempt to comply
16 with the specification, not emulate the characteristics of
17 a particular implementation such as tlb size)
19 the guest must not be able to touch host memory not assigned
22 minimize the performance penalty imposed by the mmu
24 need to scale to large memory and large vcpu guests
26 support the full range of x86 virtualization hardware
28 Linux memory management code must be in control of guest memory
29 so that swapping, page migration, page merging, transparent
30 hugepages, and similar features work without change
32 report writes to guest memory to enable live migration
33 and framebuffer-based displays
35 keep the amount of pinned kernel memory low (most memory
38 avoid multipage or GFP_ATOMIC allocations
43 ==== ====================================================================
44 pfn host page frame number
45 hpa host physical address
46 hva host virtual address
47 gfn guest frame number
48 gpa guest physical address
49 gva guest virtual address
50 ngpa nested guest physical address
51 ngva nested guest virtual address
52 pte page table entry (used also to refer generically to paging structure
54 gpte guest pte (referring to gfns)
55 spte shadow pte (referring to pfns)
56 tdp two dimensional paging (vendor neutral term for NPT and EPT)
57 ==== ====================================================================
59 Virtual and real hardware supported
60 ===================================
62 The mmu supports first-generation mmu hardware, which allows an atomic switch
63 of the current paging mode and cr3 during guest entry, as well as
64 two-dimensional paging (AMD's NPT and Intel's EPT). The emulated hardware
65 it exposes is the traditional 2/3/4 level x86 mmu, with support for global
66 pages, pae, pse, pse36, cr0.wp, and 1GB pages. Emulated hardware also
67 able to expose NPT capable hardware on NPT capable hosts.
72 The primary job of the mmu is to program the processor's mmu to translate
73 addresses for the guest. Different translations are required at different
76 - when guest paging is disabled, we translate guest physical addresses to
77 host physical addresses (gpa->hpa)
78 - when guest paging is enabled, we translate guest virtual addresses, to
79 guest physical addresses, to host physical addresses (gva->gpa->hpa)
80 - when the guest launches a guest of its own, we translate nested guest
81 virtual addresses, to nested guest physical addresses, to guest physical
82 addresses, to host physical addresses (ngva->ngpa->gpa->hpa)
84 The primary challenge is to encode between 1 and 3 translations into hardware
85 that support only 1 (traditional) and 2 (tdp) translations. When the
86 number of required translations matches the hardware, the mmu operates in
87 direct mode; otherwise it operates in shadow mode (see below).
92 Guest memory (gpa) is part of the user address space of the process that is
93 using kvm. Userspace defines the translation between guest addresses and user
94 addresses (gpa->hva); note that two gpas may alias to the same hva, but not
97 These hvas may be backed using any method available to the host: anonymous
98 memory, file backed memory, and device memory. Memory might be paged by the
104 The mmu is driven by events, some from the guest, some from the host.
106 Guest generated events:
108 - writes to control registers (especially cr3)
109 - invlpg/invlpga instruction execution
110 - access to missing or protected translations
112 Host generated events:
114 - changes in the gpa->hpa translation (either through gpa->hva changes or
115 through hva->hpa changes)
116 - memory pressure (the shrinker)
121 The principal data structure is the shadow page, 'struct kvm_mmu_page'. A
122 shadow page contains 512 sptes, which can be either leaf or nonleaf sptes. A
123 shadow page may contain a mix of leaf and nonleaf sptes.
125 A nonleaf spte allows the hardware mmu to reach the leaf pages and
126 is not related to a translation directly. It points to other shadow pages.
128 A leaf spte corresponds to either one or two translations encoded into
129 one paging structure entry. These are always the lowest level of the
130 translation stack, with optional higher level translations left to NPT/EPT.
131 Leaf ptes point at guest pages.
133 The following table shows translations encoded by leaf ptes, with higher-level
134 translations in parentheses:
139 paging: gva->gpa->hpa
140 paging, tdp: (gva->)gpa->hpa
144 non-tdp: ngva->gpa->hpa (*)
145 tdp: (ngva->)ngpa->gpa->hpa
147 (*) the guest hypervisor will encode the ngva->gpa translation into its page
148 tables if npt is not present
150 Shadow pages contain the following information:
152 The level in the shadow paging hierarchy that this shadow page belongs to.
153 1=4k sptes, 2=2M sptes, 3=1G sptes, etc.
155 If set, leaf sptes reachable from this page are for a linear range.
156 Examples include real mode translation, large guest pages backed by small
157 host pages, and gpa->hpa translations when NPT or EPT is active.
158 The linear range starts at (gfn << PAGE_SHIFT) and its size is determined
159 by role.level (2MB for first level, 1GB for second level, 0.5TB for third
160 level, 256TB for fourth level)
161 If clear, this page corresponds to a guest page table denoted by the gfn
164 When role.gpte_is_8_bytes=0, the guest uses 32-bit gptes while the host uses 64-bit
165 sptes. That means a guest page table contains more ptes than the host,
166 so multiple shadow pages are needed to shadow one guest page.
167 For first-level shadow pages, role.quadrant can be 0 or 1 and denotes the
168 first or second 512-gpte block in the guest page table. For second-level
169 page tables, each 32-bit gpte is converted to two 64-bit sptes
170 (since each first-level guest page is shadowed by two first-level
171 shadow pages) so role.quadrant takes values in the range 0..3. Each
172 quadrant maps 1GB virtual address space.
174 Inherited guest access permissions from the parent ptes in the form uwx.
175 Note execute permission is positive, not negative.
177 The page is invalid and should not be used. It is a root page that is
178 currently pinned (by a cpu hardware register pointing to it); once it is
179 unpinned it will be destroyed.
180 role.gpte_is_8_bytes:
181 Reflects the size of the guest PTE for which the page is valid, i.e. '1'
182 if 64-bit gptes are in use, '0' if 32-bit gptes are in use.
184 Contains the value of efer.nx for which the page is valid.
186 Contains the value of cr0.wp for which the page is valid.
188 Contains the value of cr4.smep && !cr0.wp for which the page is valid
189 (pages for which this is true are different from other pages; see the
190 treatment of cr0.wp=0 below).
192 Contains the value of cr4.smap && !cr0.wp for which the page is valid
193 (pages for which this is true are different from other pages; see the
194 treatment of cr0.wp=0 below).
196 Is 1 if the page is valid in system management mode. This field
197 determines which of the kvm_memslots array was used to build this
198 shadow page; it is also used to go back from a struct kvm_mmu_page
199 to a memslot, through the kvm_memslots_for_spte_role macro and
202 Is 1 if the MMU instance cannot use A/D bits. EPT did not have A/D
203 bits before Haswell; shadow EPT page tables also cannot use A/D bits
204 if the L1 hypervisor does not enable them.
206 Either the guest page table containing the translations shadowed by this
207 page, or the base page frame for linear translations. See role.direct.
209 A pageful of 64-bit sptes containing the translations for this page.
210 Accessed by both kvm and hardware.
211 The page pointed to by spt will have its page->private pointing back
212 at the shadow page structure.
213 sptes in spt point either at guest pages, or at lower-level shadow pages.
214 Specifically, if sp1 and sp2 are shadow pages, then sp1->spt[n] may point
215 at __pa(sp2->spt). sp2 will point back at sp1 through parent_pte.
216 The spt array forms a DAG structure with the shadow page as a node, and
217 guest pages as leaves.
219 An array of 512 guest frame numbers, one for each present pte. Used to
220 perform a reverse map from a pte to a gfn. When role.direct is set, any
221 element of this array can be calculated from the gfn field when used, in
222 this case, the array of gfns is not allocated. See role.direct and gfn.
224 A counter keeping track of how many hardware registers (guest cr3 or
225 pdptrs) are now pointing at the page. While this counter is nonzero, the
226 page cannot be destroyed. See role.invalid.
228 The reverse mapping for the pte/ptes pointing at this page's spt. If
229 parent_ptes bit 0 is zero, only one spte points at this page and
230 parent_ptes points at this single spte, otherwise, there exists multiple
231 sptes pointing at this page and (parent_ptes & ~0x1) points at a data
232 structure with a list of parent sptes.
234 If true, then the translations in this page may not match the guest's
235 translation. This is equivalent to the state of the tlb when a pte is
236 changed but before the tlb entry is flushed. Accordingly, unsync ptes
237 are synchronized when the guest executes invlpg or flushes its tlb by
238 other means. Valid for leaf pages.
240 How many sptes in the page point at pages that are unsync (or have
241 unsynchronized children).
243 A bitmap indicating which sptes in spt point (directly or indirectly) at
244 pages that may be unsynchronized. Used to quickly locate all unsychronized
245 pages reachable from a given page.
247 Only present on 32-bit hosts, where a 64-bit spte cannot be written
248 atomically. The reader uses this while running out of the MMU lock
249 to detect in-progress updates and retry them until the writer has
251 write_flooding_count:
252 A guest may write to a page table many times, causing a lot of
253 emulations if the page needs to be write-protected (see "Synchronized
254 and unsynchronized pages" below). Leaf pages can be unsynchronized
255 so that they do not trigger frequent emulation, but this is not
256 possible for non-leafs. This field counts the number of emulations
257 since the last time the page table was actually used; if emulation
258 is triggered too frequently on this page, KVM will unmap the page
259 to avoid emulation in the future.
264 The mmu maintains a reverse mapping whereby all ptes mapping a page can be
265 reached given its gfn. This is used, for example, when swapping out a page.
267 Synchronized and unsynchronized pages
268 =====================================
270 The guest uses two events to synchronize its tlb and page tables: tlb flushes
271 and page invalidations (invlpg).
273 A tlb flush means that we need to synchronize all sptes reachable from the
274 guest's cr3. This is expensive, so we keep all guest page tables write
275 protected, and synchronize sptes to gptes when a gpte is written.
277 A special case is when a guest page table is reachable from the current
278 guest cr3. In this case, the guest is obliged to issue an invlpg instruction
279 before using the translation. We take advantage of that by removing write
280 protection from the guest page, and allowing the guest to modify it freely.
281 We synchronize modified gptes when the guest invokes invlpg. This reduces
282 the amount of emulation we have to do when the guest modifies multiple gptes,
283 or when the a guest page is no longer used as a page table and is used for
286 As a side effect we have to resynchronize all reachable unsynchronized shadow
287 pages on a tlb flush.
293 - guest page fault (or npt page fault, or ept violation)
295 This is the most complicated event. The cause of a page fault can be:
297 - a true guest fault (the guest translation won't allow the access) (*)
298 - access to a missing translation
299 - access to a protected translation
300 - when logging dirty pages, memory is write protected
301 - synchronized shadow pages are write protected (*)
302 - access to untranslatable memory (mmio)
304 (*) not applicable in direct mode
306 Handling a page fault is performed as follows:
308 - if the RSV bit of the error code is set, the page fault is caused by guest
309 accessing MMIO and cached MMIO information is available.
311 - walk shadow page table
312 - check for valid generation number in the spte (see "Fast invalidation of
314 - cache the information to vcpu->arch.mmio_gva, vcpu->arch.mmio_access and
315 vcpu->arch.mmio_gfn, and call the emulator
317 - If both P bit and R/W bit of error code are set, this could possibly
318 be handled as a "fast page fault" (fixed without taking the MMU lock). See
319 the description in Documentation/virt/kvm/locking.rst.
321 - if needed, walk the guest page tables to determine the guest translation
322 (gva->gpa or ngpa->gpa)
324 - if permissions are insufficient, reflect the fault back to the guest
326 - determine the host page
328 - if this is an mmio request, there is no host page; cache the info to
329 vcpu->arch.mmio_gva, vcpu->arch.mmio_access and vcpu->arch.mmio_gfn
331 - walk the shadow page table to find the spte for the translation,
332 instantiating missing intermediate page tables as necessary
334 - If this is an mmio request, cache the mmio info to the spte and set some
335 reserved bit on the spte (see callers of kvm_mmu_set_mmio_spte_mask)
337 - try to unsynchronize the page
339 - if successful, we can let the guest continue and modify the gpte
341 - emulate the instruction
343 - if failed, unshadow the page and let the guest continue
345 - update any translations that were modified by the instruction
349 - walk the shadow page hierarchy and drop affected translations
350 - try to reinstantiate the indicated translation in the hope that the
351 guest will use it in the near future
353 Guest control register updates:
357 - look up new shadow roots
358 - synchronize newly reachable shadow pages
360 - mov to cr0/cr4/efer
362 - set up mmu context for new paging mode
363 - look up new shadow roots
364 - synchronize newly reachable shadow pages
366 Host translation updates:
368 - mmu notifier called with updated hva
369 - look up affected sptes through reverse map
370 - drop (or update) translations
375 If tdp is not enabled, the host must keep cr0.wp=1 so page write protection
376 works for the guest kernel, not guest guest userspace. When the guest
377 cr0.wp=1, this does not present a problem. However when the guest cr0.wp=0,
378 we cannot map the permissions for gpte.u=1, gpte.w=0 to any spte (the
379 semantics require allowing any guest kernel access plus user read access).
381 We handle this by mapping the permissions to two possible sptes, depending
384 - kernel write fault: spte.u=0, spte.w=1 (allows full kernel access,
385 disallows user access)
386 - read fault: spte.u=1, spte.w=0 (allows full read access, disallows kernel
389 (user write faults generate a #PF)
391 In the first case there are two additional complications:
393 - if CR4.SMEP is enabled: since we've turned the page into a kernel page,
394 the kernel may now execute it. We handle this by also setting spte.nx.
395 If we get a user fetch or read fault, we'll change spte.u=1 and
396 spte.nx=gpte.nx back. For this to work, KVM forces EFER.NX to 1 when
397 shadow paging is in use.
398 - if CR4.SMAP is disabled: since the page has been changed to a kernel
399 page, it can not be reused when CR4.SMAP is enabled. We set
400 CR4.SMAP && !CR0.WP into shadow page's role to avoid this case. Note,
401 here we do not care the case that CR4.SMAP is enabled since KVM will
402 directly inject #PF to guest due to failed permission check.
404 To prevent an spte that was converted into a kernel page with cr0.wp=0
405 from being written by the kernel after cr0.wp has changed to 1, we make
406 the value of cr0.wp part of the page role. This means that an spte created
407 with one value of cr0.wp cannot be used when cr0.wp has a different value -
408 it will simply be missed by the shadow page lookup code. A similar issue
409 exists when an spte created with cr0.wp=0 and cr4.smep=0 is used after
410 changing cr4.smep to 1. To avoid this, the value of !cr0.wp && cr4.smep
411 is also made a part of the page role.
416 The mmu supports all combinations of large and small guest and host pages.
417 Supported page sizes include 4k, 2M, 4M, and 1G. 4M pages are treated as
418 two separate 2M pages, on both guest and host, since the mmu always uses PAE
421 To instantiate a large spte, four constraints must be satisfied:
423 - the spte must point to a large host page
424 - the guest pte must be a large pte of at least equivalent size (if tdp is
425 enabled, there is no guest pte and this condition is satisfied)
426 - if the spte will be writeable, the large page frame may not overlap any
427 write-protected pages
428 - the guest page must be wholly contained by a single memory slot
430 To check the last two conditions, the mmu maintains a ->disallow_lpage set of
431 arrays for each memory slot and large page size. Every write protected page
432 causes its disallow_lpage to be incremented, thus preventing instantiation of
433 a large spte. The frames at the end of an unaligned memory slot have
434 artificially inflated ->disallow_lpages so they can never be instantiated.
436 Fast invalidation of MMIO sptes
437 ===============================
439 As mentioned in "Reaction to events" above, kvm will cache MMIO
440 information in leaf sptes. When a new memslot is added or an existing
441 memslot is changed, this information may become stale and needs to be
442 invalidated. This also needs to hold the MMU lock while walking all
443 shadow pages, and is made more scalable with a similar technique.
445 MMIO sptes have a few spare bits, which are used to store a
446 generation number. The global generation number is stored in
447 kvm_memslots(kvm)->generation, and increased whenever guest memory info
450 When KVM finds an MMIO spte, it checks the generation number of the spte.
451 If the generation number of the spte does not equal the global generation
452 number, it will ignore the cached MMIO information and handle the page
453 fault through the slow path.
455 Since only 18 bits are used to store generation-number on mmio spte, all
456 pages are zapped when there is an overflow.
458 Unfortunately, a single memory access might access kvm_memslots(kvm) multiple
459 times, the last one happening when the generation number is retrieved and
460 stored into the MMIO spte. Thus, the MMIO spte might be created based on
461 out-of-date information, but with an up-to-date generation number.
463 To avoid this, the generation number is incremented again after synchronize_srcu
464 returns; thus, bit 63 of kvm_memslots(kvm)->generation set to 1 only during a
465 memslot update, while some SRCU readers might be using the old copy. We do not
466 want to use an MMIO sptes created with an odd generation number, and we can do
467 this without losing a bit in the MMIO spte. The "update in-progress" bit of the
468 generation is not stored in MMIO spte, and is so is implicitly zero when the
469 generation is extracted out of the spte. If KVM is unlucky and creates an MMIO
470 spte while an update is in-progress, the next access to the spte will always be
471 a cache miss. For example, a subsequent access during the update window will
472 miss due to the in-progress flag diverging, while an access after the update
473 window closes will have a higher generation number (as compared to the spte).
479 - NPT presentation from KVM Forum 2008
480 https://www.linux-kvm.org/images/c/c8/KvmForum2008%24kdf2008_21.pdf