[PATCH] kvm: userspace interface

web site: http://kvm.sourceforge.net

mailing list: kvm-devel@lists.sourceforge.net
  (http://lists.sourceforge.net/lists/listinfo/kvm-devel)

The following patchset adds a driver for Intel's hardware virtualization
extensions to the x86 architecture.  The driver adds a character device
(/dev/kvm) that exposes the virtualization capabilities to userspace.  Using
this driver, a process can run a virtual machine (a "guest") in a fully
virtualized PC containing its own virtual hard disks, network adapters, and
display.

Using this driver, one can start multiple virtual machines on a host.

Each virtual machine is a process on the host; a virtual cpu is a thread in
that process.  kill(1), nice(1), top(1) work as expected.  In effect, the
driver adds a third execution mode to the existing two: we now have kernel
mode, user mode, and guest mode.  Guest mode has its own address space mapping
guest physical memory (which is accessible to user mode by mmap()ing
/dev/kvm).  Guest mode has no access to any I/O devices; any such access is
intercepted and directed to user mode for emulation.

The driver supports i386 and x86_64 hosts and guests.  All combinations are
allowed except x86_64 guest on i386 host.  For i386 guests and hosts, both pae
and non-pae paging modes are supported.

SMP hosts and UP guests are supported.  At the moment only Intel
hardware is supported, but AMD virtualization support is being worked on.

Performance currently is non-stellar due to the naive implementation of the
mmu virtualization, which throws away most of the shadow page table entries
every context switch.  We plan to address this in two ways:

- cache shadow page tables across tlb flushes
- wait until AMD and Intel release processors with nested page tables

Currently a virtual desktop is responsive but consumes a lot of CPU.  Under
Windows I tried playing pinball and watching a few flash movies; with a recent
CPU one can hardly feel the virtualization.  Linux/X is slower, probably due
to X being in a separate process.

In addition to the driver, you need a slightly modified qemu to provide I/O
device emulation and the BIOS.

Caveats (akpm: might no longer be true):

- The Windows install currently bluescreens due to a problem with the
  virtual APIC.  We are working on a fix.  A temporary workaround is to
  use an existing image or install through qemu
- Windows 64-bit does not work.  That's also true for qemu, so it's
  probably a problem with the device model.

[bero@arklinux.org: build fix]
[simon.kagstrom@bth.se: build fix, other fixes]
[uril@qumranet.com: KVM: Expose interrupt bitmap]
[akpm@osdl.org: i386 build fix]
[mingo@elte.hu: i386 fixes]
[rdreier@cisco.com: add log levels to all printks]
[randy.dunlap@oracle.com: Fix sparse NULL and C99 struct init warnings]
[anthony@codemonkey.ws: KVM: AMD SVM: 32-bit host support]
Signed-off-by: Yaniv Kamay <yaniv@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
Cc: Simon Kagstrom <simon.kagstrom@bth.se>
Cc: Bernhard Rosenkraenzer <bero@arklinux.org>
Signed-off-by: Uri Lublin <uril@qumranet.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Signed-off-by: Anthony Liguori <anthony@codemonkey.ws>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>

1 files changed, 296 insertions, 0 deletions

diff --git a/drivers/kvm/vmx.h b/drivers/kvm/vmx.h
new file mode 100644
index 000000000000..797278341581
--- /dev/null
+++ b/drivers/kvm/vmx.h
@@ -0,0 +1,296 @@
+#ifndef VMX_H
+#define VMX_H
+
+/*
+ * vmx.h: VMX Architecture related definitions
+ * Copyright (c) 2004, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; if not, write to the Free Software Foundation, Inc., 59 Temple
+ * Place - Suite 330, Boston, MA 02111-1307 USA.
+ *
+ * A few random additions are:
+ * Copyright (C) 2006 Qumranet
+ *    Avi Kivity <avi@qumranet.com>
+ *    Yaniv Kamay <yaniv@qumranet.com>
+ *
+ */
+
+#define CPU_BASED_VIRTUAL_INTR_PENDING  0x00000004
+#define CPU_BASED_USE_TSC_OFFSETING     0x00000008
+#define CPU_BASED_HLT_EXITING           0x00000080
+#define CPU_BASED_INVDPG_EXITING        0x00000200
+#define CPU_BASED_MWAIT_EXITING         0x00000400
+#define CPU_BASED_RDPMC_EXITING         0x00000800
+#define CPU_BASED_RDTSC_EXITING         0x00001000
+#define CPU_BASED_CR8_LOAD_EXITING      0x00080000
+#define CPU_BASED_CR8_STORE_EXITING     0x00100000
+#define CPU_BASED_TPR_SHADOW            0x00200000
+#define CPU_BASED_MOV_DR_EXITING        0x00800000
+#define CPU_BASED_UNCOND_IO_EXITING     0x01000000
+#define CPU_BASED_ACTIVATE_IO_BITMAP    0x02000000
+#define CPU_BASED_MSR_BITMAPS           0x10000000
+#define CPU_BASED_MONITOR_EXITING       0x20000000
+#define CPU_BASED_PAUSE_EXITING         0x40000000
+
+#define PIN_BASED_EXT_INTR_MASK 0x1
+#define PIN_BASED_NMI_EXITING   0x8
+
+#define VM_EXIT_ACK_INTR_ON_EXIT        0x00008000
+#define VM_EXIT_HOST_ADD_SPACE_SIZE     0x00000200
+
+
+/* VMCS Encodings */
+enum vmcs_field {
+        GUEST_ES_SELECTOR               = 0x00000800,
+        GUEST_CS_SELECTOR               = 0x00000802,
+        GUEST_SS_SELECTOR               = 0x00000804,
+        GUEST_DS_SELECTOR               = 0x00000806,
+        GUEST_FS_SELECTOR               = 0x00000808,
+        GUEST_GS_SELECTOR               = 0x0000080a,
+        GUEST_LDTR_SELECTOR             = 0x0000080c,
+        GUEST_TR_SELECTOR               = 0x0000080e,
+        HOST_ES_SELECTOR                = 0x00000c00,
+        HOST_CS_SELECTOR                = 0x00000c02,
+        HOST_SS_SELECTOR                = 0x00000c04,
+        HOST_DS_SELECTOR                = 0x00000c06,
+        HOST_FS_SELECTOR                = 0x00000c08,
+        HOST_GS_SELECTOR                = 0x00000c0a,
+        HOST_TR_SELECTOR                = 0x00000c0c,
+        IO_BITMAP_A                     = 0x00002000,
+        IO_BITMAP_A_HIGH                = 0x00002001,
+        IO_BITMAP_B                     = 0x00002002,
+        IO_BITMAP_B_HIGH                = 0x00002003,
+        MSR_BITMAP                      = 0x00002004,
+        MSR_BITMAP_HIGH                 = 0x00002005,
+        VM_EXIT_MSR_STORE_ADDR          = 0x00002006,
+        VM_EXIT_MSR_STORE_ADDR_HIGH     = 0x00002007,
+        VM_EXIT_MSR_LOAD_ADDR           = 0x00002008,
+        VM_EXIT_MSR_LOAD_ADDR_HIGH      = 0x00002009,
+        VM_ENTRY_MSR_LOAD_ADDR          = 0x0000200a,
+        VM_ENTRY_MSR_LOAD_ADDR_HIGH     = 0x0000200b,
+        TSC_OFFSET                      = 0x00002010,
+        TSC_OFFSET_HIGH                 = 0x00002011,
+        VIRTUAL_APIC_PAGE_ADDR          = 0x00002012,
+        VIRTUAL_APIC_PAGE_ADDR_HIGH     = 0x00002013,
+        VMCS_LINK_POINTER               = 0x00002800,
+        VMCS_LINK_POINTER_HIGH          = 0x00002801,
+        GUEST_IA32_DEBUGCTL             = 0x00002802,
+        GUEST_IA32_DEBUGCTL_HIGH        = 0x00002803,
+        PIN_BASED_VM_EXEC_CONTROL       = 0x00004000,
+        CPU_BASED_VM_EXEC_CONTROL       = 0x00004002,
+        EXCEPTION_BITMAP                = 0x00004004,
+        PAGE_FAULT_ERROR_CODE_MASK      = 0x00004006,
+        PAGE_FAULT_ERROR_CODE_MATCH     = 0x00004008,
+        CR3_TARGET_COUNT                = 0x0000400a,
+        VM_EXIT_CONTROLS                = 0x0000400c,
+        VM_EXIT_MSR_STORE_COUNT         = 0x0000400e,
+        VM_EXIT_MSR_LOAD_COUNT          = 0x00004010,
+        VM_ENTRY_CONTROLS               = 0x00004012,
+        VM_ENTRY_MSR_LOAD_COUNT         = 0x00004014,
+        VM_ENTRY_INTR_INFO_FIELD        = 0x00004016,
+        VM_ENTRY_EXCEPTION_ERROR_CODE   = 0x00004018,
+        VM_ENTRY_INSTRUCTION_LEN        = 0x0000401a,
+        TPR_THRESHOLD                   = 0x0000401c,
+        SECONDARY_VM_EXEC_CONTROL       = 0x0000401e,
+        VM_INSTRUCTION_ERROR            = 0x00004400,
+        VM_EXIT_REASON                  = 0x00004402,
+        VM_EXIT_INTR_INFO               = 0x00004404,
+        VM_EXIT_INTR_ERROR_CODE         = 0x00004406,
+        IDT_VECTORING_INFO_FIELD        = 0x00004408,
+        IDT_VECTORING_ERROR_CODE        = 0x0000440a,
+        VM_EXIT_INSTRUCTION_LEN         = 0x0000440c,
+        VMX_INSTRUCTION_INFO            = 0x0000440e,
+        GUEST_ES_LIMIT                  = 0x00004800,
+        GUEST_CS_LIMIT                  = 0x00004802,
+        GUEST_SS_LIMIT                  = 0x00004804,
+        GUEST_DS_LIMIT                  = 0x00004806,
+        GUEST_FS_LIMIT                  = 0x00004808,
+        GUEST_GS_LIMIT                  = 0x0000480a,
+        GUEST_LDTR_LIMIT                = 0x0000480c,
+        GUEST_TR_LIMIT                  = 0x0000480e,
+        GUEST_GDTR_LIMIT                = 0x00004810,
+        GUEST_IDTR_LIMIT                = 0x00004812,
+        GUEST_ES_AR_BYTES               = 0x00004814,
+        GUEST_CS_AR_BYTES               = 0x00004816,
+        GUEST_SS_AR_BYTES               = 0x00004818,
+        GUEST_DS_AR_BYTES               = 0x0000481a,
+        GUEST_FS_AR_BYTES               = 0x0000481c,
+        GUEST_GS_AR_BYTES               = 0x0000481e,
+        GUEST_LDTR_AR_BYTES             = 0x00004820,
+        GUEST_TR_AR_BYTES               = 0x00004822,
+        GUEST_INTERRUPTIBILITY_INFO     = 0x00004824,
+        GUEST_ACTIVITY_STATE            = 0X00004826,
+        GUEST_SYSENTER_CS               = 0x0000482A,
+        HOST_IA32_SYSENTER_CS           = 0x00004c00,
+        CR0_GUEST_HOST_MASK             = 0x00006000,
+        CR4_GUEST_HOST_MASK             = 0x00006002,
+        CR0_READ_SHADOW                 = 0x00006004,
+        CR4_READ_SHADOW                 = 0x00006006,
+        CR3_TARGET_VALUE0               = 0x00006008,
+        CR3_TARGET_VALUE1               = 0x0000600a,
+        CR3_TARGET_VALUE2               = 0x0000600c,
+        CR3_TARGET_VALUE3               = 0x0000600e,
+        EXIT_QUALIFICATION              = 0x00006400,
+        GUEST_LINEAR_ADDRESS            = 0x0000640a,
+        GUEST_CR0                       = 0x00006800,
+        GUEST_CR3                       = 0x00006802,
+        GUEST_CR4                       = 0x00006804,
+        GUEST_ES_BASE                   = 0x00006806,
+        GUEST_CS_BASE                   = 0x00006808,
+        GUEST_SS_BASE                   = 0x0000680a,
+        GUEST_DS_BASE                   = 0x0000680c,
+        GUEST_FS_BASE                   = 0x0000680e,
+        GUEST_GS_BASE                   = 0x00006810,
+        GUEST_LDTR_BASE                 = 0x00006812,
+        GUEST_TR_BASE                   = 0x00006814,
+        GUEST_GDTR_BASE                 = 0x00006816,
+        GUEST_IDTR_BASE                 = 0x00006818,
+        GUEST_DR7                       = 0x0000681a,
+        GUEST_RSP                       = 0x0000681c,
+        GUEST_RIP                       = 0x0000681e,
+        GUEST_RFLAGS                    = 0x00006820,
+        GUEST_PENDING_DBG_EXCEPTIONS    = 0x00006822,
+        GUEST_SYSENTER_ESP              = 0x00006824,
+        GUEST_SYSENTER_EIP              = 0x00006826,
+        HOST_CR0                        = 0x00006c00,
+        HOST_CR3                        = 0x00006c02,
+        HOST_CR4                        = 0x00006c04,
+        HOST_FS_BASE                    = 0x00006c06,
+        HOST_GS_BASE                    = 0x00006c08,
+        HOST_TR_BASE                    = 0x00006c0a,
+        HOST_GDTR_BASE                  = 0x00006c0c,
+        HOST_IDTR_BASE                  = 0x00006c0e,
+        HOST_IA32_SYSENTER_ESP          = 0x00006c10,
+        HOST_IA32_SYSENTER_EIP          = 0x00006c12,
+        HOST_RSP                        = 0x00006c14,
+        HOST_RIP                        = 0x00006c16,
+};
+
+#define VMX_EXIT_REASONS_FAILED_VMENTRY         0x80000000
+
+#define EXIT_REASON_EXCEPTION_NMI       0
+#define EXIT_REASON_EXTERNAL_INTERRUPT  1
+
+#define EXIT_REASON_PENDING_INTERRUPT   7
+
+#define EXIT_REASON_TASK_SWITCH         9
+#define EXIT_REASON_CPUID               10
+#define EXIT_REASON_HLT                 12
+#define EXIT_REASON_INVLPG              14
+#define EXIT_REASON_RDPMC               15
+#define EXIT_REASON_RDTSC               16
+#define EXIT_REASON_VMCALL              18
+#define EXIT_REASON_VMCLEAR             19
+#define EXIT_REASON_VMLAUNCH            20
+#define EXIT_REASON_VMPTRLD             21
+#define EXIT_REASON_VMPTRST             22
+#define EXIT_REASON_VMREAD              23
+#define EXIT_REASON_VMRESUME            24
+#define EXIT_REASON_VMWRITE             25
+#define EXIT_REASON_VMOFF               26
+#define EXIT_REASON_VMON                27
+#define EXIT_REASON_CR_ACCESS           28
+#define EXIT_REASON_DR_ACCESS           29
+#define EXIT_REASON_IO_INSTRUCTION      30
+#define EXIT_REASON_MSR_READ            31
+#define EXIT_REASON_MSR_WRITE           32
+#define EXIT_REASON_MWAIT_INSTRUCTION   36
+
+/*
+ * Interruption-information format
+ */
+#define INTR_INFO_VECTOR_MASK           0xff            /* 7:0 */
+#define INTR_INFO_INTR_TYPE_MASK        0x700           /* 10:8 */
+#define INTR_INFO_DELIEVER_CODE_MASK    0x800           /* 11 */
+#define INTR_INFO_VALID_MASK            0x80000000      /* 31 */
+
+#define VECTORING_INFO_VECTOR_MASK              INTR_INFO_VECTOR_MASK
+#define VECTORING_INFO_TYPE_MASK                INTR_INFO_INTR_TYPE_MASK
+#define VECTORING_INFO_DELIEVER_CODE_MASK       INTR_INFO_DELIEVER_CODE_MASK
+#define VECTORING_INFO_VALID_MASK               INTR_INFO_VALID_MASK
+
+#define INTR_TYPE_EXT_INTR              (0 << 8) /* external interrupt */
+#define INTR_TYPE_EXCEPTION             (3 << 8) /* processor exception */
+
+/*
+ * Exit Qualifications for MOV for Control Register Access
+ */
+#define CONTROL_REG_ACCESS_NUM          0x7     /* 2:0, number of control register */
+#define CONTROL_REG_ACCESS_TYPE         0x30    /* 5:4, access type */
+#define CONTROL_REG_ACCESS_REG          0xf00   /* 10:8, general purpose register */
+#define LMSW_SOURCE_DATA_SHIFT 16
+#define LMSW_SOURCE_DATA  (0xFFFF << LMSW_SOURCE_DATA_SHIFT) /* 16:31 lmsw source */
+#define REG_EAX                         (0 << 8)
+#define REG_ECX                         (1 << 8)
+#define REG_EDX                         (2 << 8)
+#define REG_EBX                         (3 << 8)
+#define REG_ESP                         (4 << 8)
+#define REG_EBP                         (5 << 8)
+#define REG_ESI                         (6 << 8)
+#define REG_EDI                         (7 << 8)
+#define REG_R8                         (8 << 8)
+#define REG_R9                         (9 << 8)
+#define REG_R10                        (10 << 8)
+#define REG_R11                        (11 << 8)
+#define REG_R12                        (12 << 8)
+#define REG_R13                        (13 << 8)
+#define REG_R14                        (14 << 8)
+#define REG_R15                        (15 << 8)
+
+/*
+ * Exit Qualifications for MOV for Debug Register Access
+ */
+#define DEBUG_REG_ACCESS_NUM            0x7     /* 2:0, number of debug register */
+#define DEBUG_REG_ACCESS_TYPE           0x10    /* 4, direction of access */
+#define TYPE_MOV_TO_DR                  (0 << 4)
+#define TYPE_MOV_FROM_DR                (1 << 4)
+#define DEBUG_REG_ACCESS_REG            0xf00   /* 11:8, general purpose register */
+
+
+/* segment AR */
+#define SEGMENT_AR_L_MASK (1 << 13)
+
+/* entry controls */
+#define VM_ENTRY_CONTROLS_IA32E_MASK (1 << 9)
+
+#define AR_TYPE_ACCESSES_MASK 1
+#define AR_TYPE_READABLE_MASK (1 << 1)
+#define AR_TYPE_WRITEABLE_MASK (1 << 2)
+#define AR_TYPE_CODE_MASK (1 << 3)
+#define AR_TYPE_MASK 0x0f
+#define AR_TYPE_BUSY_64_TSS 11
+#define AR_TYPE_BUSY_32_TSS 11
+#define AR_TYPE_BUSY_16_TSS 3
+#define AR_TYPE_LDT 2
+
+#define AR_UNUSABLE_MASK (1 << 16)
+#define AR_S_MASK (1 << 4)
+#define AR_P_MASK (1 << 7)
+#define AR_L_MASK (1 << 13)
+#define AR_DB_MASK (1 << 14)
+#define AR_G_MASK (1 << 15)
+#define AR_DPL_SHIFT 5
+#define AR_DPL(ar) (((ar) >> AR_DPL_SHIFT) & 3)
+
+#define AR_RESERVD_MASK 0xfffe0f00
+
+#define CR4_VMXE 0x2000
+
+#define MSR_IA32_VMX_BASIC_MSR                  0x480
+#define MSR_IA32_FEATURE_CONTROL                0x03a
+#define MSR_IA32_VMX_PINBASED_CTLS_MSR          0x481
+#define MSR_IA32_VMX_PROCBASED_CTLS_MSR         0x482
+#define MSR_IA32_VMX_EXIT_CTLS_MSR              0x483
+#define MSR_IA32_VMX_ENTRY_CTLS_MSR             0x484
+
+#endif