43 files changed, 3367 insertions, 1605 deletions

diff --git a/Documentation/ia64/paravirt_ops.txt b/Documentation/ia64/paravirt_ops.txt
deleted file mode 100644
index 39ded02ec33f..000000000000
--- a/Documentation/ia64/paravirt_ops.txt
+++ /dev/null
@@ -1,137 +0,0 @@
-Paravirt_ops on IA64
-====================
-                          21 May 2008, Isaku Yamahata <yamahata@valinux.co.jp>
-
-
-Introduction
-------------
-The aim of this documentation is to help with maintainability and/or to
-encourage people to use paravirt_ops/IA64.
-
-paravirt_ops (pv_ops in short) is a way for virtualization support of
-Linux kernel on x86. Several ways for virtualization support were
-proposed, paravirt_ops is the winner.
-On the other hand, now there are also several IA64 virtualization
-technologies like kvm/IA64, xen/IA64 and many other academic IA64
-hypervisors so that it is good to add generic virtualization
-infrastructure on Linux/IA64.
-
-
-What is paravirt_ops?
----------------------
-It has been developed on x86 as virtualization support via API, not ABI.
-It allows each hypervisor to override operations which are important for
-hypervisors at API level. And it allows a single kernel binary to run on
-all supported execution environments including native machine.
-Essentially paravirt_ops is a set of function pointers which represent
-operations corresponding to low level sensitive instructions and high
-level functionalities in various area. But one significant difference
-from usual function pointer table is that it allows optimization with
-binary patch. It is because some of these operations are very
-performance sensitive and indirect call overhead is not negligible.
-With binary patch, indirect C function call can be transformed into
-direct C function call or in-place execution to eliminate the overhead.
-
-Thus, operations of paravirt_ops are classified into three categories.
-- simple indirect call
-  These operations correspond to high level functionality so that the
-  overhead of indirect call isn't very important.
-
-- indirect call which allows optimization with binary patch
-  Usually these operations correspond to low level instructions. They
-  are called frequently and performance critical. So the overhead is
-  very important.
-
-- a set of macros for hand written assembly code
-  Hand written assembly codes (.S files) also need paravirtualization
-  because they include sensitive instructions or some of code paths in
-  them are very performance critical.
-
-
-The relation to the IA64 machine vector
----------------------------------------
-Linux/IA64 has the IA64 machine vector functionality which allows the
-kernel to switch implementations (e.g. initialization, ipi, dma api...)
-depending on executing platform.
-We can replace some implementations very easily defining a new machine
-vector. Thus another approach for virtualization support would be
-enhancing the machine vector functionality.
-But paravirt_ops approach was taken because
-- virtualization support needs wider support than machine vector does.
-  e.g. low level instruction paravirtualization. It must be
-       initialized very early before platform detection.
-
-- virtualization support needs more functionality like binary patch.
-  Probably the calling overhead might not be very large compared to the
-  emulation overhead of virtualization. However in the native case, the
-  overhead should be eliminated completely.
-  A single kernel binary should run on each environment including native,
-  and the overhead of paravirt_ops on native environment should be as
-  small as possible.
-
-- for full virtualization technology, e.g. KVM/IA64 or
-  Xen/IA64 HVM domain, the result would be
-  (the emulated platform machine vector. probably dig) + (pv_ops).
-  This means that the virtualization support layer should be under
-  the machine vector layer.
-
-Possibly it might be better to move some function pointers from
-paravirt_ops to machine vector. In fact, Xen domU case utilizes both
-pv_ops and machine vector.
-
-
-IA64 paravirt_ops
------------------
-In this section, the concrete paravirt_ops will be discussed.
-Because of the architecture difference between ia64 and x86, the
-resulting set of functions is very different from x86 pv_ops.
-
-- C function pointer tables
-They are not very performance critical so that simple C indirect
-function call is acceptable. The following structures are defined at
-this moment. For details see linux/include/asm-ia64/paravirt.h
-  - struct pv_info
-    This structure describes the execution environment.
-  - struct pv_init_ops
-    This structure describes the various initialization hooks.
-  - struct pv_iosapic_ops
-    This structure describes hooks to iosapic operations.
-  - struct pv_irq_ops
-    This structure describes hooks to irq related operations
-  - struct pv_time_op
-    This structure describes hooks to steal time accounting.
-
-- a set of indirect calls which need optimization
-Currently this class of functions correspond to a subset of IA64
-intrinsics. At this moment the optimization with binary patch isn't
-implemented yet.
-struct pv_cpu_op is defined. For details see
-linux/include/asm-ia64/paravirt_privop.h
-Mostly they correspond to ia64 intrinsics 1-to-1.
-Caveat: Now they are defined as C indirect function pointers, but in
-order to support binary patch optimization, they will be changed
-using GCC extended inline assembly code.
-
-- a set of macros for hand written assembly code (.S files)
-For maintenance purpose, the taken approach for .S files is single
-source code and compile multiple times with different macros definitions.
-Each pv_ops instance must define those macros to compile.
-The important thing here is that sensitive, but non-privileged
-instructions must be paravirtualized and that some privileged
-instructions also need paravirtualization for reasonable performance.
-Developers who modify .S files must be aware of that. At this moment
-an easy checker is implemented to detect paravirtualization breakage.
-But it doesn't cover all the cases.
-
-Sometimes this set of macros is called pv_cpu_asm_op. But there is no
-corresponding structure in the source code.
-Those macros mostly 1:1 correspond to a subset of privileged
-instructions. See linux/include/asm-ia64/native/inst.h.
-And some functions written in assembly also need to be overrided so
-that each pv_ops instance have to define some macros. Again see
-linux/include/asm-ia64/native/inst.h.
-
-
-Those structures must be initialized very early before start_kernel.
-Probably initialized in head.S using multi entry point or some other trick.
-For native case implementation see linux/arch/ia64/kernel/paravirt.c.
diff --git a/Documentation/virtual/00-INDEX b/Documentation/virtual/00-INDEX
index e952d30bbf0f..af0d23968ee7 100644
--- a/Documentation/virtual/00-INDEX
+++ b/Documentation/virtual/00-INDEX
@@ -2,6 +2,9 @@ Virtualization support in the Linux kernel.
 
 00-INDEX
         - this file.
+
+paravirt_ops.txt
+        - Describes the Linux kernel pv_ops to support different hypervisors
 kvm/
         - Kernel Virtual Machine.  See also http://linux-kvm.org
 uml/
diff --git a/Documentation/virtual/paravirt_ops.txt b/Documentation/virtual/paravirt_ops.txt
new file mode 100644
index 000000000000..d4881c00e339
--- /dev/null
+++ b/Documentation/virtual/paravirt_ops.txt
@@ -0,0 +1,32 @@
+Paravirt_ops
+============
+
+Linux provides support for different hypervisor virtualization technologies.
+Historically different binary kernels would be required in order to support
+different hypervisors, this restriction was removed with pv_ops.
+Linux pv_ops is a virtualization API which enables support for different
+hypervisors. It allows each hypervisor to override critical operations and
+allows a single kernel binary to run on all supported execution environments
+including native machine -- without any hypervisors.
+
+pv_ops provides a set of function pointers which represent operations
+corresponding to low level critical instructions and high level
+functionalities in various areas. pv-ops allows for optimizations at run
+time by enabling binary patching of the low-ops critical operations
+at boot time.
+
+pv_ops operations are classified into three categories:
+
+- simple indirect call
+  These operations correspond to high level functionality where it is
+  known that the overhead of indirect call isn't very important.
+
+- indirect call which allows optimization with binary patch
+  Usually these operations correspond to low level critical instructions. They
+  are called frequently and are performance critical. The overhead is
+  very important.
+
+- a set of macros for hand written assembly code
+  Hand written assembly codes (.S files) also need paravirtualization
+  because they include sensitive instructions or some of code paths in
+  them are very performance critical.
diff --git a/MAINTAINERS b/MAINTAINERS
index 4f4915cbeab9..1921ed58d1a0 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -7302,7 +7302,7 @@ M:	Alok Kataria <akataria@vmware.com>
 M:      Rusty Russell <rusty@rustcorp.com.au>
 L:      virtualization@lists.linux-foundation.org
 S:      Supported
-F:      Documentation/ia64/paravirt_ops.txt
+F:      Documentation/virtual/paravirt_ops.txt
 F:      arch/*/kernel/paravirt*
 F:      arch/*/include/asm/paravirt.h
 
diff --git a/arch/mn10300/unit-asb2305/pci-iomap.c b/arch/mn10300/unit-asb2305/pci-iomap.c
deleted file mode 100644
index bd65dae17f32..000000000000
--- a/arch/mn10300/unit-asb2305/pci-iomap.c
+++ /dev/null
@@ -1,35 +0,0 @@
-/* ASB2305 PCI I/O mapping handler
- *
- * Copyright (C) 2007 Red Hat, Inc. All Rights Reserved.
- * Written by David Howells (dhowells@redhat.com)
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public Licence
- * as published by the Free Software Foundation; either version
- * 2 of the Licence, or (at your option) any later version.
- */
-#include <linux/pci.h>
-#include <linux/module.h>
-
-/*
- * Create a virtual mapping cookie for a PCI BAR (memory or IO)
- */
-void __iomem *pci_iomap(struct pci_dev *dev, int bar, unsigned long maxlen)
-{
-        resource_size_t start = pci_resource_start(dev, bar);
-        resource_size_t len = pci_resource_len(dev, bar);
-        unsigned long flags = pci_resource_flags(dev, bar);
-
-        if (!len || !start)
-                return NULL;
-
-        if ((flags & IORESOURCE_IO) || (flags & IORESOURCE_MEM)) {
-                if (flags & IORESOURCE_CACHEABLE && !(flags & IORESOURCE_IO))
-                        return ioremap(start, len);
-                else
-                        return ioremap_nocache(start, len);
-        }
-
-        return NULL;
-}
-EXPORT_SYMBOL(pci_iomap);
diff --git a/arch/s390/include/asm/pci_io.h b/arch/s390/include/asm/pci_io.h
index f664e96f48c7..1a9a98de5bde 100644
--- a/arch/s390/include/asm/pci_io.h
+++ b/arch/s390/include/asm/pci_io.h
@@ -16,6 +16,7 @@
 struct zpci_iomap_entry {
         u32 fh;
         u8 bar;
+        u16 count;
 };
 
 extern struct zpci_iomap_entry *zpci_iomap_start;
diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c
index 3290f11ae1d9..753a56731951 100644
--- a/arch/s390/pci/pci.c
+++ b/arch/s390/pci/pci.c
@@ -259,7 +259,10 @@ void __iowrite64_copy(void __iomem *to, const void *from, size_t count)
 }
 
 /* Create a virtual mapping cookie for a PCI BAR */
-void __iomem *pci_iomap(struct pci_dev *pdev, int bar, unsigned long max)
+void __iomem *pci_iomap_range(struct pci_dev *pdev,
+                              int bar,
+                              unsigned long offset,
+                              unsigned long max)
 {
         struct zpci_dev *zdev = get_zdev(pdev);
         u64 addr;
@@ -270,14 +273,27 @@ void __iomem *pci_iomap(struct pci_dev *pdev, int bar, unsigned long max)
 
         idx = zdev->bars[bar].map_idx;
         spin_lock(&zpci_iomap_lock);
-        zpci_iomap_start[idx].fh = zdev->fh;
-        zpci_iomap_start[idx].bar = bar;
+        if (zpci_iomap_start[idx].count++) {
+                BUG_ON(zpci_iomap_start[idx].fh != zdev->fh ||
+                       zpci_iomap_start[idx].bar != bar);
+        } else {
+                zpci_iomap_start[idx].fh = zdev->fh;
+                zpci_iomap_start[idx].bar = bar;
+        }
+        /* Detect overrun */
+        BUG_ON(!zpci_iomap_start[idx].count);
         spin_unlock(&zpci_iomap_lock);
 
         addr = ZPCI_IOMAP_ADDR_BASE | ((u64) idx << 48);
-        return (void __iomem *) addr;
+        return (void __iomem *) addr + offset;
 }
-EXPORT_SYMBOL_GPL(pci_iomap);
+EXPORT_SYMBOL_GPL(pci_iomap_range);
+
+void __iomem *pci_iomap(struct pci_dev *dev, int bar, unsigned long maxlen)
+{
+        return pci_iomap_range(dev, bar, 0, maxlen);
+}
+EXPORT_SYMBOL(pci_iomap);
 
 void pci_iounmap(struct pci_dev *pdev, void __iomem *addr)
 {
@@ -285,8 +301,12 @@ void pci_iounmap(struct pci_dev *pdev, void __iomem *addr)
 
         idx = (((__force u64) addr) & ~ZPCI_IOMAP_ADDR_BASE) >> 48;
         spin_lock(&zpci_iomap_lock);
-        zpci_iomap_start[idx].fh = 0;
-        zpci_iomap_start[idx].bar = 0;
+        /* Detect underrun */
+        BUG_ON(!zpci_iomap_start[idx].count);
+        if (!--zpci_iomap_start[idx].count) {
+                zpci_iomap_start[idx].fh = 0;
+                zpci_iomap_start[idx].bar = 0;
+        }
         spin_unlock(&zpci_iomap_lock);
 }
 EXPORT_SYMBOL_GPL(pci_iounmap);
diff --git a/arch/x86/include/asm/lguest_hcall.h b/arch/x86/include/asm/lguest_hcall.h
index 879fd7d33877..ef01fef3eebc 100644
--- a/arch/x86/include/asm/lguest_hcall.h
+++ b/arch/x86/include/asm/lguest_hcall.h
@@ -16,7 +16,6 @@
 #define LHCALL_SET_PTE          14
 #define LHCALL_SET_PGD          15
 #define LHCALL_LOAD_TLS         16
-#define LHCALL_NOTIFY           17
 #define LHCALL_LOAD_GDT_ENTRY   18
 #define LHCALL_SEND_INTERRUPTS  19
 
diff --git a/arch/x86/lguest/boot.c b/arch/x86/lguest/boot.c
index c1c1544b8485..ac4453d8520e 100644
--- a/arch/x86/lguest/boot.c
+++ b/arch/x86/lguest/boot.c
@@ -56,6 +56,9 @@
 #include <linux/virtio_console.h>
 #include <linux/pm.h>
 #include <linux/export.h>
+#include <linux/pci.h>
+#include <linux/virtio_pci.h>
+#include <asm/acpi.h>
 #include <asm/apic.h>
 #include <asm/lguest.h>
 #include <asm/paravirt.h>
@@ -71,6 +74,8 @@
 #include <asm/stackprotector.h>
 #include <asm/reboot.h>         /* for struct machine_ops */
 #include <asm/kvm_para.h>
+#include <asm/pci_x86.h>
+#include <asm/pci-direct.h>
 
 /*G:010
  * Welcome to the Guest!
@@ -831,6 +836,24 @@ static struct irq_chip lguest_irq_controller = {
         .irq_unmask     = enable_lguest_irq,
 };
 
+static int lguest_enable_irq(struct pci_dev *dev)
+{
+        u8 line = 0;
+
+        /* We literally use the PCI interrupt line as the irq number. */
+        pci_read_config_byte(dev, PCI_INTERRUPT_LINE, &line);
+        irq_set_chip_and_handler_name(line, &lguest_irq_controller,
+                                      handle_level_irq, "level");
+        dev->irq = line;
+        return 0;
+}
+
+/* We don't do hotplug PCI, so this shouldn't be called. */
+static void lguest_disable_irq(struct pci_dev *dev)
+{
+        WARN_ON(1);
+}
+
 /*
  * This sets up the Interrupt Descriptor Table (IDT) entry for each hardware
  * interrupt (except 128, which is used for system calls), and then tells the
@@ -1181,25 +1204,136 @@ static __init char *lguest_memory_setup(void)
         return "LGUEST";
 }
 
+/* Offset within PCI config space of BAR access capability. */
+static int console_cfg_offset = 0;
+static int console_access_cap;
+
+/* Set up so that we access off in bar0 (on bus 0, device 1, function 0) */
+static void set_cfg_window(u32 cfg_offset, u32 off)
+{
+        write_pci_config_byte(0, 1, 0,
+                              cfg_offset + offsetof(struct virtio_pci_cap, bar),
+                              0);
+        write_pci_config(0, 1, 0,
+                         cfg_offset + offsetof(struct virtio_pci_cap, length),
+                         4);
+        write_pci_config(0, 1, 0,
+                         cfg_offset + offsetof(struct virtio_pci_cap, offset),
+                         off);
+}
+
+static void write_bar_via_cfg(u32 cfg_offset, u32 off, u32 val)
+{
+        /*
+         * We could set this up once, then leave it; nothing else in the *
+         * kernel should touch these registers.  But if it went wrong, that
+         * would be a horrible bug to find.
+         */
+        set_cfg_window(cfg_offset, off);
+        write_pci_config(0, 1, 0,
+                         cfg_offset + sizeof(struct virtio_pci_cap), val);
+}
+
+static void probe_pci_console(void)
+{
+        u8 cap, common_cap = 0, device_cap = 0;
+        /* Offset within BAR0 */
+        u32 device_offset;
+        u32 device_len;
+
+        /* Avoid recursive printk into here. */
+        console_cfg_offset = -1;
+
+        if (!early_pci_allowed()) {
+                printk(KERN_ERR "lguest: early PCI access not allowed!\n");
+                return;
+        }
+
+        /* We expect a console PCI device at BUS0, slot 1. */
+        if (read_pci_config(0, 1, 0, 0) != 0x10431AF4) {
+                printk(KERN_ERR "lguest: PCI device is %#x!\n",
+                       read_pci_config(0, 1, 0, 0));
+                return;
+        }
+
+        /* Find the capabilities we need (must be in bar0) */
+        cap = read_pci_config_byte(0, 1, 0, PCI_CAPABILITY_LIST);
+        while (cap) {
+                u8 vndr = read_pci_config_byte(0, 1, 0, cap);
+                if (vndr == PCI_CAP_ID_VNDR) {
+                        u8 type, bar;
+                        u32 offset, length;
+
+                        type = read_pci_config_byte(0, 1, 0,
+                            cap + offsetof(struct virtio_pci_cap, cfg_type));
+                        bar = read_pci_config_byte(0, 1, 0,
+                            cap + offsetof(struct virtio_pci_cap, bar));
+                        offset = read_pci_config(0, 1, 0,
+                            cap + offsetof(struct virtio_pci_cap, offset));
+                        length = read_pci_config(0, 1, 0,
+                            cap + offsetof(struct virtio_pci_cap, length));
+
+                        switch (type) {
+                        case VIRTIO_PCI_CAP_DEVICE_CFG:
+                                if (bar == 0) {
+                                        device_cap = cap;
+                                        device_offset = offset;
+                                        device_len = length;
+                                }
+                                break;
+                        case VIRTIO_PCI_CAP_PCI_CFG:
+                                console_access_cap = cap;
+                                break;
+                        }
+                }
+                cap = read_pci_config_byte(0, 1, 0, cap + PCI_CAP_LIST_NEXT);
+        }
+        if (!device_cap || !console_access_cap) {
+                printk(KERN_ERR "lguest: No caps (%u/%u/%u) in console!\n",
+                       common_cap, device_cap, console_access_cap);
+                return;
+        }
+
+        /*
+         * Note that we can't check features, until we've set the DRIVER
+         * status bit.  We don't want to do that until we have a real driver,
+         * so we just check that the device-specific config has room for
+         * emerg_wr.  If it doesn't support VIRTIO_CONSOLE_F_EMERG_WRITE
+         * it should ignore the access.
+         */
+        if (device_len < (offsetof(struct virtio_console_config, emerg_wr)
+                          + sizeof(u32))) {
+                printk(KERN_ERR "lguest: console missing emerg_wr field\n");
+                return;
+        }
+
+        console_cfg_offset = device_offset;
+        printk(KERN_INFO "lguest: Console via virtio-pci emerg_wr\n");
+}
+
 /*
  * We will eventually use the virtio console device to produce console output,
- * but before that is set up we use LHCALL_NOTIFY on normal memory to produce
- * console output.
+ * but before that is set up we use the virtio PCI console's backdoor mmio
+ * access and the "emergency" write facility (which is legal even before the
+ * device is configured).
  */
 static __init int early_put_chars(u32 vtermno, const char *buf, int count)
 {
-        char scratch[17];
-        unsigned int len = count;
+        /* If we couldn't find PCI console, forget it. */
+        if (console_cfg_offset < 0)
+                return count;
 
-        /* We use a nul-terminated string, so we make a copy.  Icky, huh? */
-        if (len > sizeof(scratch) - 1)
-                len = sizeof(scratch) - 1;
-        scratch[len] = '\0';
-        memcpy(scratch, buf, len);
-        hcall(LHCALL_NOTIFY, __pa(scratch), 0, 0, 0);
+        if (unlikely(!console_cfg_offset)) {
+                probe_pci_console();
+                if (console_cfg_offset < 0)
+                        return count;
+        }
 
-        /* This routine returns the number of bytes actually written. */
-        return len;
+        write_bar_via_cfg(console_access_cap,
+                          console_cfg_offset
+                          + offsetof(struct virtio_console_config, emerg_wr),
+                          buf[0]);
+        return 1;
 }
 
 /*
@@ -1400,14 +1534,6 @@ __init void lguest_init(void)
         atomic_notifier_chain_register(&panic_notifier_list, &paniced);
 
         /*
-         * The IDE code spends about 3 seconds probing for disks: if we reserve
-         * all the I/O ports up front it can't get them and so doesn't probe.
-         * Other device drivers are similar (but less severe).  This cuts the
-         * kernel boot time on my machine from 4.1 seconds to 0.45 seconds.
-         */
-        paravirt_disable_iospace();
-
-        /*
          * This is messy CPU setup stuff which the native boot code does before
          * start_kernel, so we have to do, too:
          */
@@ -1436,6 +1562,13 @@ __init void lguest_init(void)
         /* Register our very early console. */
         virtio_cons_early_init(early_put_chars);
 
+        /* Don't let ACPI try to control our PCI interrupts. */
+        disable_acpi();
+
+        /* We control them ourselves, by overriding these two hooks. */
+        pcibios_enable_irq = lguest_enable_irq;
+        pcibios_disable_irq = lguest_disable_irq;
+
         /*
          * Last of all, we set the power management poweroff hook to point to
          * the Guest routine to power off, and the reboot hook to our restart
diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index cdfbd21e3597..655e570b9b31 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -28,8 +28,7 @@ struct virtio_blk_vq {
         char name[VQ_NAME_LEN];
 } ____cacheline_aligned_in_smp;
 
-struct virtio_blk
-{
+struct virtio_blk {
         struct virtio_device *vdev;
 
         /* The disk structure for the kernel. */
@@ -52,8 +51,7 @@ struct virtio_blk
         struct virtio_blk_vq *vqs;
 };
 
-struct virtblk_req
-{
+struct virtblk_req {
         struct request *req;
         struct virtio_blk_outhdr out_hdr;
         struct virtio_scsi_inhdr in_hdr;
@@ -575,6 +573,12 @@ static int virtblk_probe(struct virtio_device *vdev)
         u16 min_io_size;
         u8 physical_block_exp, alignment_offset;
 
+        if (!vdev->config->get) {
+                dev_err(&vdev->dev, "%s failure: config access disabled\n",
+                        __func__);
+                return -EINVAL;
+        }
+
         err = ida_simple_get(&vd_index_ida, 0, minor_to_index(1 << MINORBITS),
                              GFP_KERNEL);
         if (err < 0)
diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
index 26afb56a8073..fae2dbbf5745 100644
--- a/drivers/char/virtio_console.c
+++ b/drivers/char/virtio_console.c
@@ -1986,7 +1986,10 @@ static int virtcons_probe(struct virtio_device *vdev)
         bool multiport;
         bool early = early_put_chars != NULL;
 
-        if (!vdev->config->get) {
+        /* We only need a config space if features are offered */
+        if (!vdev->config->get &&
+            (virtio_has_feature(vdev, VIRTIO_CONSOLE_F_SIZE)
+             || virtio_has_feature(vdev, VIRTIO_CONSOLE_F_MULTIPORT))) {
                 dev_err(&vdev->dev, "%s failure: config access disabled\n",
                         __func__);
                 return -EINVAL;
diff --git a/drivers/lguest/Makefile b/drivers/lguest/Makefile
index c4197503900e..16f52ee73994 100644
--- a/drivers/lguest/Makefile
+++ b/drivers/lguest/Makefile
@@ -1,6 +1,3 @@
-# Guest requires the device configuration and probing code.
-obj-$(CONFIG_LGUEST_GUEST) += lguest_device.o
-
 # Host requires the other files, which can be a module.
 obj-$(CONFIG_LGUEST)    += lg.o
 lg-y = core.o hypercalls.o page_tables.o interrupts_and_traps.o \
diff --git a/drivers/lguest/core.c b/drivers/lguest/core.c
index 6590558d1d31..7dc93aa004c8 100644
--- a/drivers/lguest/core.c
+++ b/drivers/lguest/core.c
@@ -208,6 +208,14 @@ void __lgwrite(struct lg_cpu *cpu, unsigned long addr, const void *b,
  */
 int run_guest(struct lg_cpu *cpu, unsigned long __user *user)
 {
+        /* If the launcher asked for a register with LHREQ_GETREG */
+        if (cpu->reg_read) {
+                if (put_user(*cpu->reg_read, user))
+                        return -EFAULT;
+                cpu->reg_read = NULL;
+                return sizeof(*cpu->reg_read);
+        }
+
         /* We stop running once the Guest is dead. */
         while (!cpu->lg->dead) {
                 unsigned int irq;
@@ -217,21 +225,12 @@ int run_guest(struct lg_cpu *cpu, unsigned long __user *user)
                 if (cpu->hcall)
                         do_hypercalls(cpu);
 
-                /*
-                 * It's possible the Guest did a NOTIFY hypercall to the
-                 * Launcher.
-                 */
-                if (cpu->pending_notify) {
-                        /*
-                         * Does it just needs to write to a registered
-                         * eventfd (ie. the appropriate virtqueue thread)?
-                         */
-                        if (!send_notify_to_eventfd(cpu)) {
-                                /* OK, we tell the main Launcher. */
-                                if (put_user(cpu->pending_notify, user))
-                                        return -EFAULT;
-                                return sizeof(cpu->pending_notify);
-                        }
+                /* Do we have to tell the Launcher about a trap? */
+                if (cpu->pending.trap) {
+                        if (copy_to_user(user, &cpu->pending,
+                                         sizeof(cpu->pending)))
+                                return -EFAULT;
+                        return sizeof(cpu->pending);
                 }
 
                 /*
diff --git a/drivers/lguest/hypercalls.c b/drivers/lguest/hypercalls.c
index 83511eb0923d..1219af493c0f 100644
--- a/drivers/lguest/hypercalls.c
+++ b/drivers/lguest/hypercalls.c
@@ -117,9 +117,6 @@ static void do_hcall(struct lg_cpu *cpu, struct hcall_args *args)
                 /* Similarly, this sets the halted flag for run_guest(). */
                 cpu->halted = 1;
                 break;
-        case LHCALL_NOTIFY:
-                cpu->pending_notify = args->arg1;
-                break;
         default:
                 /* It should be an architecture-specific hypercall. */
                 if (lguest_arch_do_hcall(cpu, args))
@@ -189,7 +186,7 @@ static void do_async_hcalls(struct lg_cpu *cpu)
                  * Stop doing hypercalls if they want to notify the Launcher:
                  * it needs to service this first.
                  */
-                if (cpu->pending_notify)
+                if (cpu->pending.trap)
                         break;
         }
 }
@@ -280,7 +277,7 @@ void do_hypercalls(struct lg_cpu *cpu)
          * NOTIFY to the Launcher, we want to return now.  Otherwise we do
          * the hypercall.
          */
-        if (!cpu->pending_notify) {
+        if (!cpu->pending.trap) {
                 do_hcall(cpu, cpu->hcall);
                 /*
                  * Tricky point: we reset the hcall pointer to mark the
diff --git a/drivers/lguest/lg.h b/drivers/lguest/lg.h
index 2eef40be4c04..307e8b39e7d1 100644
--- a/drivers/lguest/lg.h
+++ b/drivers/lguest/lg.h
@@ -50,7 +50,10 @@ struct lg_cpu {
         /* Bitmap of what has changed: see CHANGED_* above. */
         int changed;
 
-        unsigned long pending_notify; /* pfn from LHCALL_NOTIFY */
+        /* Pending operation. */
+        struct lguest_pending pending;
+
+        unsigned long *reg_read; /* register from LHREQ_GETREG */
 
         /* At end of a page shared mapped over lguest_pages in guest. */
         unsigned long regs_page;
@@ -78,24 +81,18 @@ struct lg_cpu {
         struct lg_cpu_arch arch;
 };
 
-struct lg_eventfd {
-        unsigned long addr;
-        struct eventfd_ctx *event;
-};
-
-struct lg_eventfd_map {
-        unsigned int num;
-        struct lg_eventfd map[];
-};
-
 /* The private info the thread maintains about the guest. */
 struct lguest {
         struct lguest_data __user *lguest_data;
         struct lg_cpu cpus[NR_CPUS];
         unsigned int nr_cpus;
 
+        /* Valid guest memory pages must be < this. */
         u32 pfn_limit;
 
+        /* Device memory is >= pfn_limit and < device_limit. */
+        u32 device_limit;
+
         /*
          * This provides the offset to the base of guest-physical memory in the
          * Launcher.
@@ -110,8 +107,6 @@ struct lguest {
         unsigned int stack_pages;
         u32 tsc_khz;
 
-        struct lg_eventfd_map *eventfds;
-
         /* Dead? */
         const char *dead;
 };
@@ -197,8 +192,10 @@ void guest_pagetable_flush_user(struct lg_cpu *cpu);
 void guest_set_pte(struct lg_cpu *cpu, unsigned long gpgdir,
                    unsigned long vaddr, pte_t val);
 void map_switcher_in_guest(struct lg_cpu *cpu, struct lguest_pages *pages);
-bool demand_page(struct lg_cpu *cpu, unsigned long cr2, int errcode);
+bool demand_page(struct lg_cpu *cpu, unsigned long cr2, int errcode,
+                 unsigned long *iomem);
 void pin_page(struct lg_cpu *cpu, unsigned long vaddr);
+bool __guest_pa(struct lg_cpu *cpu, unsigned long vaddr, unsigned long *paddr);
 unsigned long guest_pa(struct lg_cpu *cpu, unsigned long vaddr);
 void page_table_guest_data_init(struct lg_cpu *cpu);
 
@@ -210,6 +207,7 @@ void lguest_arch_handle_trap(struct lg_cpu *cpu);
 int lguest_arch_init_hypercalls(struct lg_cpu *cpu);
 int lguest_arch_do_hcall(struct lg_cpu *cpu, struct hcall_args *args);
 void lguest_arch_setup_regs(struct lg_cpu *cpu, unsigned long start);
+unsigned long *lguest_arch_regptr(struct lg_cpu *cpu, size_t reg_off, bool any);
 
 /* <arch>/switcher.S: */
 extern char start_switcher_text[], end_switcher_text[], switch_to_guest[];
diff --git a/drivers/lguest/lguest_device.c b/drivers/lguest/lguest_device.c
deleted file mode 100644
index 89088d6538fd..000000000000
--- a/drivers/lguest/lguest_device.c
+++ /dev/null
@@ -1,540 +0,0 @@
-/*P:050
- * Lguest guests use a very simple method to describe devices.  It's a
- * series of device descriptors contained just above the top of normal Guest
- * memory.
- *
- * We use the standard "virtio" device infrastructure, which provides us with a
- * console, a network and a block driver.  Each one expects some configuration
- * information and a "virtqueue" or two to send and receive data.
-:*/
-#include <linux/init.h>
-#include <linux/bootmem.h>
-#include <linux/lguest_launcher.h>
-#include <linux/virtio.h>
-#include <linux/virtio_config.h>
-#include <linux/interrupt.h>
-#include <linux/virtio_ring.h>
-#include <linux/err.h>
-#include <linux/export.h>
-#include <linux/slab.h>
-#include <asm/io.h>
-#include <asm/paravirt.h>
-#include <asm/lguest_hcall.h>
-
-/* The pointer to our (page) of device descriptions. */
-static void *lguest_devices;
-
-/*
- * For Guests, device memory can be used as normal memory, so we cast away the
- * __iomem to quieten sparse.
- */
-static inline void *lguest_map(unsigned long phys_addr, unsigned long pages)
-{
-        return (__force void *)ioremap_cache(phys_addr, PAGE_SIZE*pages);
-}
-
-static inline void lguest_unmap(void *addr)
-{
-        iounmap((__force void __iomem *)addr);
-}
-
-/*D:100
- * Each lguest device is just a virtio device plus a pointer to its entry
- * in the lguest_devices page.
- */
-struct lguest_device {
-        struct virtio_device vdev;
-
-        /* The entry in the lguest_devices page for this device. */
-        struct lguest_device_desc *desc;
-};
-
-/*
- * Since the virtio infrastructure hands us a pointer to the virtio_device all
- * the time, it helps to have a curt macro to get a pointer to the struct
- * lguest_device it's enclosed in.
- */
-#define to_lgdev(vd) container_of(vd, struct lguest_device, vdev)
-
-/*D:130
- * Device configurations
- *
- * The configuration information for a device consists of one or more
- * virtqueues, a feature bitmap, and some configuration bytes.  The
- * configuration bytes don't really matter to us: the Launcher sets them up, and
- * the driver will look at them during setup.
- *
- * A convenient routine to return the device's virtqueue config array:
- * immediately after the descriptor.
- */
-static struct lguest_vqconfig *lg_vq(const struct lguest_device_desc *desc)
-{
-        return (void *)(desc + 1);
-}
-
-/* The features come immediately after the virtqueues. */
-static u8 *lg_features(const struct lguest_device_desc *desc)
-{
-        return (void *)(lg_vq(desc) + desc->num_vq);
-}
-
-/* The config space comes after the two feature bitmasks. */
-static u8 *lg_config(const struct lguest_device_desc *desc)
-{
-        return lg_features(desc) + desc->feature_len * 2;
-}
-
-/* The total size of the config page used by this device (incl. desc) */
-static unsigned desc_size(const struct lguest_device_desc *desc)
-{
-        return sizeof(*desc)
-                + desc->num_vq * sizeof(struct lguest_vqconfig)
-                + desc->feature_len * 2
-                + desc->config_len;
-}
-
-/* This gets the device's feature bits. */
-static u64 lg_get_features(struct virtio_device *vdev)
-{
-        unsigned int i;
-        u32 features = 0;
-        struct lguest_device_desc *desc = to_lgdev(vdev)->desc;
-        u8 *in_features = lg_features(desc);
-
-        /* We do this the slow but generic way. */
-        for (i = 0; i < min(desc->feature_len * 8, 32); i++)
-                if (in_features[i / 8] & (1 << (i % 8)))
-                        features |= (1 << i);
-
-        return features;
-}
-
-/*
- * To notify on reset or feature finalization, we (ab)use the NOTIFY
- * hypercall, with the descriptor address of the device.
- */
-static void status_notify(struct virtio_device *vdev)
-{
-        unsigned long offset = (void *)to_lgdev(vdev)->desc - lguest_devices;
-
-        hcall(LHCALL_NOTIFY, (max_pfn << PAGE_SHIFT) + offset, 0, 0, 0);
-}
-
-/*
- * The virtio core takes the features the Host offers, and copies the ones
- * supported by the driver into the vdev->features array.  Once that's all
- * sorted out, this routine is called so we can tell the Host which features we
- * understand and accept.
- */
-static int lg_finalize_features(struct virtio_device *vdev)
-{
-        unsigned int i, bits;
-        struct lguest_device_desc *desc = to_lgdev(vdev)->desc;
-        /* Second half of bitmap is features we accept. */
-        u8 *out_features = lg_features(desc) + desc->feature_len;
-
-        /* Give virtio_ring a chance to accept features. */
-        vring_transport_features(vdev);
-
-        /* Make sure we don't have any features > 32 bits! */
-        BUG_ON((u32)vdev->features != vdev->features);
-
-        /*
-         * Since lguest is currently x86-only, we're little-endian.  That
-         * means we could just memcpy.  But it's not time critical, and in
-         * case someone copies this code, we do it the slow, obvious way.
-         */
-        memset(out_features, 0, desc->feature_len);
-        bits = min_t(unsigned, desc->feature_len, sizeof(vdev->features)) * 8;
-        for (i = 0; i < bits; i++) {
-                if (__virtio_test_bit(vdev, i))
-                        out_features[i / 8] |= (1 << (i % 8));
-        }
-
-        /* Tell Host we've finished with this device's feature negotiation */
-        status_notify(vdev);
-
-        return 0;
-}
-
-/* Once they've found a field, getting a copy of it is easy. */
-static void lg_get(struct virtio_device *vdev, unsigned int offset,
-                   void *buf, unsigned len)
-{
-        struct lguest_device_desc *desc = to_lgdev(vdev)->desc;
-
-        /* Check they didn't ask for more than the length of the config! */
-        BUG_ON(offset + len > desc->config_len);
-        memcpy(buf, lg_config(desc) + offset, len);
-}
-
-/* Setting the contents is also trivial. */
-static void lg_set(struct virtio_device *vdev, unsigned int offset,
-                   const void *buf, unsigned len)
-{
-        struct lguest_device_desc *desc = to_lgdev(vdev)->desc;
-
-        /* Check they didn't ask for more than the length of the config! */
-        BUG_ON(offset + len > desc->config_len);
-        memcpy(lg_config(desc) + offset, buf, len);
-}
-
-/*
- * The operations to get and set the status word just access the status field
- * of the device descriptor.
- */
-static u8 lg_get_status(struct virtio_device *vdev)
-{
-        return to_lgdev(vdev)->desc->status;
-}
-
-static void lg_set_status(struct virtio_device *vdev, u8 status)
-{
-        BUG_ON(!status);
-        to_lgdev(vdev)->desc->status = status;
-
-        /* Tell Host immediately if we failed. */
-        if (status & VIRTIO_CONFIG_S_FAILED)
-                status_notify(vdev);
-}
-
-static void lg_reset(struct virtio_device *vdev)
-{
-        /* 0 status means "reset" */
-        to_lgdev(vdev)->desc->status = 0;
-        status_notify(vdev);
-}
-
-/*
- * Virtqueues
- *
- * The other piece of infrastructure virtio needs is a "virtqueue": a way of
- * the Guest device registering buffers for the other side to read from or
- * write into (ie. send and receive buffers).  Each device can have multiple
- * virtqueues: for example the console driver uses one queue for sending and
- * another for receiving.
- *
- * Fortunately for us, a very fast shared-memory-plus-descriptors virtqueue
- * already exists in virtio_ring.c.  We just need to connect it up.
- *
- * We start with the information we need to keep about each virtqueue.
- */
-
-/*D:140 This is the information we remember about each virtqueue. */
-struct lguest_vq_info {
-        /* A copy of the information contained in the device config. */
-        struct lguest_vqconfig config;
-
-        /* The address where we mapped the virtio ring, so we can unmap it. */
-        void *pages;
-};
-
-/*
- * When the virtio_ring code wants to prod the Host, it calls us here and we
- * make a hypercall.  We hand the physical address of the virtqueue so the Host
- * knows which virtqueue we're talking about.
- */
-static bool lg_notify(struct virtqueue *vq)
-{
-        /*
-         * We store our virtqueue information in the "priv" pointer of the
-         * virtqueue structure.
-         */
-        struct lguest_vq_info *lvq = vq->priv;
-
-        hcall(LHCALL_NOTIFY, lvq->config.pfn << PAGE_SHIFT, 0, 0, 0);
-        return true;
-}
-
-/* An extern declaration inside a C file is bad form.  Don't do it. */
-extern int lguest_setup_irq(unsigned int irq);
-
-/*
- * This routine finds the Nth virtqueue described in the configuration of
- * this device and sets it up.
- *
- * This is kind of an ugly duckling.  It'd be nicer to have a standard
- * representation of a virtqueue in the configuration space, but it seems that
- * everyone wants to do it differently.  The KVM coders want the Guest to
- * allocate its own pages and tell the Host where they are, but for lguest it's
- * simpler for the Host to simply tell us where the pages are.
- */
-static struct virtqueue *lg_find_vq(struct virtio_device *vdev,
-                                    unsigned index,
-                                    void (*callback)(struct virtqueue *vq),
-                                    const char *name)
-{
-        struct lguest_device *ldev = to_lgdev(vdev);
-        struct lguest_vq_info *lvq;
-        struct virtqueue *vq;
-        int err;
-
-        if (!name)
-                return NULL;
-
-        /* We must have this many virtqueues. */
-        if (index >= ldev->desc->num_vq)
-                return ERR_PTR(-ENOENT);
-
-        lvq = kmalloc(sizeof(*lvq), GFP_KERNEL);
-        if (!lvq)
-                return ERR_PTR(-ENOMEM);
-
-        /*
-         * Make a copy of the "struct lguest_vqconfig" entry, which sits after
-         * the descriptor.  We need a copy because the config space might not
-         * be aligned correctly.
-         */
-        memcpy(&lvq->config, lg_vq(ldev->desc)+index, sizeof(lvq->config));
-
-        printk("Mapping virtqueue %i addr %lx\n", index,
-               (unsigned long)lvq->config.pfn << PAGE_SHIFT);
-        /* Figure out how many pages the ring will take, and map that memory */
-        lvq->pages = lguest_map((unsigned long)lvq->config.pfn << PAGE_SHIFT,
-                                DIV_ROUND_UP(vring_size(lvq->config.num,
-                                                        LGUEST_VRING_ALIGN),
-                                             PAGE_SIZE));
-        if (!lvq->pages) {
-                err = -ENOMEM;
-                goto free_lvq;
-        }
-
-        /*
-         * OK, tell virtio_ring.c to set up a virtqueue now we know its size
-         * and we've got a pointer to its pages.  Note that we set weak_barriers
-         * to 'true': the host just a(nother) SMP CPU, so we only need inter-cpu
-         * barriers.
-         */
-        vq = vring_new_virtqueue(index, lvq->config.num, LGUEST_VRING_ALIGN, vdev,
-                                 true, lvq->pages, lg_notify, callback, name);
-        if (!vq) {
-                err = -ENOMEM;
-                goto unmap;
-        }
-
-        /* Make sure the interrupt is allocated. */
-        err = lguest_setup_irq(lvq->config.irq);
-        if (err)
-                goto destroy_vring;
-
-        /*
-         * Tell the interrupt for this virtqueue to go to the virtio_ring
-         * interrupt handler.
-         *
-         * FIXME: We used to have a flag for the Host to tell us we could use
-         * the interrupt as a source of randomness: it'd be nice to have that
-         * back.
-         */
-        err = request_irq(lvq->config.irq, vring_interrupt, IRQF_SHARED,
-                          dev_name(&vdev->dev), vq);
-        if (err)
-                goto free_desc;
-
-        /*
-         * Last of all we hook up our 'struct lguest_vq_info" to the
-         * virtqueue's priv pointer.
-         */
-        vq->priv = lvq;
-        return vq;
-
-free_desc:
-        irq_free_desc(lvq->config.irq);
-destroy_vring:
-        vring_del_virtqueue(vq);
-unmap:
-        lguest_unmap(lvq->pages);
-free_lvq:
-        kfree(lvq);
-        return ERR_PTR(err);
-}
-/*:*/
-
-/* Cleaning up a virtqueue is easy */
-static void lg_del_vq(struct virtqueue *vq)
-{
-        struct lguest_vq_info *lvq = vq->priv;
-
-        /* Release the interrupt */
-        free_irq(lvq->config.irq, vq);
-        /* Tell virtio_ring.c to free the virtqueue. */
-        vring_del_virtqueue(vq);
-        /* Unmap the pages containing the ring. */
-        lguest_unmap(lvq->pages);
-        /* Free our own queue information. */
-        kfree(lvq);
-}
-
-static void lg_del_vqs(struct virtio_device *vdev)
-{
-        struct virtqueue *vq, *n;
-
-        list_for_each_entry_safe(vq, n, &vdev->vqs, list)
-                lg_del_vq(vq);
-}
-
-static int lg_find_vqs(struct virtio_device *vdev, unsigned nvqs,
-                       struct virtqueue *vqs[],
-                       vq_callback_t *callbacks[],
-                       const char *names[])
-{
-        struct lguest_device *ldev = to_lgdev(vdev);
-        int i;
-
-        /* We must have this many virtqueues. */
-        if (nvqs > ldev->desc->num_vq)
-                return -ENOENT;
-
-        for (i = 0; i < nvqs; ++i) {
-                vqs[i] = lg_find_vq(vdev, i, callbacks[i], names[i]);
-                if (IS_ERR(vqs[i]))
-                        goto error;
-        }
-        return 0;
-
-error:
-        lg_del_vqs(vdev);
-        return PTR_ERR(vqs[i]);
-}
-
-static const char *lg_bus_name(struct virtio_device *vdev)
-{
-        return "";
-}
-
-/* The ops structure which hooks everything together. */
-static const struct virtio_config_ops lguest_config_ops = {
-        .get_features = lg_get_features,
-        .finalize_features = lg_finalize_features,
-        .get = lg_get,
-        .set = lg_set,
-        .get_status = lg_get_status,
-        .set_status = lg_set_status,
-        .reset = lg_reset,
-        .find_vqs = lg_find_vqs,
-        .del_vqs = lg_del_vqs,
-        .bus_name = lg_bus_name,
-};
-
-/*
- * The root device for the lguest virtio devices.  This makes them appear as
- * /sys/devices/lguest/0,1,2 not /sys/devices/0,1,2.
- */
-static struct device *lguest_root;
-
-/*D:120
- * This is the core of the lguest bus: actually adding a new device.
- * It's a separate function because it's neater that way, and because an
- * earlier version of the code supported hotplug and unplug.  They were removed
- * early on because they were never used.
- *
- * As Andrew Tridgell says, "Untested code is buggy code".
- *
- * It's worth reading this carefully: we start with a pointer to the new device
- * descriptor in the "lguest_devices" page, and the offset into the device
- * descriptor page so we can uniquely identify it if things go badly wrong.
- */
-static void add_lguest_device(struct lguest_device_desc *d,
-                              unsigned int offset)
-{
-        struct lguest_device *ldev;
-
-        /* Start with zeroed memory; Linux's device layer counts on it. */
-        ldev = kzalloc(sizeof(*ldev), GFP_KERNEL);
-        if (!ldev) {
-                printk(KERN_EMERG "Cannot allocate lguest dev %u type %u\n",
-                       offset, d->type);
-                return;
-        }
-
-        /* This devices' parent is the lguest/ dir. */
-        ldev->vdev.dev.parent = lguest_root;
-        /*
-         * The device type comes straight from the descriptor.  There's also a
-         * device vendor field in the virtio_device struct, which we leave as
-         * 0.
-         */
-        ldev->vdev.id.device = d->type;
-        /*
-         * We have a simple set of routines for querying the device's
-         * configuration information and setting its status.
-         */
-        ldev->vdev.config = &lguest_config_ops;
-        /* And we remember the device's descriptor for lguest_config_ops. */
-        ldev->desc = d;
-
-        /*
-         * register_virtio_device() sets up the generic fields for the struct
-         * virtio_device and calls device_register().  This makes the bus
-         * infrastructure look for a matching driver.
-         */
-        if (register_virtio_device(&ldev->vdev) != 0) {
-                printk(KERN_ERR "Failed to register lguest dev %u type %u\n",
-                       offset, d->type);
-                kfree(ldev);
-        }
-}
-
-/*D:110
- * scan_devices() simply iterates through the device page.  The type 0 is
- * reserved to mean "end of devices".
- */
-static void scan_devices(void)
-{
-        unsigned int i;
-        struct lguest_device_desc *d;
-
-        /* We start at the page beginning, and skip over each entry. */
-        for (i = 0; i < PAGE_SIZE; i += desc_size(d)) {
-                d = lguest_devices + i;
-
-                /* Once we hit a zero, stop. */
-                if (d->type == 0)
-                        break;
-
-                printk("Device at %i has size %u\n", i, desc_size(d));
-                add_lguest_device(d, i);
-        }
-}
-
-/*D:105
- * Fairly early in boot, lguest_devices_init() is called to set up the
- * lguest device infrastructure.  We check that we are a Guest by checking
- * pv_info.name: there are other ways of checking, but this seems most
- * obvious to me.
- *
- * So we can access the "struct lguest_device_desc"s easily, we map that memory
- * and store the pointer in the global "lguest_devices".  Then we register a
- * root device from which all our devices will hang (this seems to be the
- * correct sysfs incantation).
- *
- * Finally we call scan_devices() which adds all the devices found in the
- * lguest_devices page.
- */
-static int __init lguest_devices_init(void)
-{
-        if (strcmp(pv_info.name, "lguest") != 0)
-                return 0;
-
-        lguest_root = root_device_register("lguest");
-        if (IS_ERR(lguest_root))
-                panic("Could not register lguest root");
-
-        /* Devices are in a single page above top of "normal" mem */
-        lguest_devices = lguest_map(max_pfn<<PAGE_SHIFT, 1);
-
-        scan_devices();
-        return 0;
-}
-/* We do this after core stuff, but before the drivers. */
-postcore_initcall(lguest_devices_init);
-
-/*D:150
- * At this point in the journey we used to now wade through the lguest
- * devices themselves: net, block and console.  Since they're all now virtio
- * devices rather than lguest-specific, I've decided to ignore them.  Mostly,
- * they're kind of boring.  But this does mean you'll never experience the
- * thrill of reading the forbidden love scene buried deep in the block driver.
- *
- * "make Launcher" beckons, where we answer questions like "Where do Guests
- * come from?", and "What do you do when someone asks for optimization?".
- */
diff --git a/drivers/lguest/lguest_user.c b/drivers/lguest/lguest_user.c
index 4263f4cc8c55..c4c6113eb9a6 100644
--- a/drivers/lguest/lguest_user.c
+++ b/drivers/lguest/lguest_user.c
@@ -2,175 +2,62 @@
  * launcher controls and communicates with the Guest.  For example,
  * the first write will tell us the Guest's memory layout and entry
  * point.  A read will run the Guest until something happens, such as
- * a signal or the Guest doing a NOTIFY out to the Launcher.  There is
- * also a way for the Launcher to attach eventfds to particular NOTIFY
- * values instead of returning from the read() call.
+ * a signal or the Guest accessing a device.
 :*/
 #include <linux/uaccess.h>
 #include <linux/miscdevice.h>
 #include <linux/fs.h>
 #include <linux/sched.h>
-#include <linux/eventfd.h>
 #include <linux/file.h>
 #include <linux/slab.h>
 #include <linux/export.h>
 #include "lg.h"
 
-/*L:056
- * Before we move on, let's jump ahead and look at what the kernel does when
- * it needs to look up the eventfds.  That will complete our picture of how we
- * use RCU.
- *
- * The notification value is in cpu->pending_notify: we return true if it went
- * to an eventfd.
- */
-bool send_notify_to_eventfd(struct lg_cpu *cpu)
-{
-        unsigned int i;
-        struct lg_eventfd_map *map;
-
-        /*
-         * This "rcu_read_lock()" helps track when someone is still looking at
-         * the (RCU-using) eventfds array.  It's not actually a lock at all;
-         * indeed it's a noop in many configurations.  (You didn't expect me to
-         * explain all the RCU secrets here, did you?)
-         */
-        rcu_read_lock();
-        /*
-         * rcu_dereference is the counter-side of rcu_assign_pointer(); it
-         * makes sure we don't access the memory pointed to by
-         * cpu->lg->eventfds before cpu->lg->eventfds is set.  Sounds crazy,
-         * but Alpha allows this!  Paul McKenney points out that a really
-         * aggressive compiler could have the same effect:
-         *   http://lists.ozlabs.org/pipermail/lguest/2009-July/001560.html
-         *
-         * So play safe, use rcu_dereference to get the rcu-protected pointer:
-         */
-        map = rcu_dereference(cpu->lg->eventfds);
-        /*
-         * Simple array search: even if they add an eventfd while we do this,
-         * we'll continue to use the old array and just won't see the new one.
-         */
-        for (i = 0; i < map->num; i++) {
-                if (map->map[i].addr == cpu->pending_notify) {
-                        eventfd_signal(map->map[i].event, 1);
-                        cpu->pending_notify = 0;
-                        break;
-                }
-        }
-        /* We're done with the rcu-protected variable cpu->lg->eventfds. */
-        rcu_read_unlock();
-
-        /* If we cleared the notification, it's because we found a match. */
-        return cpu->pending_notify == 0;
-}
-
-/*L:055
- * One of the more tricksy tricks in the Linux Kernel is a technique called
- * Read Copy Update.  Since one point of lguest is to teach lguest journeyers
- * about kernel coding, I use it here.  (In case you're curious, other purposes
- * include learning about virtualization and instilling a deep appreciation for
- * simplicity and puppies).
- *
- * We keep a simple array which maps LHCALL_NOTIFY values to eventfds, but we
- * add new eventfds without ever blocking readers from accessing the array.
- * The current Launcher only does this during boot, so that never happens.  But
- * Read Copy Update is cool, and adding a lock risks damaging even more puppies
- * than this code does.
- *
- * We allocate a brand new one-larger array, copy the old one and add our new
- * element.  Then we make the lg eventfd pointer point to the new array.
- * That's the easy part: now we need to free the old one, but we need to make
- * sure no slow CPU somewhere is still looking at it.  That's what
- * synchronize_rcu does for us: waits until every CPU has indicated that it has
- * moved on to know it's no longer using the old one.
- *
- * If that's unclear, see http://en.wikipedia.org/wiki/Read-copy-update.
- */
-static int add_eventfd(struct lguest *lg, unsigned long addr, int fd)
+/*L:052
+  The Launcher can get the registers, and also set some of them.
+*/
+static int getreg_setup(struct lg_cpu *cpu, const unsigned long __user *input)
 {
-        struct lg_eventfd_map *new, *old = lg->eventfds;
-
-        /*
-         * We don't allow notifications on value 0 anyway (pending_notify of
-         * 0 means "nothing pending").
-         */
-        if (!addr)
-                return -EINVAL;
-
-        /*
-         * Replace the old array with the new one, carefully: others can
-         * be accessing it at the same time.
-         */
-        new = kmalloc(sizeof(*new) + sizeof(new->map[0]) * (old->num + 1),
-                      GFP_KERNEL);
-        if (!new)
-                return -ENOMEM;
+        unsigned long which;
 
-        /* First make identical copy. */
-        memcpy(new->map, old->map, sizeof(old->map[0]) * old->num);
-        new->num = old->num;
-
-        /* Now append new entry. */
-        new->map[new->num].addr = addr;
-        new->map[new->num].event = eventfd_ctx_fdget(fd);
-        if (IS_ERR(new->map[new->num].event)) {
-                int err =  PTR_ERR(new->map[new->num].event);
-                kfree(new);
-                return err;
-        }
-        new->num++;
+        /* We re-use the ptrace structure to specify which register to read. */
+        if (get_user(which, input) != 0)
+                return -EFAULT;
 
         /*
-         * Now put new one in place: rcu_assign_pointer() is a fancy way of
-         * doing "lg->eventfds = new", but it uses memory barriers to make
-         * absolutely sure that the contents of "new" written above is nailed
-         * down before we actually do the assignment.
+         * We set up the cpu register pointer, and their next read will
+         * actually get the value (instead of running the guest).
          *
-         * We have to think about these kinds of things when we're operating on
-         * live data without locks.
+         * The last argument 'true' says we can access any register.
          */
-        rcu_assign_pointer(lg->eventfds, new);
+        cpu->reg_read = lguest_arch_regptr(cpu, which, true);
+        if (!cpu->reg_read)
+                return -ENOENT;
 
-        /*
-         * We're not in a big hurry.  Wait until no one's looking at old
-         * version, then free it.
-         */
-        synchronize_rcu();
-        kfree(old);
-
-        return 0;
+        /* And because this is a write() call, we return the length used. */
+        return sizeof(unsigned long) * 2;
 }
 
-/*L:052
- * Receiving notifications from the Guest is usually done by attaching a
- * particular LHCALL_NOTIFY value to an event filedescriptor.  The eventfd will
- * become readable when the Guest does an LHCALL_NOTIFY with that value.
- *
- * This is really convenient for processing each virtqueue in a separate
- * thread.
- */
-static int attach_eventfd(struct lguest *lg, const unsigned long __user *input)
+static int setreg(struct lg_cpu *cpu, const unsigned long __user *input)
 {
-        unsigned long addr, fd;
-        int err;
+        unsigned long which, value, *reg;
 
-        if (get_user(addr, input) != 0)
+        /* We re-use the ptrace structure to specify which register to read. */
+        if (get_user(which, input) != 0)
                 return -EFAULT;
         input++;
-        if (get_user(fd, input) != 0)
+        if (get_user(value, input) != 0)
                 return -EFAULT;
 
-        /*
-         * Just make sure two callers don't add eventfds at once.  We really
-         * only need to lock against callers adding to the same Guest, so using
-         * the Big Lguest Lock is overkill.  But this is setup, not a fast path.
-         */
-        mutex_lock(&lguest_lock);
-        err = add_eventfd(lg, addr, fd);
-        mutex_unlock(&lguest_lock);
+        /* The last argument 'false' means we can't access all registers. */
+        reg = lguest_arch_regptr(cpu, which, false);
+        if (!reg)
+                return -ENOENT;
 
-        return err;
+        *reg = value;
+
+        /* And because this is a write() call, we return the length used. */
+        return sizeof(unsigned long) * 3;
 }
 
 /*L:050
@@ -194,6 +81,23 @@ static int user_send_irq(struct lg_cpu *cpu, const unsigned long __user *input)
         return 0;
 }
 
+/*L:053
+ * Deliver a trap: this is used by the Launcher if it can't emulate
+ * an instruction.
+ */
+static int trap(struct lg_cpu *cpu, const unsigned long __user *input)
+{
+        unsigned long trapnum;
+
+        if (get_user(trapnum, input) != 0)
+                return -EFAULT;
+
+        if (!deliver_trap(cpu, trapnum))
+                return -EINVAL;
+
+        return 0;
+}
+
 /*L:040
  * Once our Guest is initialized, the Launcher makes it run by reading
  * from /dev/lguest.
@@ -237,8 +141,8 @@ static ssize_t read(struct file *file, char __user *user, size_t size,loff_t*o)
          * If we returned from read() last time because the Guest sent I/O,
          * clear the flag.
          */
-        if (cpu->pending_notify)
-                cpu->pending_notify = 0;
+        if (cpu->pending.trap)
+                cpu->pending.trap = 0;
 
         /* Run the Guest until something interesting happens. */
         return run_guest(cpu, (unsigned long __user *)user);
@@ -319,7 +223,7 @@ static int initialize(struct file *file, const unsigned long __user *input)
         /* "struct lguest" contains all we (the Host) know about a Guest. */
         struct lguest *lg;
         int err;
-        unsigned long args[3];
+        unsigned long args[4];
 
         /*
          * We grab the Big Lguest lock, which protects against multiple
@@ -343,21 +247,15 @@ static int initialize(struct file *file, const unsigned long __user *input)
                 goto unlock;
         }
 
-        lg->eventfds = kmalloc(sizeof(*lg->eventfds), GFP_KERNEL);
-        if (!lg->eventfds) {
-                err = -ENOMEM;
-                goto free_lg;
-        }
-        lg->eventfds->num = 0;
-
         /* Populate the easy fields of our "struct lguest" */
         lg->mem_base = (void __user *)args[0];
         lg->pfn_limit = args[1];
+        lg->device_limit = args[3];
 
         /* This is the first cpu (cpu 0) and it will start booting at args[2] */
         err = lg_cpu_start(&lg->cpus[0], 0, args[2]);
         if (err)
-                goto free_eventfds;
+                goto free_lg;
 
         /*
          * Initialize the Guest's shadow page tables.  This allocates
@@ -378,8 +276,6 @@ static int initialize(struct file *file, const unsigned long __user *input)
 free_regs:
         /* FIXME: This should be in free_vcpu */
         free_page(lg->cpus[0].regs_page);
-free_eventfds:
-        kfree(lg->eventfds);
 free_lg:
         kfree(lg);
 unlock:
@@ -432,8 +328,12 @@ static ssize_t write(struct file *file, const char __user *in,
                 return initialize(file, input);
         case LHREQ_IRQ:
                 return user_send_irq(cpu, input);
-        case LHREQ_EVENTFD:
-                return attach_eventfd(lg, input);
+        case LHREQ_GETREG:
+                return getreg_setup(cpu, input);
+        case LHREQ_SETREG:
+                return setreg(cpu, input);
+        case LHREQ_TRAP:
+                return trap(cpu, input);
         default:
                 return -EINVAL;
         }
@@ -478,11 +378,6 @@ static int close(struct inode *inode, struct file *file)
                 mmput(lg->cpus[i].mm);
         }
 
-        /* Release any eventfds they registered. */
-        for (i = 0; i < lg->eventfds->num; i++)
-                eventfd_ctx_put(lg->eventfds->map[i].event);
-        kfree(lg->eventfds);
-
         /*
          * If lg->dead doesn't contain an error code it will be NULL or a
          * kmalloc()ed string, either of which is ok to hand to kfree().
diff --git a/drivers/lguest/page_tables.c b/drivers/lguest/page_tables.c
index e8b55c3a6170..e3abebc912c0 100644
--- a/drivers/lguest/page_tables.c
+++ b/drivers/lguest/page_tables.c
@@ -250,6 +250,16 @@ static void release_pte(pte_t pte)
 }
 /*:*/
 
+static bool gpte_in_iomem(struct lg_cpu *cpu, pte_t gpte)
+{
+        /* We don't handle large pages. */
+        if (pte_flags(gpte) & _PAGE_PSE)
+                return false;
+
+        return (pte_pfn(gpte) >= cpu->lg->pfn_limit
+                && pte_pfn(gpte) < cpu->lg->device_limit);
+}
+
 static bool check_gpte(struct lg_cpu *cpu, pte_t gpte)
 {
         if ((pte_flags(gpte) & _PAGE_PSE) ||
@@ -374,8 +384,14 @@ static pte_t *find_spte(struct lg_cpu *cpu, unsigned long vaddr, bool allocate,
  *
  * If we fixed up the fault (ie. we mapped the address), this routine returns
  * true.  Otherwise, it was a real fault and we need to tell the Guest.
+ *
+ * There's a corner case: they're trying to access memory between
+ * pfn_limit and device_limit, which is I/O memory.  In this case, we
+ * return false and set @iomem to the physical address, so the the
+ * Launcher can handle the instruction manually.
  */
-bool demand_page(struct lg_cpu *cpu, unsigned long vaddr, int errcode)
+bool demand_page(struct lg_cpu *cpu, unsigned long vaddr, int errcode,
+                 unsigned long *iomem)
 {
         unsigned long gpte_ptr;
         pte_t gpte;
@@ -383,6 +399,8 @@ bool demand_page(struct lg_cpu *cpu, unsigned long vaddr, int errcode)
         pmd_t gpmd;
         pgd_t gpgd;
 
+        *iomem = 0;
+
         /* We never demand page the Switcher, so trying is a mistake. */
         if (vaddr >= switcher_addr)
                 return false;
@@ -459,6 +477,12 @@ bool demand_page(struct lg_cpu *cpu, unsigned long vaddr, int errcode)
         if ((errcode & 4) && !(pte_flags(gpte) & _PAGE_USER))
                 return false;
 
+        /* If they're accessing io memory, we expect a fault. */
+        if (gpte_in_iomem(cpu, gpte)) {
+                *iomem = (pte_pfn(gpte) << PAGE_SHIFT) | (vaddr & ~PAGE_MASK);
+                return false;
+        }
+
         /*
          * Check that the Guest PTE flags are OK, and the page number is below
          * the pfn_limit (ie. not mapping the Launcher binary).
@@ -553,7 +577,9 @@ static bool page_writable(struct lg_cpu *cpu, unsigned long vaddr)
  */
 void pin_page(struct lg_cpu *cpu, unsigned long vaddr)
 {
-        if (!page_writable(cpu, vaddr) && !demand_page(cpu, vaddr, 2))
+        unsigned long iomem;
+
+        if (!page_writable(cpu, vaddr) && !demand_page(cpu, vaddr, 2, &iomem))
                 kill_guest(cpu, "bad stack page %#lx", vaddr);
 }
 /*:*/
@@ -647,7 +673,7 @@ void guest_pagetable_flush_user(struct lg_cpu *cpu)
 /*:*/
 
 /* We walk down the guest page tables to get a guest-physical address */
-unsigned long guest_pa(struct lg_cpu *cpu, unsigned long vaddr)
+bool __guest_pa(struct lg_cpu *cpu, unsigned long vaddr, unsigned long *paddr)
 {
         pgd_t gpgd;
         pte_t gpte;
@@ -656,31 +682,47 @@ unsigned long guest_pa(struct lg_cpu *cpu, unsigned long vaddr)
 #endif
 
         /* Still not set up?  Just map 1:1. */
-        if (unlikely(cpu->linear_pages))
-                return vaddr;
+        if (unlikely(cpu->linear_pages)) {
+                *paddr = vaddr;
+                return true;
+        }
 
         /* First step: get the top-level Guest page table entry. */
         gpgd = lgread(cpu, gpgd_addr(cpu, vaddr), pgd_t);
         /* Toplevel not present?  We can't map it in. */
-        if (!(pgd_flags(gpgd) & _PAGE_PRESENT)) {
-                kill_guest(cpu, "Bad address %#lx", vaddr);
-                return -1UL;
-        }
+        if (!(pgd_flags(gpgd) & _PAGE_PRESENT))
+                goto fail;
 
 #ifdef CONFIG_X86_PAE
         gpmd = lgread(cpu, gpmd_addr(gpgd, vaddr), pmd_t);
-        if (!(pmd_flags(gpmd) & _PAGE_PRESENT)) {
-                kill_guest(cpu, "Bad address %#lx", vaddr);
-                return -1UL;
-        }
+        if (!(pmd_flags(gpmd) & _PAGE_PRESENT))
+                goto fail;
         gpte = lgread(cpu, gpte_addr(cpu, gpmd, vaddr), pte_t);
 #else
         gpte = lgread(cpu, gpte_addr(cpu, gpgd, vaddr), pte_t);
 #endif
         if (!(pte_flags(gpte) & _PAGE_PRESENT))
-                kill_guest(cpu, "Bad address %#lx", vaddr);
+                goto fail;
+
+        *paddr = pte_pfn(gpte) * PAGE_SIZE | (vaddr & ~PAGE_MASK);
+        return true;
+
+fail:
+        *paddr = -1UL;
+        return false;
+}
 
-        return pte_pfn(gpte) * PAGE_SIZE | (vaddr & ~PAGE_MASK);
+/*
+ * This is the version we normally use: kills the Guest if it uses a
+ * bad address
+ */
+unsigned long guest_pa(struct lg_cpu *cpu, unsigned long vaddr)
+{
+        unsigned long paddr;
+
+        if (!__guest_pa(cpu, vaddr, &paddr))
+                kill_guest(cpu, "Bad address %#lx", vaddr);
+        return paddr;
 }
 
 /*
@@ -912,7 +954,8 @@ static void __guest_set_pte(struct lg_cpu *cpu, int idx,
                          * now.  This shaves 10% off a copy-on-write
                          * micro-benchmark.
                          */
-                        if (pte_flags(gpte) & (_PAGE_DIRTY | _PAGE_ACCESSED)) {
+                        if ((pte_flags(gpte) & (_PAGE_DIRTY | _PAGE_ACCESSED))
+                            && !gpte_in_iomem(cpu, gpte)) {
                                 if (!check_gpte(cpu, gpte))
                                         return;
                                 set_pte(spte,
diff --git a/drivers/lguest/x86/core.c b/drivers/lguest/x86/core.c
index 6adfd7ba4c97..30f2aef69d78 100644
--- a/drivers/lguest/x86/core.c
+++ b/drivers/lguest/x86/core.c
@@ -182,6 +182,52 @@ static void run_guest_once(struct lg_cpu *cpu, struct lguest_pages *pages)
 }
 /*:*/
 
+unsigned long *lguest_arch_regptr(struct lg_cpu *cpu, size_t reg_off, bool any)
+{
+        switch (reg_off) {
+        case offsetof(struct pt_regs, bx):
+                return &cpu->regs->ebx;
+        case offsetof(struct pt_regs, cx):
+                return &cpu->regs->ecx;
+        case offsetof(struct pt_regs, dx):
+                return &cpu->regs->edx;
+        case offsetof(struct pt_regs, si):
+                return &cpu->regs->esi;
+        case offsetof(struct pt_regs, di):
+                return &cpu->regs->edi;
+        case offsetof(struct pt_regs, bp):
+                return &cpu->regs->ebp;
+        case offsetof(struct pt_regs, ax):
+                return &cpu->regs->eax;
+        case offsetof(struct pt_regs, ip):
+                return &cpu->regs->eip;
+        case offsetof(struct pt_regs, sp):
+                return &cpu->regs->esp;
+        }
+
+        /* Launcher can read these, but we don't allow any setting. */
+        if (any) {
+                switch (reg_off) {
+                case offsetof(struct pt_regs, ds):
+                        return &cpu->regs->ds;
+                case offsetof(struct pt_regs, es):
+                        return &cpu->regs->es;
+                case offsetof(struct pt_regs, fs):
+                        return &cpu->regs->fs;
+                case offsetof(struct pt_regs, gs):
+                        return &cpu->regs->gs;
+                case offsetof(struct pt_regs, cs):
+                        return &cpu->regs->cs;
+                case offsetof(struct pt_regs, flags):
+                        return &cpu->regs->eflags;
+                case offsetof(struct pt_regs, ss):
+                        return &cpu->regs->ss;
+                }
+        }
+
+        return NULL;
+}
+
 /*M:002
  * There are hooks in the scheduler which we can register to tell when we
  * get kicked off the CPU (preempt_notifier_register()).  This would allow us
@@ -269,110 +315,73 @@ void lguest_arch_run_guest(struct lg_cpu *cpu)
  * usually attached to a PC.
  *
  * When the Guest uses one of these instructions, we get a trap (General
- * Protection Fault) and come here.  We see if it's one of those troublesome
- * instructions and skip over it.  We return true if we did.
+ * Protection Fault) and come here.  We queue this to be sent out to the
+ * Launcher to handle.
  */
-static int emulate_insn(struct lg_cpu *cpu)
-{
-        u8 insn;
-        unsigned int insnlen = 0, in = 0, small_operand = 0;
-        /*
-         * The eip contains the *virtual* address of the Guest's instruction:
-         * walk the Guest's page tables to find the "physical" address.
-         */
-        unsigned long physaddr = guest_pa(cpu, cpu->regs->eip);
-
-        /*
-         * This must be the Guest kernel trying to do something, not userspace!
-         * The bottom two bits of the CS segment register are the privilege
-         * level.
-         */
-        if ((cpu->regs->cs & 3) != GUEST_PL)
-                return 0;
-
-        /* Decoding x86 instructions is icky. */
-        insn = lgread(cpu, physaddr, u8);
 
-        /*
-         * Around 2.6.33, the kernel started using an emulation for the
-         * cmpxchg8b instruction in early boot on many configurations.  This
-         * code isn't paravirtualized, and it tries to disable interrupts.
-         * Ignore it, which will Mostly Work.
-         */
-        if (insn == 0xfa) {
-                /* "cli", or Clear Interrupt Enable instruction.  Skip it. */
-                cpu->regs->eip++;
-                return 1;
+/*
+ * The eip contains the *virtual* address of the Guest's instruction:
+ * we copy the instruction here so the Launcher doesn't have to walk
+ * the page tables to decode it.  We handle the case (eg. in a kernel
+ * module) where the instruction is over two pages, and the pages are
+ * virtually but not physically contiguous.
+ *
+ * The longest possible x86 instruction is 15 bytes, but we don't handle
+ * anything that strange.
+ */
+static void copy_from_guest(struct lg_cpu *cpu,
+                            void *dst, unsigned long vaddr, size_t len)
+{
+        size_t to_page_end = PAGE_SIZE - (vaddr % PAGE_SIZE);
+        unsigned long paddr;
+
+        BUG_ON(len > PAGE_SIZE);
+
+        /* If it goes over a page, copy in two parts. */
+        if (len > to_page_end) {
+                /* But make sure the next page is mapped! */
+                if (__guest_pa(cpu, vaddr + to_page_end, &paddr))
+                        copy_from_guest(cpu, dst + to_page_end,
+                                        vaddr + to_page_end,
+                                        len - to_page_end);
+                else
+                        /* Otherwise fill with zeroes. */
+                        memset(dst + to_page_end, 0, len - to_page_end);
+                len = to_page_end;
         }
 
-        /*
-         * 0x66 is an "operand prefix".  It means a 16, not 32 bit in/out.
-         */
-        if (insn == 0x66) {
-                small_operand = 1;
-                /* The instruction is 1 byte so far, read the next byte. */
-                insnlen = 1;
-                insn = lgread(cpu, physaddr + insnlen, u8);
-        }
+        /* This will kill the guest if it isn't mapped, but that
+         * shouldn't happen. */
+        __lgread(cpu, dst, guest_pa(cpu, vaddr), len);
+}
 
-        /*
-         * We can ignore the lower bit for the moment and decode the 4 opcodes
-         * we need to emulate.
-         */
-        switch (insn & 0xFE) {
-        case 0xE4: /* in     <next byte>,%al */
-                insnlen += 2;
-                in = 1;
-                break;
-        case 0xEC: /* in     (%dx),%al */
-                insnlen += 1;
-                in = 1;
-                break;
-        case 0xE6: /* out    %al,<next byte> */
-                insnlen += 2;
-                break;
-        case 0xEE: /* out    %al,(%dx) */
-                insnlen += 1;
-                break;
-        default:
-                /* OK, we don't know what this is, can't emulate. */
-                return 0;
-        }
 
-        /*
-         * If it was an "IN" instruction, they expect the result to be read
-         * into %eax, so we change %eax.  We always return all-ones, which
-         * traditionally means "there's nothing there".
-         */
-        if (in) {
-                /* Lower bit tells means it's a 32/16 bit access */
-                if (insn & 0x1) {
-                        if (small_operand)
-                                cpu->regs->eax |= 0xFFFF;
-                        else
-                                cpu->regs->eax = 0xFFFFFFFF;
-                } else
-                        cpu->regs->eax |= 0xFF;
-        }
-        /* Finally, we've "done" the instruction, so move past it. */
-        cpu->regs->eip += insnlen;
-        /* Success! */
-        return 1;
+static void setup_emulate_insn(struct lg_cpu *cpu)
+{
+        cpu->pending.trap = 13;
+        copy_from_guest(cpu, cpu->pending.insn, cpu->regs->eip,
+                        sizeof(cpu->pending.insn));
+}
+
+static void setup_iomem_insn(struct lg_cpu *cpu, unsigned long iomem_addr)
+{
+        cpu->pending.trap = 14;
+        cpu->pending.addr = iomem_addr;
+        copy_from_guest(cpu, cpu->pending.insn, cpu->regs->eip,
+                        sizeof(cpu->pending.insn));
 }
 
 /*H:050 Once we've re-enabled interrupts, we look at why the Guest exited. */
 void lguest_arch_handle_trap(struct lg_cpu *cpu)
 {
+        unsigned long iomem_addr;
+
         switch (cpu->regs->trapnum) {
         case 13: /* We've intercepted a General Protection Fault. */
-                /*
-                 * Check if this was one of those annoying IN or OUT
-                 * instructions which we need to emulate.  If so, we just go
-                 * back into the Guest after we've done it.
-                 */
+                /* Hand to Launcher to emulate those pesky IN and OUT insns */
                 if (cpu->regs->errcode == 0) {
-                        if (emulate_insn(cpu))
-                                return;
+                        setup_emulate_insn(cpu);
+                        return;
                 }
                 break;
         case 14: /* We've intercepted a Page Fault. */
@@ -387,9 +396,16 @@ void lguest_arch_handle_trap(struct lg_cpu *cpu)
                  * whether kernel or userspace code.
                  */
                 if (demand_page(cpu, cpu->arch.last_pagefault,
-                                cpu->regs->errcode))
+                                cpu->regs->errcode, &iomem_addr))
                         return;
 
+                /* Was this an access to memory mapped IO? */
+                if (iomem_addr) {
+                        /* Tell Launcher, let it handle it. */
+                        setup_iomem_insn(cpu, iomem_addr);
+                        return;
+                }
+
                 /*
                  * OK, it's really not there (or not OK): the Guest needs to
                  * know.  We write out the cr2 value so it knows where the
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 110a2cf67244..f1ff3666f090 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -1710,6 +1710,12 @@ static int virtnet_probe(struct virtio_device *vdev)
         struct virtnet_info *vi;
         u16 max_queue_pairs;
 
+        if (!vdev->config->get) {
+                dev_err(&vdev->dev, "%s failure: config access disabled\n",
+                        __func__);
+                return -EINVAL;
+        }
+
         if (!virtnet_validate_features(vdev))
                 return -EINVAL;
 
diff --git a/drivers/scsi/virtio_scsi.c b/drivers/scsi/virtio_scsi.c
index c52bb5dfaedb..f164f24a4a55 100644
--- a/drivers/scsi/virtio_scsi.c
+++ b/drivers/scsi/virtio_scsi.c
@@ -950,6 +950,12 @@ static int virtscsi_probe(struct virtio_device *vdev)
         u32 num_queues;
         struct scsi_host_template *hostt;
 
+        if (!vdev->config->get) {
+                dev_err(&vdev->dev, "%s failure: config access disabled\n",
+                        __func__);
+                return -EINVAL;
+        }
+
         /* We need to know how many queues before we allocate. */
         num_queues = virtscsi_config_get(vdev, num_queues) ? : 1;
 
diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig
index 00b228638274..b546da5d8ea3 100644
--- a/drivers/virtio/Kconfig
+++ b/drivers/virtio/Kconfig
@@ -12,16 +12,32 @@ config VIRTIO_PCI
         depends on PCI
         select VIRTIO
         ---help---
-          This drivers provides support for virtio based paravirtual device
+          This driver provides support for virtio based paravirtual device
           drivers over PCI.  This requires that your VMM has appropriate PCI
           virtio backends.  Most QEMU based VMMs should support these devices
           (like KVM or Xen).
 
-          Currently, the ABI is not considered stable so there is no guarantee
-          that this version of the driver will work with your VMM.
-
           If unsure, say M.
 
+config VIRTIO_PCI_LEGACY
+        bool "Support for legacy virtio draft 0.9.X and older devices"
+        default y
+        depends on VIRTIO_PCI
+        ---help---
+          Virtio PCI Card 0.9.X Draft (circa 2014) and older device support.
+
+          This option enables building a transitional driver, supporting
+          both devices conforming to Virtio 1 specification, and legacy devices.
+          If disabled, you get a slightly smaller, non-transitional driver,
+          with no legacy compatibility.
+
+          So look out into your driveway.  Do you have a flying car?  If
+          so, you can happily disable this option and virtio will not
+          break.  Otherwise, leave it set.  Unless you're testing what
+          life will be like in The Future.
+
+          If unsure, say Y.
+
 config VIRTIO_BALLOON
         tristate "Virtio balloon driver"
         depends on VIRTIO
diff --git a/drivers/virtio/Makefile b/drivers/virtio/Makefile
index bf5104b56894..d85565b8ea46 100644
--- a/drivers/virtio/Makefile
+++ b/drivers/virtio/Makefile
@@ -1,5 +1,6 @@
 obj-$(CONFIG_VIRTIO) += virtio.o virtio_ring.o
 obj-$(CONFIG_VIRTIO_MMIO) += virtio_mmio.o
 obj-$(CONFIG_VIRTIO_PCI) += virtio_pci.o
-virtio_pci-y := virtio_pci_legacy.o virtio_pci_common.o
+virtio_pci-y := virtio_pci_modern.o virtio_pci_common.o
+virtio_pci-$(CONFIG_VIRTIO_PCI_LEGACY) += virtio_pci_legacy.o
 obj-$(CONFIG_VIRTIO_BALLOON) += virtio_balloon.o
diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
index b9f70dfc4751..5ce2aa48fc6e 100644
--- a/drivers/virtio/virtio.c
+++ b/drivers/virtio/virtio.c
@@ -236,7 +236,10 @@ static int virtio_dev_probe(struct device *_d)
         if (err)
                 goto err;
 
-        add_status(dev, VIRTIO_CONFIG_S_DRIVER_OK);
+        /* If probe didn't do it, mark device DRIVER_OK ourselves. */
+        if (!(dev->config->get_status(dev) & VIRTIO_CONFIG_S_DRIVER_OK))
+                virtio_device_ready(dev);
+
         if (drv->scan)
                 drv->scan(dev);
 
diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index 50c5f42d7a9f..0413157f3b49 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -44,8 +44,7 @@ static int oom_pages = OOM_VBALLOON_DEFAULT_PAGES;
 module_param(oom_pages, int, S_IRUSR | S_IWUSR);
 MODULE_PARM_DESC(oom_pages, "pages to free on OOM");
 
-struct virtio_balloon
-{
+struct virtio_balloon {
         struct virtio_device *vdev;
         struct virtqueue *inflate_vq, *deflate_vq, *stats_vq;
 
@@ -466,6 +465,12 @@ static int virtballoon_probe(struct virtio_device *vdev)
         struct virtio_balloon *vb;
         int err;
 
+        if (!vdev->config->get) {
+                dev_err(&vdev->dev, "%s failure: config access disabled\n",
+                        __func__);
+                return -EINVAL;
+        }
+
         vdev->priv = vb = kmalloc(sizeof(*vb), GFP_KERNEL);
         if (!vb) {
                 err = -ENOMEM;
diff --git a/drivers/virtio/virtio_mmio.c b/drivers/virtio/virtio_mmio.c
index 00d115b22bd8..cad569890908 100644
--- a/drivers/virtio/virtio_mmio.c
+++ b/drivers/virtio/virtio_mmio.c
@@ -1,7 +1,7 @@
 /*
  * Virtio memory mapped device driver
  *
- * Copyright 2011, ARM Ltd.
+ * Copyright 2011-2014, ARM Ltd.
  *
  * This module allows virtio devices to be used over a virtual, memory mapped
  * platform device.
@@ -50,36 +50,6 @@
  *
  *
  *
- * Registers layout (all 32-bit wide):
- *
- * offset d. name             description
- * ------ -- ---------------- -----------------
- *
- * 0x000  R  MagicValue       Magic value "virt"
- * 0x004  R  Version          Device version (current max. 1)
- * 0x008  R  DeviceID         Virtio device ID
- * 0x00c  R  VendorID         Virtio vendor ID
- *
- * 0x010  R  HostFeatures     Features supported by the host
- * 0x014  W  HostFeaturesSel  Set of host features to access via HostFeatures
- *
- * 0x020  W  GuestFeatures    Features activated by the guest
- * 0x024  W  GuestFeaturesSel Set of activated features to set via GuestFeatures
- * 0x028  W  GuestPageSize    Size of guest's memory page in bytes
- *
- * 0x030  W  QueueSel         Queue selector
- * 0x034  R  QueueNumMax      Maximum size of the currently selected queue
- * 0x038  W  QueueNum         Queue size for the currently selected queue
- * 0x03c  W  QueueAlign       Used Ring alignment for the current queue
- * 0x040  RW QueuePFN         PFN for the currently selected queue
- *
- * 0x050  W  QueueNotify      Queue notifier
- * 0x060  R  InterruptStatus  Interrupt status register
- * 0x064  W  InterruptACK     Interrupt acknowledge register
- * 0x070  RW Status           Device status register
- *
- * 0x100+ RW                  Device-specific configuration space
- *
  * Based on Virtio PCI driver by Anthony Liguori, copyright IBM Corp. 2007
  *
  * This work is licensed under the terms of the GNU GPL, version 2 or later.
@@ -145,11 +115,16 @@ struct virtio_mmio_vq_info {
 static u64 vm_get_features(struct virtio_device *vdev)
 {
         struct virtio_mmio_device *vm_dev = to_virtio_mmio_device(vdev);
+        u64 features;
+
+        writel(1, vm_dev->base + VIRTIO_MMIO_DEVICE_FEATURES_SEL);
+        features = readl(vm_dev->base + VIRTIO_MMIO_DEVICE_FEATURES);
+        features <<= 32;
 
-        /* TODO: Features > 32 bits */
-        writel(0, vm_dev->base + VIRTIO_MMIO_HOST_FEATURES_SEL);
+        writel(0, vm_dev->base + VIRTIO_MMIO_DEVICE_FEATURES_SEL);
+        features |= readl(vm_dev->base + VIRTIO_MMIO_DEVICE_FEATURES);
 
-        return readl(vm_dev->base + VIRTIO_MMIO_HOST_FEATURES);
+        return features;
 }
 
 static int vm_finalize_features(struct virtio_device *vdev)
@@ -159,11 +134,20 @@ static int vm_finalize_features(struct virtio_device *vdev)
         /* Give virtio_ring a chance to accept features. */
         vring_transport_features(vdev);
 
-        /* Make sure we don't have any features > 32 bits! */
-        BUG_ON((u32)vdev->features != vdev->features);
+        /* Make sure there is are no mixed devices */
+        if (vm_dev->version == 2 &&
+                        !__virtio_test_bit(vdev, VIRTIO_F_VERSION_1)) {
+                dev_err(&vdev->dev, "New virtio-mmio devices (version 2) must provide VIRTIO_F_VERSION_1 feature!\n");
+                return -EINVAL;
+        }
+
+        writel(1, vm_dev->base + VIRTIO_MMIO_DRIVER_FEATURES_SEL);
+        writel((u32)(vdev->features >> 32),
+                        vm_dev->base + VIRTIO_MMIO_DRIVER_FEATURES);
 
-        writel(0, vm_dev->base + VIRTIO_MMIO_GUEST_FEATURES_SEL);
-        writel(vdev->features, vm_dev->base + VIRTIO_MMIO_GUEST_FEATURES);
+        writel(0, vm_dev->base + VIRTIO_MMIO_DRIVER_FEATURES_SEL);
+        writel((u32)vdev->features,
+                        vm_dev->base + VIRTIO_MMIO_DRIVER_FEATURES);
 
         return 0;
 }
@@ -275,7 +259,12 @@ static void vm_del_vq(struct virtqueue *vq)
 
         /* Select and deactivate the queue */
         writel(index, vm_dev->base + VIRTIO_MMIO_QUEUE_SEL);
-        writel(0, vm_dev->base + VIRTIO_MMIO_QUEUE_PFN);
+        if (vm_dev->version == 1) {
+                writel(0, vm_dev->base + VIRTIO_MMIO_QUEUE_PFN);
+        } else {
+                writel(0, vm_dev->base + VIRTIO_MMIO_QUEUE_READY);
+                WARN_ON(readl(vm_dev->base + VIRTIO_MMIO_QUEUE_READY));
+        }
 
         size = PAGE_ALIGN(vring_size(info->num, VIRTIO_MMIO_VRING_ALIGN));
         free_pages_exact(info->queue, size);
@@ -312,7 +301,8 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned index,
         writel(index, vm_dev->base + VIRTIO_MMIO_QUEUE_SEL);
 
         /* Queue shouldn't already be set up. */
-        if (readl(vm_dev->base + VIRTIO_MMIO_QUEUE_PFN)) {
+        if (readl(vm_dev->base + (vm_dev->version == 1 ?
+                        VIRTIO_MMIO_QUEUE_PFN : VIRTIO_MMIO_QUEUE_READY))) {
                 err = -ENOENT;
                 goto error_available;
         }
@@ -356,13 +346,6 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned index,
                 info->num /= 2;
         }
 
-        /* Activate the queue */
-        writel(info->num, vm_dev->base + VIRTIO_MMIO_QUEUE_NUM);
-        writel(VIRTIO_MMIO_VRING_ALIGN,
-                        vm_dev->base + VIRTIO_MMIO_QUEUE_ALIGN);
-        writel(virt_to_phys(info->queue) >> PAGE_SHIFT,
-                        vm_dev->base + VIRTIO_MMIO_QUEUE_PFN);
-
         /* Create the vring */
         vq = vring_new_virtqueue(index, info->num, VIRTIO_MMIO_VRING_ALIGN, vdev,
                                  true, info->queue, vm_notify, callback, name);
@@ -371,6 +354,33 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned index,
                 goto error_new_virtqueue;
         }
 
+        /* Activate the queue */
+        writel(info->num, vm_dev->base + VIRTIO_MMIO_QUEUE_NUM);
+        if (vm_dev->version == 1) {
+                writel(PAGE_SIZE, vm_dev->base + VIRTIO_MMIO_QUEUE_ALIGN);
+                writel(virt_to_phys(info->queue) >> PAGE_SHIFT,
+                                vm_dev->base + VIRTIO_MMIO_QUEUE_PFN);
+        } else {
+                u64 addr;
+
+                addr = virt_to_phys(info->queue);
+                writel((u32)addr, vm_dev->base + VIRTIO_MMIO_QUEUE_DESC_LOW);
+                writel((u32)(addr >> 32),
+                                vm_dev->base + VIRTIO_MMIO_QUEUE_DESC_HIGH);
+
+                addr = virt_to_phys(virtqueue_get_avail(vq));
+                writel((u32)addr, vm_dev->base + VIRTIO_MMIO_QUEUE_AVAIL_LOW);
+                writel((u32)(addr >> 32),
+                                vm_dev->base + VIRTIO_MMIO_QUEUE_AVAIL_HIGH);
+
+                addr = virt_to_phys(virtqueue_get_used(vq));
+                writel((u32)addr, vm_dev->base + VIRTIO_MMIO_QUEUE_USED_LOW);
+                writel((u32)(addr >> 32),
+                                vm_dev->base + VIRTIO_MMIO_QUEUE_USED_HIGH);
+
+                writel(1, vm_dev->base + VIRTIO_MMIO_QUEUE_READY);
+        }
+
         vq->priv = info;
         info->vq = vq;
 
@@ -381,7 +391,12 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned index,
         return vq;
 
 error_new_virtqueue:
-        writel(0, vm_dev->base + VIRTIO_MMIO_QUEUE_PFN);
+        if (vm_dev->version == 1) {
+                writel(0, vm_dev->base + VIRTIO_MMIO_QUEUE_PFN);
+        } else {
+                writel(0, vm_dev->base + VIRTIO_MMIO_QUEUE_READY);
+                WARN_ON(readl(vm_dev->base + VIRTIO_MMIO_QUEUE_READY));
+        }
         free_pages_exact(info->queue, size);
 error_alloc_pages:
         kfree(info);
@@ -476,16 +491,32 @@ static int virtio_mmio_probe(struct platform_device *pdev)
 
         /* Check device version */
         vm_dev->version = readl(vm_dev->base + VIRTIO_MMIO_VERSION);
-        if (vm_dev->version != 1) {
+        if (vm_dev->version < 1 || vm_dev->version > 2) {
                 dev_err(&pdev->dev, "Version %ld not supported!\n",
                                 vm_dev->version);
                 return -ENXIO;
         }
 
         vm_dev->vdev.id.device = readl(vm_dev->base + VIRTIO_MMIO_DEVICE_ID);
+        if (vm_dev->vdev.id.device == 0) {
+                /*
+                 * virtio-mmio device with an ID 0 is a (dummy) placeholder
+                 * with no function. End probing now with no error reported.
+                 */
+                return -ENODEV;
+        }
         vm_dev->vdev.id.vendor = readl(vm_dev->base + VIRTIO_MMIO_VENDOR_ID);
 
-        writel(PAGE_SIZE, vm_dev->base + VIRTIO_MMIO_GUEST_PAGE_SIZE);
+        /* Reject legacy-only IDs for version 2 devices */
+        if (vm_dev->version == 2 &&
+                        virtio_device_is_legacy_only(vm_dev->vdev.id)) {
+                dev_err(&pdev->dev, "Version 2 not supported for devices %u!\n",
+                                vm_dev->vdev.id.device);
+                return -ENODEV;
+        }
+
+        if (vm_dev->version == 1)
+                writel(PAGE_SIZE, vm_dev->base + VIRTIO_MMIO_GUEST_PAGE_SIZE);
 
         platform_set_drvdata(pdev, vm_dev);
 
diff --git a/drivers/virtio/virtio_pci_common.c b/drivers/virtio/virtio_pci_common.c
index 9756f21b809e..e894eb278d83 100644
--- a/drivers/virtio/virtio_pci_common.c
+++ b/drivers/virtio/virtio_pci_common.c
@@ -19,6 +19,14 @@
 
 #include "virtio_pci_common.h"
 
+static bool force_legacy = false;
+
+#if IS_ENABLED(CONFIG_VIRTIO_PCI_LEGACY)
+module_param(force_legacy, bool, 0444);
+MODULE_PARM_DESC(force_legacy,
+                 "Force legacy mode for transitional virtio 1 devices");
+#endif
+
 /* wait for pending irq handlers */
 void vp_synchronize_vectors(struct virtio_device *vdev)
 {
@@ -464,15 +472,97 @@ static const struct pci_device_id virtio_pci_id_table[] = {
 
 MODULE_DEVICE_TABLE(pci, virtio_pci_id_table);
 
+static void virtio_pci_release_dev(struct device *_d)
+{
+        struct virtio_device *vdev = dev_to_virtio(_d);
+        struct virtio_pci_device *vp_dev = to_vp_device(vdev);
+
+        /* As struct device is a kobject, it's not safe to
+         * free the memory (including the reference counter itself)
+         * until it's release callback. */
+        kfree(vp_dev);
+}
+
 static int virtio_pci_probe(struct pci_dev *pci_dev,
                             const struct pci_device_id *id)
 {
-        return virtio_pci_legacy_probe(pci_dev, id);
+        struct virtio_pci_device *vp_dev;
+        int rc;
+
+        /* allocate our structure and fill it out */
+        vp_dev = kzalloc(sizeof(struct virtio_pci_device), GFP_KERNEL);
+        if (!vp_dev)
+                return -ENOMEM;
+
+        pci_set_drvdata(pci_dev, vp_dev);
+        vp_dev->vdev.dev.parent = &pci_dev->dev;
+        vp_dev->vdev.dev.release = virtio_pci_release_dev;
+        vp_dev->pci_dev = pci_dev;
+        INIT_LIST_HEAD(&vp_dev->virtqueues);
+        spin_lock_init(&vp_dev->lock);
+
+        /* Disable MSI/MSIX to bring device to a known good state. */
+        pci_msi_off(pci_dev);
+
+        /* enable the device */
+        rc = pci_enable_device(pci_dev);
+        if (rc)
+                goto err_enable_device;
+
+        rc = pci_request_regions(pci_dev, "virtio-pci");
+        if (rc)
+                goto err_request_regions;
+
+        if (force_legacy) {
+                rc = virtio_pci_legacy_probe(vp_dev);
+                /* Also try modern mode if we can't map BAR0 (no IO space). */
+                if (rc == -ENODEV || rc == -ENOMEM)
+                        rc = virtio_pci_modern_probe(vp_dev);
+                if (rc)
+                        goto err_probe;
+        } else {
+                rc = virtio_pci_modern_probe(vp_dev);
+                if (rc == -ENODEV)
+                        rc = virtio_pci_legacy_probe(vp_dev);
+                if (rc)
+                        goto err_probe;
+        }
+
+        pci_set_master(pci_dev);
+
+        rc = register_virtio_device(&vp_dev->vdev);
+        if (rc)
+                goto err_register;
+
+        return 0;
+
+err_register:
+        if (vp_dev->ioaddr)
+             virtio_pci_legacy_remove(vp_dev);
+        else
+             virtio_pci_modern_remove(vp_dev);
+err_probe:
+        pci_release_regions(pci_dev);
+err_request_regions:
+        pci_disable_device(pci_dev);
+err_enable_device:
+        kfree(vp_dev);
+        return rc;
 }
 
 static void virtio_pci_remove(struct pci_dev *pci_dev)
 {
-     virtio_pci_legacy_remove(pci_dev);
+        struct virtio_pci_device *vp_dev = pci_get_drvdata(pci_dev);
+
+        unregister_virtio_device(&vp_dev->vdev);
+
+        if (vp_dev->ioaddr)
+                virtio_pci_legacy_remove(vp_dev);
+        else
+                virtio_pci_modern_remove(vp_dev);
+
+        pci_release_regions(pci_dev);
+        pci_disable_device(pci_dev);
 }
 
 static struct pci_driver virtio_pci_driver = {
diff --git a/drivers/virtio/virtio_pci_common.h b/drivers/virtio/virtio_pci_common.h
index 5a497289b7e9..28ee4e56badf 100644
--- a/drivers/virtio/virtio_pci_common.h
+++ b/drivers/virtio/virtio_pci_common.h
@@ -53,12 +53,32 @@ struct virtio_pci_device {
         struct virtio_device vdev;
         struct pci_dev *pci_dev;
 
+        /* In legacy mode, these two point to within ->legacy. */
+        /* Where to read and clear interrupt */
+        u8 __iomem *isr;
+
+        /* Modern only fields */
+        /* The IO mapping for the PCI config space (non-legacy mode) */
+        struct virtio_pci_common_cfg __iomem *common;
+        /* Device-specific data (non-legacy mode)  */
+        void __iomem *device;
+        /* Base of vq notifications (non-legacy mode). */
+        void __iomem *notify_base;
+
+        /* So we can sanity-check accesses. */
+        size_t notify_len;
+        size_t device_len;
+
+        /* Capability for when we need to map notifications per-vq. */
+        int notify_map_cap;
+
+        /* Multiply queue_notify_off by this value. (non-legacy mode). */
+        u32 notify_offset_multiplier;
+
+        /* Legacy only field */
         /* the IO mapping for the PCI config space */
         void __iomem *ioaddr;
 
-        /* the IO mapping for ISR operation */
-        void __iomem *isr;
-
         /* a list of queues so we can dispatch IRQs */
         spinlock_t lock;
         struct list_head virtqueues;
@@ -127,8 +147,19 @@ const char *vp_bus_name(struct virtio_device *vdev);
  */
 int vp_set_vq_affinity(struct virtqueue *vq, int cpu);
 
-int virtio_pci_legacy_probe(struct pci_dev *pci_dev,
-                            const struct pci_device_id *id);
-void virtio_pci_legacy_remove(struct pci_dev *pci_dev);
+#if IS_ENABLED(CONFIG_VIRTIO_PCI_LEGACY)
+int virtio_pci_legacy_probe(struct virtio_pci_device *);
+void virtio_pci_legacy_remove(struct virtio_pci_device *);
+#else
+static inline int virtio_pci_legacy_probe(struct virtio_pci_device *vp_dev)
+{
+        return -ENODEV;
+}
+static inline void virtio_pci_legacy_remove(struct virtio_pci_device *vp_dev)
+{
+}
+#endif
+int virtio_pci_modern_probe(struct virtio_pci_device *);
+void virtio_pci_modern_remove(struct virtio_pci_device *);
 
 #endif
diff --git a/drivers/virtio/virtio_pci_legacy.c b/drivers/virtio/virtio_pci_legacy.c
index a5486e65e04b..256a5278a515 100644
--- a/drivers/virtio/virtio_pci_legacy.c
+++ b/drivers/virtio/virtio_pci_legacy.c
@@ -211,23 +211,10 @@ static const struct virtio_config_ops virtio_pci_config_ops = {
         .set_vq_affinity = vp_set_vq_affinity,
 };
 
-static void virtio_pci_release_dev(struct device *_d)
-{
-        struct virtio_device *vdev = dev_to_virtio(_d);
-        struct virtio_pci_device *vp_dev = to_vp_device(vdev);
-
-        /* As struct device is a kobject, it's not safe to
-         * free the memory (including the reference counter itself)
-         * until it's release callback. */
-        kfree(vp_dev);
-}
-
 /* the PCI probing function */
-int virtio_pci_legacy_probe(struct pci_dev *pci_dev,
-                            const struct pci_device_id *id)
+int virtio_pci_legacy_probe(struct virtio_pci_device *vp_dev)
 {
-        struct virtio_pci_device *vp_dev;
-        int err;
+        struct pci_dev *pci_dev = vp_dev->pci_dev;
 
         /* We only own devices >= 0x1000 and <= 0x103f: leave the rest. */
         if (pci_dev->device < 0x1000 || pci_dev->device > 0x103f)
@@ -239,41 +226,12 @@ int virtio_pci_legacy_probe(struct pci_dev *pci_dev,
                 return -ENODEV;
         }
 
-        /* allocate our structure and fill it out */
-        vp_dev = kzalloc(sizeof(struct virtio_pci_device), GFP_KERNEL);
-        if (vp_dev == NULL)
-                return -ENOMEM;
-
-        vp_dev->vdev.dev.parent = &pci_dev->dev;
-        vp_dev->vdev.dev.release = virtio_pci_release_dev;
-        vp_dev->vdev.config = &virtio_pci_config_ops;
-        vp_dev->pci_dev = pci_dev;
-        INIT_LIST_HEAD(&vp_dev->virtqueues);
-        spin_lock_init(&vp_dev->lock);
-
-        /* Disable MSI/MSIX to bring device to a known good state. */
-        pci_msi_off(pci_dev);
-
-        /* enable the device */
-        err = pci_enable_device(pci_dev);
-        if (err)
-                goto out;
-
-        err = pci_request_regions(pci_dev, "virtio-pci");
-        if (err)
-                goto out_enable_device;
-
         vp_dev->ioaddr = pci_iomap(pci_dev, 0, 0);
-        if (vp_dev->ioaddr == NULL) {
-                err = -ENOMEM;
-                goto out_req_regions;
-        }
+        if (!vp_dev->ioaddr)
+                return -ENOMEM;
 
         vp_dev->isr = vp_dev->ioaddr + VIRTIO_PCI_ISR;
 
-        pci_set_drvdata(pci_dev, vp_dev);
-        pci_set_master(pci_dev);
-
         /* we use the subsystem vendor/device id as the virtio vendor/device
          * id.  this allows us to use the same PCI vendor/device id for all
          * virtio devices and to identify the particular virtio driver by
@@ -281,36 +239,18 @@ int virtio_pci_legacy_probe(struct pci_dev *pci_dev,
         vp_dev->vdev.id.vendor = pci_dev->subsystem_vendor;
         vp_dev->vdev.id.device = pci_dev->subsystem_device;
 
+        vp_dev->vdev.config = &virtio_pci_config_ops;
+
         vp_dev->config_vector = vp_config_vector;
         vp_dev->setup_vq = setup_vq;
         vp_dev->del_vq = del_vq;
 
-        /* finally register the virtio device */
-        err = register_virtio_device(&vp_dev->vdev);
-        if (err)
-                goto out_set_drvdata;
-
         return 0;
-
-out_set_drvdata:
-        pci_iounmap(pci_dev, vp_dev->ioaddr);
-out_req_regions:
-        pci_release_regions(pci_dev);
-out_enable_device:
-        pci_disable_device(pci_dev);
-out:
-        kfree(vp_dev);
-        return err;
 }
 
-void virtio_pci_legacy_remove(struct pci_dev *pci_dev)
+void virtio_pci_legacy_remove(struct virtio_pci_device *vp_dev)
 {
-        struct virtio_pci_device *vp_dev = pci_get_drvdata(pci_dev);
-
-        unregister_virtio_device(&vp_dev->vdev);
+        struct pci_dev *pci_dev = vp_dev->pci_dev;
 
-        vp_del_vqs(&vp_dev->vdev);
         pci_iounmap(pci_dev, vp_dev->ioaddr);
-        pci_release_regions(pci_dev);
-        pci_disable_device(pci_dev);
 }
diff --git a/drivers/virtio/virtio_pci_modern.c b/drivers/virtio/virtio_pci_modern.c
new file mode 100644
index 000000000000..2aa38e59db2e
--- /dev/null
+++ b/drivers/virtio/virtio_pci_modern.c
@@ -0,0 +1,695 @@
+/*
+ * Virtio PCI driver - modern (virtio 1.0) device support
+ *
+ * This module allows virtio devices to be used over a virtual PCI device.
+ * This can be used with QEMU based VMMs like KVM or Xen.
+ *
+ * Copyright IBM Corp. 2007
+ * Copyright Red Hat, Inc. 2014
+ *
+ * Authors:
+ *  Anthony Liguori  <aliguori@us.ibm.com>
+ *  Rusty Russell <rusty@rustcorp.com.au>
+ *  Michael S. Tsirkin <mst@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#define VIRTIO_PCI_NO_LEGACY
+#include "virtio_pci_common.h"
+
+static void __iomem *map_capability(struct pci_dev *dev, int off,
+                                    size_t minlen,
+                                    u32 align,
+                                    u32 start, u32 size,
+                                    size_t *len)
+{
+        u8 bar;
+        u32 offset, length;
+        void __iomem *p;
+
+        pci_read_config_byte(dev, off + offsetof(struct virtio_pci_cap,
+                                                 bar),
+                             &bar);
+        pci_read_config_dword(dev, off + offsetof(struct virtio_pci_cap, offset),
+                             &offset);
+        pci_read_config_dword(dev, off + offsetof(struct virtio_pci_cap, length),
+                              &length);
+
+        if (length <= start) {
+                dev_err(&dev->dev,
+                        "virtio_pci: bad capability len %u (>%u expected)\n",
+                        length, start);
+                return NULL;
+        }
+
+        if (length - start < minlen) {
+                dev_err(&dev->dev,
+                        "virtio_pci: bad capability len %u (>=%zu expected)\n",
+                        length, minlen);
+                return NULL;
+        }
+
+        length -= start;
+
+        if (start + offset < offset) {
+                dev_err(&dev->dev,
+                        "virtio_pci: map wrap-around %u+%u\n",
+                        start, offset);
+                return NULL;
+        }
+
+        offset += start;
+
+        if (offset & (align - 1)) {
+                dev_err(&dev->dev,
+                        "virtio_pci: offset %u not aligned to %u\n",
+                        offset, align);
+                return NULL;
+        }
+
+        if (length > size)
+                length = size;
+
+        if (len)
+                *len = length;
+
+        if (minlen + offset < minlen ||
+            minlen + offset > pci_resource_len(dev, bar)) {
+                dev_err(&dev->dev,
+                        "virtio_pci: map virtio %zu@%u "
+                        "out of range on bar %i length %lu\n",
+                        minlen, offset,
+                        bar, (unsigned long)pci_resource_len(dev, bar));
+                return NULL;
+        }
+
+        p = pci_iomap_range(dev, bar, offset, length);
+        if (!p)
+                dev_err(&dev->dev,
+                        "virtio_pci: unable to map virtio %u@%u on bar %i\n",
+                        length, offset, bar);
+        return p;
+}
+
+static void iowrite64_twopart(u64 val, __le32 __iomem *lo, __le32 __iomem *hi)
+{
+        iowrite32((u32)val, lo);
+        iowrite32(val >> 32, hi);
+}
+
+/* virtio config->get_features() implementation */
+static u64 vp_get_features(struct virtio_device *vdev)
+{
+        struct virtio_pci_device *vp_dev = to_vp_device(vdev);
+        u64 features;
+
+        iowrite32(0, &vp_dev->common->device_feature_select);
+        features = ioread32(&vp_dev->common->device_feature);
+        iowrite32(1, &vp_dev->common->device_feature_select);
+        features |= ((u64)ioread32(&vp_dev->common->device_feature) << 32);
+
+        return features;
+}
+
+/* virtio config->finalize_features() implementation */
+static int vp_finalize_features(struct virtio_device *vdev)
+{
+        struct virtio_pci_device *vp_dev = to_vp_device(vdev);
+
+        /* Give virtio_ring a chance to accept features. */
+        vring_transport_features(vdev);
+
+        if (!__virtio_test_bit(vdev, VIRTIO_F_VERSION_1)) {
+                dev_err(&vdev->dev, "virtio: device uses modern interface "
+                        "but does not have VIRTIO_F_VERSION_1\n");
+                return -EINVAL;
+        }
+
+        iowrite32(0, &vp_dev->common->guest_feature_select);
+        iowrite32((u32)vdev->features, &vp_dev->common->guest_feature);
+        iowrite32(1, &vp_dev->common->guest_feature_select);
+        iowrite32(vdev->features >> 32, &vp_dev->common->guest_feature);
+
+        return 0;
+}
+
+/* virtio config->get() implementation */
+static void vp_get(struct virtio_device *vdev, unsigned offset,
+                   void *buf, unsigned len)
+{
+        struct virtio_pci_device *vp_dev = to_vp_device(vdev);
+        u8 b;
+        __le16 w;
+        __le32 l;
+
+        BUG_ON(offset + len > vp_dev->device_len);
+
+        switch (len) {
+        case 1:
+                b = ioread8(vp_dev->device + offset);
+                memcpy(buf, &b, sizeof b);
+                break;
+        case 2:
+                w = cpu_to_le16(ioread16(vp_dev->device + offset));
+                memcpy(buf, &w, sizeof w);
+                break;
+        case 4:
+                l = cpu_to_le32(ioread32(vp_dev->device + offset));
+                memcpy(buf, &l, sizeof l);
+                break;
+        case 8:
+                l = cpu_to_le32(ioread32(vp_dev->device + offset));
+                memcpy(buf, &l, sizeof l);
+                l = cpu_to_le32(ioread32(vp_dev->device + offset + sizeof l));
+                memcpy(buf + sizeof l, &l, sizeof l);
+                break;
+        default:
+                BUG();
+        }
+}
+
+/* the config->set() implementation.  it's symmetric to the config->get()
+ * implementation */
+static void vp_set(struct virtio_device *vdev, unsigned offset,
+                   const void *buf, unsigned len)
+{
+        struct virtio_pci_device *vp_dev = to_vp_device(vdev);
+        u8 b;
+        __le16 w;
+        __le32 l;
+
+        BUG_ON(offset + len > vp_dev->device_len);
+
+        switch (len) {
+        case 1:
+                memcpy(&b, buf, sizeof b);
+                iowrite8(b, vp_dev->device + offset);
+                break;
+        case 2:
+                memcpy(&w, buf, sizeof w);
+                iowrite16(le16_to_cpu(w), vp_dev->device + offset);
+                break;
+        case 4:
+                memcpy(&l, buf, sizeof l);
+                iowrite32(le32_to_cpu(l), vp_dev->device + offset);
+                break;
+        case 8:
+                memcpy(&l, buf, sizeof l);
+                iowrite32(le32_to_cpu(l), vp_dev->device + offset);
+                memcpy(&l, buf + sizeof l, sizeof l);
+                iowrite32(le32_to_cpu(l), vp_dev->device + offset + sizeof l);
+                break;
+        default:
+                BUG();
+        }
+}
+
+static u32 vp_generation(struct virtio_device *vdev)
+{
+        struct virtio_pci_device *vp_dev = to_vp_device(vdev);
+        return ioread8(&vp_dev->common->config_generation);
+}
+
+/* config->{get,set}_status() implementations */
+static u8 vp_get_status(struct virtio_device *vdev)
+{
+        struct virtio_pci_device *vp_dev = to_vp_device(vdev);
+        return ioread8(&vp_dev->common->device_status);
+}
+
+static void vp_set_status(struct virtio_device *vdev, u8 status)
+{
+        struct virtio_pci_device *vp_dev = to_vp_device(vdev);
+        /* We should never be setting status to 0. */
+        BUG_ON(status == 0);
+        iowrite8(status, &vp_dev->common->device_status);
+}
+
+static void vp_reset(struct virtio_device *vdev)
+{
+        struct virtio_pci_device *vp_dev = to_vp_device(vdev);
+        /* 0 status means a reset. */
+        iowrite8(0, &vp_dev->common->device_status);
+        /* Flush out the status write, and flush in device writes,
+         * including MSI-X interrupts, if any. */
+        ioread8(&vp_dev->common->device_status);
+        /* Flush pending VQ/configuration callbacks. */
+        vp_synchronize_vectors(vdev);
+}
+
+static u16 vp_config_vector(struct virtio_pci_device *vp_dev, u16 vector)
+{
+        /* Setup the vector used for configuration events */
+        iowrite16(vector, &vp_dev->common->msix_config);
+        /* Verify we had enough resources to assign the vector */
+        /* Will also flush the write out to device */
+        return ioread16(&vp_dev->common->msix_config);
+}
+
+static size_t vring_pci_size(u16 num)
+{
+        /* We only need a cacheline separation. */
+        return PAGE_ALIGN(vring_size(num, SMP_CACHE_BYTES));
+}
+
+static void *alloc_virtqueue_pages(int *num)
+{
+        void *pages;
+
+        /* TODO: allocate each queue chunk individually */
+        for (; *num && vring_pci_size(*num) > PAGE_SIZE; *num /= 2) {
+                pages = alloc_pages_exact(vring_pci_size(*num),
+                                          GFP_KERNEL|__GFP_ZERO|__GFP_NOWARN);
+                if (pages)
+                        return pages;
+        }
+
+        if (!*num)
+                return NULL;
+
+        /* Try to get a single page. You are my only hope! */
+        return alloc_pages_exact(vring_pci_size(*num), GFP_KERNEL|__GFP_ZERO);
+}
+
+static struct virtqueue *setup_vq(struct virtio_pci_device *vp_dev,
+                                  struct virtio_pci_vq_info *info,
+                                  unsigned index,
+                                  void (*callback)(struct virtqueue *vq),
+                                  const char *name,
+                                  u16 msix_vec)
+{
+        struct virtio_pci_common_cfg __iomem *cfg = vp_dev->common;
+        struct virtqueue *vq;
+        u16 num, off;
+        int err;
+
+        if (index >= ioread16(&cfg->num_queues))
+                return ERR_PTR(-ENOENT);
+
+        /* Select the queue we're interested in */
+        iowrite16(index, &cfg->queue_select);
+
+        /* Check if queue is either not available or already active. */
+        num = ioread16(&cfg->queue_size);
+        if (!num || ioread16(&cfg->queue_enable))
+                return ERR_PTR(-ENOENT);
+
+        if (num & (num - 1)) {
+                dev_warn(&vp_dev->pci_dev->dev, "bad queue size %u", num);
+                return ERR_PTR(-EINVAL);
+        }
+
+        /* get offset of notification word for this vq */
+        off = ioread16(&cfg->queue_notify_off);
+
+        info->num = num;
+        info->msix_vector = msix_vec;
+
+        info->queue = alloc_virtqueue_pages(&info->num);
+        if (info->queue == NULL)
+                return ERR_PTR(-ENOMEM);
+
+        /* create the vring */
+        vq = vring_new_virtqueue(index, info->num,
+                                 SMP_CACHE_BYTES, &vp_dev->vdev,
+                                 true, info->queue, vp_notify, callback, name);
+        if (!vq) {
+                err = -ENOMEM;
+                goto err_new_queue;
+        }
+
+        /* activate the queue */
+        iowrite16(num, &cfg->queue_size);
+        iowrite64_twopart(virt_to_phys(info->queue),
+                          &cfg->queue_desc_lo, &cfg->queue_desc_hi);
+        iowrite64_twopart(virt_to_phys(virtqueue_get_avail(vq)),
+                          &cfg->queue_avail_lo, &cfg->queue_avail_hi);
+        iowrite64_twopart(virt_to_phys(virtqueue_get_used(vq)),
+                          &cfg->queue_used_lo, &cfg->queue_used_hi);
+
+        if (vp_dev->notify_base) {
+                /* offset should not wrap */
+                if ((u64)off * vp_dev->notify_offset_multiplier + 2
+                    > vp_dev->notify_len) {
+                        dev_warn(&vp_dev->pci_dev->dev,
+                                 "bad notification offset %u (x %u) "
+                                 "for queue %u > %zd",
+                                 off, vp_dev->notify_offset_multiplier,
+                                 index, vp_dev->notify_len);
+                        err = -EINVAL;
+                        goto err_map_notify;
+                }
+                vq->priv = (void __force *)vp_dev->notify_base +
+                        off * vp_dev->notify_offset_multiplier;
+        } else {
+                vq->priv = (void __force *)map_capability(vp_dev->pci_dev,
+                                          vp_dev->notify_map_cap, 2, 2,
+                                          off * vp_dev->notify_offset_multiplier, 2,
+                                          NULL);
+        }
+
+        if (!vq->priv) {
+                err = -ENOMEM;
+                goto err_map_notify;
+        }
+
+        if (msix_vec != VIRTIO_MSI_NO_VECTOR) {
+                iowrite16(msix_vec, &cfg->queue_msix_vector);
+                msix_vec = ioread16(&cfg->queue_msix_vector);
+                if (msix_vec == VIRTIO_MSI_NO_VECTOR) {
+                        err = -EBUSY;
+                        goto err_assign_vector;
+                }
+        }
+
+        return vq;
+
+err_assign_vector:
+        if (!vp_dev->notify_base)
+                pci_iounmap(vp_dev->pci_dev, (void __iomem __force *)vq->priv);
+err_map_notify:
+        vring_del_virtqueue(vq);
+err_new_queue:
+        free_pages_exact(info->queue, vring_pci_size(info->num));
+        return ERR_PTR(err);
+}
+
+static int vp_modern_find_vqs(struct virtio_device *vdev, unsigned nvqs,
+                              struct virtqueue *vqs[],
+                              vq_callback_t *callbacks[],
+                              const char *names[])
+{
+        struct virtio_pci_device *vp_dev = to_vp_device(vdev);
+        struct virtqueue *vq;
+        int rc = vp_find_vqs(vdev, nvqs, vqs, callbacks, names);
+
+        if (rc)
+                return rc;
+
+        /* Select and activate all queues. Has to be done last: once we do
+         * this, there's no way to go back except reset.
+         */
+        list_for_each_entry(vq, &vdev->vqs, list) {
+                iowrite16(vq->index, &vp_dev->common->queue_select);
+                iowrite16(1, &vp_dev->common->queue_enable);
+        }
+
+        return 0;
+}
+
+static void del_vq(struct virtio_pci_vq_info *info)
+{
+        struct virtqueue *vq = info->vq;
+        struct virtio_pci_device *vp_dev = to_vp_device(vq->vdev);
+
+        iowrite16(vq->index, &vp_dev->common->queue_select);
+
+        if (vp_dev->msix_enabled) {
+                iowrite16(VIRTIO_MSI_NO_VECTOR,
+                          &vp_dev->common->queue_msix_vector);
+                /* Flush the write out to device */
+                ioread16(&vp_dev->common->queue_msix_vector);
+        }
+
+        if (!vp_dev->notify_base)
+                pci_iounmap(vp_dev->pci_dev, (void __force __iomem *)vq->priv);
+
+        vring_del_virtqueue(vq);
+
+        free_pages_exact(info->queue, vring_pci_size(info->num));
+}
+
+static const struct virtio_config_ops virtio_pci_config_nodev_ops = {
+        .get            = NULL,
+        .set            = NULL,
+        .generation     = vp_generation,
+        .get_status     = vp_get_status,
+        .set_status     = vp_set_status,
+        .reset          = vp_reset,
+        .find_vqs       = vp_modern_find_vqs,
+        .del_vqs        = vp_del_vqs,
+        .get_features   = vp_get_features,
+        .finalize_features = vp_finalize_features,
+        .bus_name       = vp_bus_name,
+        .set_vq_affinity = vp_set_vq_affinity,
+};
+
+static const struct virtio_config_ops virtio_pci_config_ops = {
+        .get            = vp_get,
+        .set            = vp_set,
+        .generation     = vp_generation,
+        .get_status     = vp_get_status,
+        .set_status     = vp_set_status,
+        .reset          = vp_reset,
+        .find_vqs       = vp_modern_find_vqs,
+        .del_vqs        = vp_del_vqs,
+        .get_features   = vp_get_features,
+        .finalize_features = vp_finalize_features,
+        .bus_name       = vp_bus_name,
+        .set_vq_affinity = vp_set_vq_affinity,
+};
+
+/**
+ * virtio_pci_find_capability - walk capabilities to find device info.
+ * @dev: the pci device
+ * @cfg_type: the VIRTIO_PCI_CAP_* value we seek
+ * @ioresource_types: IORESOURCE_MEM and/or IORESOURCE_IO.
+ *
+ * Returns offset of the capability, or 0.
+ */
+static inline int virtio_pci_find_capability(struct pci_dev *dev, u8 cfg_type,
+                                             u32 ioresource_types)
+{
+        int pos;
+
+        for (pos = pci_find_capability(dev, PCI_CAP_ID_VNDR);
+             pos > 0;
+             pos = pci_find_next_capability(dev, pos, PCI_CAP_ID_VNDR)) {
+                u8 type, bar;
+                pci_read_config_byte(dev, pos + offsetof(struct virtio_pci_cap,
+                                                         cfg_type),
+                                     &type);
+                pci_read_config_byte(dev, pos + offsetof(struct virtio_pci_cap,
+                                                         bar),
+                                     &bar);
+
+                /* Ignore structures with reserved BAR values */
+                if (bar > 0x5)
+                        continue;
+
+                if (type == cfg_type) {
+                        if (pci_resource_len(dev, bar) &&
+                            pci_resource_flags(dev, bar) & ioresource_types)
+                                return pos;
+                }
+        }
+        return 0;
+}
+
+/* This is part of the ABI.  Don't screw with it. */
+static inline void check_offsets(void)
+{
+        /* Note: disk space was harmed in compilation of this function. */
+        BUILD_BUG_ON(VIRTIO_PCI_CAP_VNDR !=
+                     offsetof(struct virtio_pci_cap, cap_vndr));
+        BUILD_BUG_ON(VIRTIO_PCI_CAP_NEXT !=
+                     offsetof(struct virtio_pci_cap, cap_next));
+        BUILD_BUG_ON(VIRTIO_PCI_CAP_LEN !=
+                     offsetof(struct virtio_pci_cap, cap_len));
+        BUILD_BUG_ON(VIRTIO_PCI_CAP_CFG_TYPE !=
+                     offsetof(struct virtio_pci_cap, cfg_type));
+        BUILD_BUG_ON(VIRTIO_PCI_CAP_BAR !=
+                     offsetof(struct virtio_pci_cap, bar));
+        BUILD_BUG_ON(VIRTIO_PCI_CAP_OFFSET !=
+                     offsetof(struct virtio_pci_cap, offset));
+        BUILD_BUG_ON(VIRTIO_PCI_CAP_LENGTH !=
+                     offsetof(struct virtio_pci_cap, length));
+        BUILD_BUG_ON(VIRTIO_PCI_NOTIFY_CAP_MULT !=
+                     offsetof(struct virtio_pci_notify_cap,
+                              notify_off_multiplier));
+        BUILD_BUG_ON(VIRTIO_PCI_COMMON_DFSELECT !=
+                     offsetof(struct virtio_pci_common_cfg,
+                              device_feature_select));
+        BUILD_BUG_ON(VIRTIO_PCI_COMMON_DF !=
+                     offsetof(struct virtio_pci_common_cfg, device_feature));
+        BUILD_BUG_ON(VIRTIO_PCI_COMMON_GFSELECT !=
+                     offsetof(struct virtio_pci_common_cfg,
+                              guest_feature_select));
+        BUILD_BUG_ON(VIRTIO_PCI_COMMON_GF !=
+                     offsetof(struct virtio_pci_common_cfg, guest_feature));
+        BUILD_BUG_ON(VIRTIO_PCI_COMMON_MSIX !=
+                     offsetof(struct virtio_pci_common_cfg, msix_config));
+        BUILD_BUG_ON(VIRTIO_PCI_COMMON_NUMQ !=
+                     offsetof(struct virtio_pci_common_cfg, num_queues));
+        BUILD_BUG_ON(VIRTIO_PCI_COMMON_STATUS !=
+                     offsetof(struct virtio_pci_common_cfg, device_status));
+        BUILD_BUG_ON(VIRTIO_PCI_COMMON_CFGGENERATION !=
+                     offsetof(struct virtio_pci_common_cfg, config_generation));
+        BUILD_BUG_ON(VIRTIO_PCI_COMMON_Q_SELECT !=
+                     offsetof(struct virtio_pci_common_cfg, queue_select));
+        BUILD_BUG_ON(VIRTIO_PCI_COMMON_Q_SIZE !=
+                     offsetof(struct virtio_pci_common_cfg, queue_size));
+        BUILD_BUG_ON(VIRTIO_PCI_COMMON_Q_MSIX !=
+                     offsetof(struct virtio_pci_common_cfg, queue_msix_vector));
+        BUILD_BUG_ON(VIRTIO_PCI_COMMON_Q_ENABLE !=
+                     offsetof(struct virtio_pci_common_cfg, queue_enable));
+        BUILD_BUG_ON(VIRTIO_PCI_COMMON_Q_NOFF !=
+                     offsetof(struct virtio_pci_common_cfg, queue_notify_off));
+        BUILD_BUG_ON(VIRTIO_PCI_COMMON_Q_DESCLO !=
+                     offsetof(struct virtio_pci_common_cfg, queue_desc_lo));
+        BUILD_BUG_ON(VIRTIO_PCI_COMMON_Q_DESCHI !=
+                     offsetof(struct virtio_pci_common_cfg, queue_desc_hi));
+        BUILD_BUG_ON(VIRTIO_PCI_COMMON_Q_AVAILLO !=
+                     offsetof(struct virtio_pci_common_cfg, queue_avail_lo));
+        BUILD_BUG_ON(VIRTIO_PCI_COMMON_Q_AVAILHI !=
+                     offsetof(struct virtio_pci_common_cfg, queue_avail_hi));
+        BUILD_BUG_ON(VIRTIO_PCI_COMMON_Q_USEDLO !=
+                     offsetof(struct virtio_pci_common_cfg, queue_used_lo));
+        BUILD_BUG_ON(VIRTIO_PCI_COMMON_Q_USEDHI !=
+                     offsetof(struct virtio_pci_common_cfg, queue_used_hi));
+}
+
+/* the PCI probing function */
+int virtio_pci_modern_probe(struct virtio_pci_device *vp_dev)
+{
+        struct pci_dev *pci_dev = vp_dev->pci_dev;
+        int err, common, isr, notify, device;
+        u32 notify_length;
+        u32 notify_offset;
+
+        check_offsets();
+
+        /* We only own devices >= 0x1000 and <= 0x107f: leave the rest. */
+        if (pci_dev->device < 0x1000 || pci_dev->device > 0x107f)
+                return -ENODEV;
+
+        if (pci_dev->device < 0x1040) {
+                /* Transitional devices: use the PCI subsystem device id as
+                 * virtio device id, same as legacy driver always did.
+                 */
+                vp_dev->vdev.id.device = pci_dev->subsystem_device;
+        } else {
+                /* Modern devices: simply use PCI device id, but start from 0x1040. */
+                vp_dev->vdev.id.device = pci_dev->device - 0x1040;
+        }
+        vp_dev->vdev.id.vendor = pci_dev->subsystem_vendor;
+
+        if (virtio_device_is_legacy_only(vp_dev->vdev.id))
+                return -ENODEV;
+
+        /* check for a common config: if not, use legacy mode (bar 0). */
+        common = virtio_pci_find_capability(pci_dev, VIRTIO_PCI_CAP_COMMON_CFG,
+                                            IORESOURCE_IO | IORESOURCE_MEM);
+        if (!common) {
+                dev_info(&pci_dev->dev,
+                         "virtio_pci: leaving for legacy driver\n");
+                return -ENODEV;
+        }
+
+        /* If common is there, these should be too... */
+        isr = virtio_pci_find_capability(pci_dev, VIRTIO_PCI_CAP_ISR_CFG,
+                                         IORESOURCE_IO | IORESOURCE_MEM);
+        notify = virtio_pci_find_capability(pci_dev, VIRTIO_PCI_CAP_NOTIFY_CFG,
+                                            IORESOURCE_IO | IORESOURCE_MEM);
+        if (!isr || !notify) {
+                dev_err(&pci_dev->dev,
+                        "virtio_pci: missing capabilities %i/%i/%i\n",
+                        common, isr, notify);
+                return -EINVAL;
+        }
+
+        /* Device capability is only mandatory for devices that have
+         * device-specific configuration.
+         */
+        device = virtio_pci_find_capability(pci_dev, VIRTIO_PCI_CAP_DEVICE_CFG,
+                                            IORESOURCE_IO | IORESOURCE_MEM);
+
+        err = -EINVAL;
+        vp_dev->common = map_capability(pci_dev, common,
+                                        sizeof(struct virtio_pci_common_cfg), 4,
+                                        0, sizeof(struct virtio_pci_common_cfg),
+                                        NULL);
+        if (!vp_dev->common)
+                goto err_map_common;
+        vp_dev->isr = map_capability(pci_dev, isr, sizeof(u8), 1,
+                                     0, 1,
+                                     NULL);
+        if (!vp_dev->isr)
+                goto err_map_isr;
+
+        /* Read notify_off_multiplier from config space. */
+        pci_read_config_dword(pci_dev,
+                              notify + offsetof(struct virtio_pci_notify_cap,
+                                                notify_off_multiplier),
+                              &vp_dev->notify_offset_multiplier);
+        /* Read notify length and offset from config space. */
+        pci_read_config_dword(pci_dev,
+                              notify + offsetof(struct virtio_pci_notify_cap,
+                                                cap.length),
+                              &notify_length);
+
+        pci_read_config_dword(pci_dev,
+                              notify + offsetof(struct virtio_pci_notify_cap,
+                                                cap.length),
+                              &notify_offset);
+
+        /* We don't know how many VQs we'll map, ahead of the time.
+         * If notify length is small, map it all now.
+         * Otherwise, map each VQ individually later.
+         */
+        if ((u64)notify_length + (notify_offset % PAGE_SIZE) <= PAGE_SIZE) {
+                vp_dev->notify_base = map_capability(pci_dev, notify, 2, 2,
+                                                     0, notify_length,
+                                                     &vp_dev->notify_len);
+                if (!vp_dev->notify_base)
+                        goto err_map_notify;
+        } else {
+                vp_dev->notify_map_cap = notify;
+        }
+
+        /* Again, we don't know how much we should map, but PAGE_SIZE
+         * is more than enough for all existing devices.
+         */
+        if (device) {
+                vp_dev->device = map_capability(pci_dev, device, 0, 4,
+                                                0, PAGE_SIZE,
+                                                &vp_dev->device_len);
+                if (!vp_dev->device)
+                        goto err_map_device;
+
+                vp_dev->vdev.config = &virtio_pci_config_ops;
+        } else {
+                vp_dev->vdev.config = &virtio_pci_config_nodev_ops;
+        }
+
+        vp_dev->config_vector = vp_config_vector;
+        vp_dev->setup_vq = setup_vq;
+        vp_dev->del_vq = del_vq;
+
+        return 0;
+
+err_map_device:
+        if (vp_dev->notify_base)
+                pci_iounmap(pci_dev, vp_dev->notify_base);
+err_map_notify:
+        pci_iounmap(pci_dev, vp_dev->isr);
+err_map_isr:
+        pci_iounmap(pci_dev, vp_dev->common);
+err_map_common:
+        return err;
+}
+
+void virtio_pci_modern_remove(struct virtio_pci_device *vp_dev)
+{
+        struct pci_dev *pci_dev = vp_dev->pci_dev;
+
+        if (vp_dev->device)
+                pci_iounmap(pci_dev, vp_dev->device);
+        if (vp_dev->notify_base)
+                pci_iounmap(pci_dev, vp_dev->notify_base);
+        pci_iounmap(pci_dev, vp_dev->isr);
+        pci_iounmap(pci_dev, vp_dev->common);
+}
diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
index 00ec6b3f96b2..096b857e7b75 100644
--- a/drivers/virtio/virtio_ring.c
+++ b/drivers/virtio/virtio_ring.c
@@ -54,8 +54,7 @@
 #define END_USE(vq)
 #endif
 
-struct vring_virtqueue
-{
+struct vring_virtqueue {
         struct virtqueue vq;
 
         /* Actual memory layout for this queue */
@@ -245,14 +244,14 @@ static inline int virtqueue_add(struct virtqueue *_vq,
         vq->vring.avail->idx = cpu_to_virtio16(_vq->vdev, virtio16_to_cpu(_vq->vdev, vq->vring.avail->idx) + 1);
         vq->num_added++;
 
+        pr_debug("Added buffer head %i to %p\n", head, vq);
+        END_USE(vq);
+
         /* This is very unlikely, but theoretically possible.  Kick
          * just in case. */
         if (unlikely(vq->num_added == (1 << 16) - 1))
                 virtqueue_kick(_vq);
 
-        pr_debug("Added buffer head %i to %p\n", head, vq);
-        END_USE(vq);
-
         return 0;
 }
 
diff --git a/include/asm-generic/pci_iomap.h b/include/asm-generic/pci_iomap.h
index ce37349860fe..7389c87116a0 100644
--- a/include/asm-generic/pci_iomap.h
+++ b/include/asm-generic/pci_iomap.h
@@ -15,6 +15,9 @@ struct pci_dev;
 #ifdef CONFIG_PCI
 /* Create a virtual mapping cookie for a PCI BAR (memory or IO) */
 extern void __iomem *pci_iomap(struct pci_dev *dev, int bar, unsigned long max);
+extern void __iomem *pci_iomap_range(struct pci_dev *dev, int bar,
+                                     unsigned long offset,
+                                     unsigned long maxlen);
 /* Create a virtual mapping cookie for a port on a given PCI device.
  * Do not call this directly, it exists to make it easier for architectures
  * to override */
@@ -30,6 +33,13 @@ static inline void __iomem *pci_iomap(struct pci_dev *dev, int bar, unsigned lon
 {
         return NULL;
 }
+
+static inline void __iomem *pci_iomap_range(struct pci_dev *dev, int bar,
+                                            unsigned long offset,
+                                            unsigned long maxlen)
+{
+        return NULL;
+}
 #endif
 
 #endif /* __ASM_GENERIC_IO_H */
diff --git a/include/linux/lguest_launcher.h b/include/linux/lguest_launcher.h
index 495203ff221c..acd5b12565cc 100644
--- a/include/linux/lguest_launcher.h
+++ b/include/linux/lguest_launcher.h
@@ -8,52 +8,13 @@
  *
  * The Guest needs devices to do anything useful.  Since we don't let it touch
  * real devices (think of the damage it could do!) we provide virtual devices.
- * We could emulate a PCI bus with various devices on it, but that is a fairly
- * complex burden for the Host and suboptimal for the Guest, so we have our own
- * simple lguest bus and we use "virtio" drivers.  These drivers need a set of
- * routines from us which will actually do the virtual I/O, but they handle all
- * the net/block/console stuff themselves.  This means that if we want to add
- * a new device, we simply need to write a new virtio driver and create support
- * for it in the Launcher: this code won't need to change.
+ * We emulate a PCI bus with virtio devices on it; we used to have our own
+ * lguest bus which was far simpler, but this tests the virtio 1.0 standard.
  *
  * Virtio devices are also used by kvm, so we can simply reuse their optimized
  * device drivers.  And one day when everyone uses virtio, my plan will be
  * complete.  Bwahahahah!
- *
- * Devices are described by a simplified ID, a status byte, and some "config"
- * bytes which describe this device's configuration.  This is placed by the
- * Launcher just above the top of physical memory:
- */
-struct lguest_device_desc {
-        /* The device type: console, network, disk etc.  Type 0 terminates. */
-        __u8 type;
-        /* The number of virtqueues (first in config array) */
-        __u8 num_vq;
-        /*
-         * The number of bytes of feature bits.  Multiply by 2: one for host
-         * features and one for Guest acknowledgements.
-         */
-        __u8 feature_len;
-        /* The number of bytes of the config array after virtqueues. */
-        __u8 config_len;
-        /* A status byte, written by the Guest. */
-        __u8 status;
-        __u8 config[0];
-};
-
-/*D:135
- * This is how we expect the device configuration field for a virtqueue
- * to be laid out in config space.
  */
-struct lguest_vqconfig {
-        /* The number of entries in the virtio_ring */
-        __u16 num;
-        /* The interrupt we get when something happens. */
-        __u16 irq;
-        /* The page number of the virtio ring for this device. */
-        __u32 pfn;
-};
-/*:*/
 
 /* Write command first word is a request. */
 enum lguest_req
@@ -62,12 +23,22 @@ enum lguest_req
         LHREQ_GETDMA, /* No longer used */
         LHREQ_IRQ, /* + irq */
         LHREQ_BREAK, /* No longer used */
-        LHREQ_EVENTFD, /* + address, fd. */
+        LHREQ_EVENTFD, /* No longer used. */
+        LHREQ_GETREG, /* + offset within struct pt_regs (then read value). */
+        LHREQ_SETREG, /* + offset within struct pt_regs, value. */
+        LHREQ_TRAP, /* + trap number to deliver to guest. */
 };
 
 /*
- * The alignment to use between consumer and producer parts of vring.
- * x86 pagesize for historical reasons.
+ * This is what read() of the lguest fd populates.  trap ==
+ * LGUEST_TRAP_ENTRY for an LHCALL_NOTIFY (addr is the
+ * argument), 14 for a page fault in the MMIO region (addr is
+ * the trap address, insn is the instruction), or 13 for a GPF
+ * (insn is the instruction).
  */
-#define LGUEST_VRING_ALIGN      4096
+struct lguest_pending {
+        __u8 trap;
+        __u8 insn[7];
+        __u32 addr;
+};
 #endif /* _LINUX_LGUEST_LAUNCHER */
diff --git a/include/linux/virtio_mmio.h b/include/linux/virtio_mmio.h
index 5c7b6f0daef8..c4b09689ab64 100644
--- a/include/linux/virtio_mmio.h
+++ b/include/linux/virtio_mmio.h
@@ -51,23 +51,29 @@
 /* Virtio vendor ID - Read Only */
 #define VIRTIO_MMIO_VENDOR_ID           0x00c
 
-/* Bitmask of the features supported by the host
+/* Bitmask of the features supported by the device (host)
  * (32 bits per set) - Read Only */
-#define VIRTIO_MMIO_HOST_FEATURES       0x010
+#define VIRTIO_MMIO_DEVICE_FEATURES     0x010
 
-/* Host features set selector - Write Only */
-#define VIRTIO_MMIO_HOST_FEATURES_SEL   0x014
+/* Device (host) features set selector - Write Only */
+#define VIRTIO_MMIO_DEVICE_FEATURES_SEL 0x014
 
-/* Bitmask of features activated by the guest
+/* Bitmask of features activated by the driver (guest)
  * (32 bits per set) - Write Only */
-#define VIRTIO_MMIO_GUEST_FEATURES      0x020
+#define VIRTIO_MMIO_DRIVER_FEATURES     0x020
 
 /* Activated features set selector - Write Only */
-#define VIRTIO_MMIO_GUEST_FEATURES_SEL  0x024
+#define VIRTIO_MMIO_DRIVER_FEATURES_SEL 0x024
+
+
+#ifndef VIRTIO_MMIO_NO_LEGACY /* LEGACY DEVICES ONLY! */
 
 /* Guest's memory page size in bytes - Write Only */
 #define VIRTIO_MMIO_GUEST_PAGE_SIZE     0x028
 
+#endif
+
+
 /* Queue selector - Write Only */
 #define VIRTIO_MMIO_QUEUE_SEL           0x030
 
@@ -77,12 +83,21 @@
 /* Queue size for the currently selected queue - Write Only */
 #define VIRTIO_MMIO_QUEUE_NUM           0x038
 
+
+#ifndef VIRTIO_MMIO_NO_LEGACY /* LEGACY DEVICES ONLY! */
+
 /* Used Ring alignment for the currently selected queue - Write Only */
 #define VIRTIO_MMIO_QUEUE_ALIGN         0x03c
 
 /* Guest's PFN for the currently selected queue - Read Write */
 #define VIRTIO_MMIO_QUEUE_PFN           0x040
 
+#endif
+
+
+/* Ready bit for the currently selected queue - Read Write */
+#define VIRTIO_MMIO_QUEUE_READY         0x044
+
 /* Queue notifier - Write Only */
 #define VIRTIO_MMIO_QUEUE_NOTIFY        0x050
 
@@ -95,6 +110,21 @@
 /* Device status register - Read Write */
 #define VIRTIO_MMIO_STATUS              0x070
 
+/* Selected queue's Descriptor Table address, 64 bits in two halves */
+#define VIRTIO_MMIO_QUEUE_DESC_LOW      0x080
+#define VIRTIO_MMIO_QUEUE_DESC_HIGH     0x084
+
+/* Selected queue's Available Ring address, 64 bits in two halves */
+#define VIRTIO_MMIO_QUEUE_AVAIL_LOW     0x090
+#define VIRTIO_MMIO_QUEUE_AVAIL_HIGH    0x094
+
+/* Selected queue's Used Ring address, 64 bits in two halves */
+#define VIRTIO_MMIO_QUEUE_USED_LOW      0x0a0
+#define VIRTIO_MMIO_QUEUE_USED_HIGH     0x0a4
+
+/* Configuration atomicity value */
+#define VIRTIO_MMIO_CONFIG_GENERATION   0x0fc
+
 /* The config space is defined by each driver as
  * the per-driver configuration space - Read Write */
 #define VIRTIO_MMIO_CONFIG              0x100
diff --git a/include/uapi/linux/virtio_balloon.h b/include/uapi/linux/virtio_balloon.h
index be40f7059e93..4b0488f20b2e 100644
--- a/include/uapi/linux/virtio_balloon.h
+++ b/include/uapi/linux/virtio_balloon.h
@@ -36,8 +36,7 @@
 /* Size of a PFN in the balloon interface. */
 #define VIRTIO_BALLOON_PFN_SHIFT 12
 
-struct virtio_balloon_config
-{
+struct virtio_balloon_config {
         /* Number of pages host wants Guest to give up. */
         __le32 num_pages;
         /* Number of pages we've actually got in balloon. */
diff --git a/include/uapi/linux/virtio_blk.h b/include/uapi/linux/virtio_blk.h
index 247c8ba8544a..3c53eec4ae22 100644
--- a/include/uapi/linux/virtio_blk.h
+++ b/include/uapi/linux/virtio_blk.h
@@ -31,22 +31,25 @@
 #include <linux/virtio_types.h>
 
 /* Feature bits */
-#define VIRTIO_BLK_F_BARRIER    0       /* Does host support barriers? */
 #define VIRTIO_BLK_F_SIZE_MAX   1       /* Indicates maximum segment size */
 #define VIRTIO_BLK_F_SEG_MAX    2       /* Indicates maximum # of segments */
 #define VIRTIO_BLK_F_GEOMETRY   4       /* Legacy geometry available  */
 #define VIRTIO_BLK_F_RO         5       /* Disk is read-only */
 #define VIRTIO_BLK_F_BLK_SIZE   6       /* Block size of disk is available*/
-#define VIRTIO_BLK_F_SCSI       7       /* Supports scsi command passthru */
-#define VIRTIO_BLK_F_WCE        9       /* Writeback mode enabled after reset */
 #define VIRTIO_BLK_F_TOPOLOGY   10      /* Topology information is available */
-#define VIRTIO_BLK_F_CONFIG_WCE 11      /* Writeback mode available in config */
 #define VIRTIO_BLK_F_MQ         12      /* support more than one vq */
 
+/* Legacy feature bits */
+#ifndef VIRTIO_BLK_NO_LEGACY
+#define VIRTIO_BLK_F_BARRIER    0       /* Does host support barriers? */
+#define VIRTIO_BLK_F_SCSI       7       /* Supports scsi command passthru */
+#define VIRTIO_BLK_F_WCE        9       /* Writeback mode enabled after reset */
+#define VIRTIO_BLK_F_CONFIG_WCE 11      /* Writeback mode available in config */
 #ifndef __KERNEL__
 /* Old (deprecated) name for VIRTIO_BLK_F_WCE. */
 #define VIRTIO_BLK_F_FLUSH VIRTIO_BLK_F_WCE
 #endif
+#endif /* !VIRTIO_BLK_NO_LEGACY */
 
 #define VIRTIO_BLK_ID_BYTES     20      /* ID string length */
 
@@ -100,8 +103,10 @@ struct virtio_blk_config {
 #define VIRTIO_BLK_T_IN         0
 #define VIRTIO_BLK_T_OUT        1
 
+#ifndef VIRTIO_BLK_NO_LEGACY
 /* This bit says it's a scsi command, not an actual read or write. */
 #define VIRTIO_BLK_T_SCSI_CMD   2
+#endif /* VIRTIO_BLK_NO_LEGACY */
 
 /* Cache flush command */
 #define VIRTIO_BLK_T_FLUSH      4
@@ -109,8 +114,10 @@ struct virtio_blk_config {
 /* Get device ID command */
 #define VIRTIO_BLK_T_GET_ID    8
 
+#ifndef VIRTIO_BLK_NO_LEGACY
 /* Barrier before this op. */
 #define VIRTIO_BLK_T_BARRIER    0x80000000
+#endif /* !VIRTIO_BLK_NO_LEGACY */
 
 /* This is the first element of the read scatter-gather list. */
 struct virtio_blk_outhdr {
@@ -122,12 +129,14 @@ struct virtio_blk_outhdr {
         __virtio64 sector;
 };
 
+#ifndef VIRTIO_BLK_NO_LEGACY
 struct virtio_scsi_inhdr {
         __virtio32 errors;
         __virtio32 data_len;
         __virtio32 sense_len;
         __virtio32 residual;
 };
+#endif /* !VIRTIO_BLK_NO_LEGACY */
 
 /* And this is the final byte of the write scatter-gather list. */
 #define VIRTIO_BLK_S_OK         0
diff --git a/include/uapi/linux/virtio_config.h b/include/uapi/linux/virtio_config.h
index a6d0cdeaacd4..c18264df9504 100644
--- a/include/uapi/linux/virtio_config.h
+++ b/include/uapi/linux/virtio_config.h
@@ -49,12 +49,14 @@
 #define VIRTIO_TRANSPORT_F_START        28
 #define VIRTIO_TRANSPORT_F_END          33
 
+#ifndef VIRTIO_CONFIG_NO_LEGACY
 /* Do we get callbacks when the ring is completely used, even if we've
  * suppressed them? */
 #define VIRTIO_F_NOTIFY_ON_EMPTY        24
 
 /* Can the device handle any descriptor layout? */
 #define VIRTIO_F_ANY_LAYOUT             27
+#endif /* VIRTIO_CONFIG_NO_LEGACY */
 
 /* v1.0 compliant. */
 #define VIRTIO_F_VERSION_1              32
diff --git a/include/uapi/linux/virtio_net.h b/include/uapi/linux/virtio_net.h
index b5f1677b291c..7bbee79ca293 100644
--- a/include/uapi/linux/virtio_net.h
+++ b/include/uapi/linux/virtio_net.h
@@ -35,7 +35,6 @@
 #define VIRTIO_NET_F_CSUM       0       /* Host handles pkts w/ partial csum */
 #define VIRTIO_NET_F_GUEST_CSUM 1       /* Guest handles pkts w/ partial csum */
 #define VIRTIO_NET_F_MAC        5       /* Host has given MAC address. */
-#define VIRTIO_NET_F_GSO        6       /* Host handles pkts w/ any GSO type */
 #define VIRTIO_NET_F_GUEST_TSO4 7       /* Guest can handle TSOv4 in. */
 #define VIRTIO_NET_F_GUEST_TSO6 8       /* Guest can handle TSOv6 in. */
 #define VIRTIO_NET_F_GUEST_ECN  9       /* Guest can handle TSO[6] w/ ECN in. */
@@ -56,6 +55,10 @@
                                          * Steering */
 #define VIRTIO_NET_F_CTRL_MAC_ADDR 23   /* Set MAC address */
 
+#ifndef VIRTIO_NET_NO_LEGACY
+#define VIRTIO_NET_F_GSO        6       /* Host handles pkts w/ any GSO type */
+#endif /* VIRTIO_NET_NO_LEGACY */
+
 #define VIRTIO_NET_S_LINK_UP    1       /* Link is up */
 #define VIRTIO_NET_S_ANNOUNCE   2       /* Announcement is needed */
 
@@ -71,19 +74,39 @@ struct virtio_net_config {
         __u16 max_virtqueue_pairs;
 } __attribute__((packed));
 
+/*
+ * This header comes first in the scatter-gather list.  If you don't
+ * specify GSO or CSUM features, you can simply ignore the header.
+ *
+ * This is bitwise-equivalent to the legacy struct virtio_net_hdr_mrg_rxbuf,
+ * only flattened.
+ */
+struct virtio_net_hdr_v1 {
+#define VIRTIO_NET_HDR_F_NEEDS_CSUM     1       /* Use csum_start, csum_offset */
+#define VIRTIO_NET_HDR_F_DATA_VALID     2       /* Csum is valid */
+        __u8 flags;
+#define VIRTIO_NET_HDR_GSO_NONE         0       /* Not a GSO frame */
+#define VIRTIO_NET_HDR_GSO_TCPV4        1       /* GSO frame, IPv4 TCP (TSO) */
+#define VIRTIO_NET_HDR_GSO_UDP          3       /* GSO frame, IPv4 UDP (UFO) */
+#define VIRTIO_NET_HDR_GSO_TCPV6        4       /* GSO frame, IPv6 TCP */
+#define VIRTIO_NET_HDR_GSO_ECN          0x80    /* TCP has ECN set */
+        __u8 gso_type;
+        __virtio16 hdr_len;     /* Ethernet + IP + tcp/udp hdrs */
+        __virtio16 gso_size;    /* Bytes to append to hdr_len per frame */
+        __virtio16 csum_start;  /* Position to start checksumming from */
+        __virtio16 csum_offset; /* Offset after that to place checksum */
+        __virtio16 num_buffers; /* Number of merged rx buffers */
+};
+
+#ifndef VIRTIO_NET_NO_LEGACY
 /* This header comes first in the scatter-gather list.
- * If VIRTIO_F_ANY_LAYOUT is not negotiated, it must
+ * For legacy virtio, if VIRTIO_F_ANY_LAYOUT is not negotiated, it must
  * be the first element of the scatter-gather list.  If you don't
  * specify GSO or CSUM features, you can simply ignore the header. */
 struct virtio_net_hdr {
-#define VIRTIO_NET_HDR_F_NEEDS_CSUM     1       // Use csum_start, csum_offset
-#define VIRTIO_NET_HDR_F_DATA_VALID     2       // Csum is valid
+        /* See VIRTIO_NET_HDR_F_* */
         __u8 flags;
-#define VIRTIO_NET_HDR_GSO_NONE         0       // Not a GSO frame
-#define VIRTIO_NET_HDR_GSO_TCPV4        1       // GSO frame, IPv4 TCP (TSO)
-#define VIRTIO_NET_HDR_GSO_UDP          3       // GSO frame, IPv4 UDP (UFO)
-#define VIRTIO_NET_HDR_GSO_TCPV6        4       // GSO frame, IPv6 TCP
-#define VIRTIO_NET_HDR_GSO_ECN          0x80    // TCP has ECN set
+        /* See VIRTIO_NET_HDR_GSO_* */
         __u8 gso_type;
         __virtio16 hdr_len;             /* Ethernet + IP + tcp/udp hdrs */
         __virtio16 gso_size;            /* Bytes to append to hdr_len per frame */
@@ -97,6 +120,7 @@ struct virtio_net_hdr_mrg_rxbuf {
         struct virtio_net_hdr hdr;
         __virtio16 num_buffers; /* Number of merged rx buffers */
 };
+#endif /* ...VIRTIO_NET_NO_LEGACY */
 
 /*
  * Control virtqueue data structures
diff --git a/include/uapi/linux/virtio_pci.h b/include/uapi/linux/virtio_pci.h
index 35b552c7f330..75301468359f 100644
--- a/include/uapi/linux/virtio_pci.h
+++ b/include/uapi/linux/virtio_pci.h
@@ -39,7 +39,7 @@
 #ifndef _LINUX_VIRTIO_PCI_H
 #define _LINUX_VIRTIO_PCI_H
 
-#include <linux/virtio_config.h>
+#include <linux/types.h>
 
 #ifndef VIRTIO_PCI_NO_LEGACY
 
@@ -99,4 +99,95 @@
 /* Vector value used to disable MSI for queue */
 #define VIRTIO_MSI_NO_VECTOR            0xffff
 
+#ifndef VIRTIO_PCI_NO_MODERN
+
+/* IDs for different capabilities.  Must all exist. */
+
+/* Common configuration */
+#define VIRTIO_PCI_CAP_COMMON_CFG       1
+/* Notifications */
+#define VIRTIO_PCI_CAP_NOTIFY_CFG       2
+/* ISR access */
+#define VIRTIO_PCI_CAP_ISR_CFG          3
+/* Device specific configuration */
+#define VIRTIO_PCI_CAP_DEVICE_CFG       4
+/* PCI configuration access */
+#define VIRTIO_PCI_CAP_PCI_CFG          5
+
+/* This is the PCI capability header: */
+struct virtio_pci_cap {
+        __u8 cap_vndr;          /* Generic PCI field: PCI_CAP_ID_VNDR */
+        __u8 cap_next;          /* Generic PCI field: next ptr. */
+        __u8 cap_len;           /* Generic PCI field: capability length */
+        __u8 cfg_type;          /* Identifies the structure. */
+        __u8 bar;               /* Where to find it. */
+        __u8 padding[3];        /* Pad to full dword. */
+        __le32 offset;          /* Offset within bar. */
+        __le32 length;          /* Length of the structure, in bytes. */
+};
+
+struct virtio_pci_notify_cap {
+        struct virtio_pci_cap cap;
+        __le32 notify_off_multiplier;   /* Multiplier for queue_notify_off. */
+};
+
+/* Fields in VIRTIO_PCI_CAP_COMMON_CFG: */
+struct virtio_pci_common_cfg {
+        /* About the whole device. */
+        __le32 device_feature_select;   /* read-write */
+        __le32 device_feature;          /* read-only */
+        __le32 guest_feature_select;    /* read-write */
+        __le32 guest_feature;           /* read-write */
+        __le16 msix_config;             /* read-write */
+        __le16 num_queues;              /* read-only */
+        __u8 device_status;             /* read-write */
+        __u8 config_generation;         /* read-only */
+
+        /* About a specific virtqueue. */
+        __le16 queue_select;            /* read-write */
+        __le16 queue_size;              /* read-write, power of 2. */
+        __le16 queue_msix_vector;       /* read-write */
+        __le16 queue_enable;            /* read-write */
+        __le16 queue_notify_off;        /* read-only */
+        __le32 queue_desc_lo;           /* read-write */
+        __le32 queue_desc_hi;           /* read-write */
+        __le32 queue_avail_lo;          /* read-write */
+        __le32 queue_avail_hi;          /* read-write */
+        __le32 queue_used_lo;           /* read-write */
+        __le32 queue_used_hi;           /* read-write */
+};
+
+/* Macro versions of offsets for the Old Timers! */
+#define VIRTIO_PCI_CAP_VNDR             0
+#define VIRTIO_PCI_CAP_NEXT             1
+#define VIRTIO_PCI_CAP_LEN              2
+#define VIRTIO_PCI_CAP_CFG_TYPE         3
+#define VIRTIO_PCI_CAP_BAR              4
+#define VIRTIO_PCI_CAP_OFFSET           8
+#define VIRTIO_PCI_CAP_LENGTH           12
+
+#define VIRTIO_PCI_NOTIFY_CAP_MULT      16
+
+#define VIRTIO_PCI_COMMON_DFSELECT      0
+#define VIRTIO_PCI_COMMON_DF            4
+#define VIRTIO_PCI_COMMON_GFSELECT      8
+#define VIRTIO_PCI_COMMON_GF            12
+#define VIRTIO_PCI_COMMON_MSIX          16
+#define VIRTIO_PCI_COMMON_NUMQ          18
+#define VIRTIO_PCI_COMMON_STATUS        20
+#define VIRTIO_PCI_COMMON_CFGGENERATION 21
+#define VIRTIO_PCI_COMMON_Q_SELECT      22
+#define VIRTIO_PCI_COMMON_Q_SIZE        24
+#define VIRTIO_PCI_COMMON_Q_MSIX        26
+#define VIRTIO_PCI_COMMON_Q_ENABLE      28
+#define VIRTIO_PCI_COMMON_Q_NOFF        30
+#define VIRTIO_PCI_COMMON_Q_DESCLO      32
+#define VIRTIO_PCI_COMMON_Q_DESCHI      36
+#define VIRTIO_PCI_COMMON_Q_AVAILLO     40
+#define VIRTIO_PCI_COMMON_Q_AVAILHI     44
+#define VIRTIO_PCI_COMMON_Q_USEDLO      48
+#define VIRTIO_PCI_COMMON_Q_USEDHI      52
+
+#endif /* VIRTIO_PCI_NO_MODERN */
+
 #endif
diff --git a/lib/pci_iomap.c b/lib/pci_iomap.c
index 0d83ea8a9605..bcce5f149310 100644
--- a/lib/pci_iomap.c
+++ b/lib/pci_iomap.c
@@ -10,10 +10,11 @@
 
 #ifdef CONFIG_PCI
 /**
- * pci_iomap - create a virtual mapping cookie for a PCI BAR
+ * pci_iomap_range - create a virtual mapping cookie for a PCI BAR
  * @dev: PCI device that owns the BAR
  * @bar: BAR number
- * @maxlen: length of the memory to map
+ * @offset: map memory at the given offset in BAR
+ * @maxlen: max length of the memory to map
  *
  * Using this function you will get a __iomem address to your device BAR.
  * You can access it using ioread*() and iowrite*(). These functions hide
@@ -21,16 +22,21 @@
  * you expect from them in the correct way.
  *
  * @maxlen specifies the maximum length to map. If you want to get access to
- * the complete BAR without checking for its length first, pass %0 here.
+ * the complete BAR from offset to the end, pass %0 here.
  * */
-void __iomem *pci_iomap(struct pci_dev *dev, int bar, unsigned long maxlen)
+void __iomem *pci_iomap_range(struct pci_dev *dev,
+                              int bar,
+                              unsigned long offset,
+                              unsigned long maxlen)
 {
         resource_size_t start = pci_resource_start(dev, bar);
         resource_size_t len = pci_resource_len(dev, bar);
         unsigned long flags = pci_resource_flags(dev, bar);
 
-        if (!len || !start)
+        if (len <= offset || !start)
                 return NULL;
+        len -= offset;
+        start += offset;
         if (maxlen && len > maxlen)
                 len = maxlen;
         if (flags & IORESOURCE_IO)
@@ -43,6 +49,25 @@ void __iomem *pci_iomap(struct pci_dev *dev, int bar, unsigned long maxlen)
         /* What? */
         return NULL;
 }
+EXPORT_SYMBOL(pci_iomap_range);
 
+/**
+ * pci_iomap - create a virtual mapping cookie for a PCI BAR
+ * @dev: PCI device that owns the BAR
+ * @bar: BAR number
+ * @maxlen: length of the memory to map
+ *
+ * Using this function you will get a __iomem address to your device BAR.
+ * You can access it using ioread*() and iowrite*(). These functions hide
+ * the details if this is a MMIO or PIO address space and will just do what
+ * you expect from them in the correct way.
+ *
+ * @maxlen specifies the maximum length to map. If you want to get access to
+ * the complete BAR without checking for its length first, pass %0 here.
+ * */
+void __iomem *pci_iomap(struct pci_dev *dev, int bar, unsigned long maxlen)
+{
+        return pci_iomap_range(dev, bar, 0, maxlen);
+}
 EXPORT_SYMBOL(pci_iomap);
 #endif /* CONFIG_PCI */
diff --git a/net/9p/trans_virtio.c b/net/9p/trans_virtio.c
index daa749c8b3fb..d8e376a5f0f1 100644
--- a/net/9p/trans_virtio.c
+++ b/net/9p/trans_virtio.c
@@ -524,6 +524,12 @@ static int p9_virtio_probe(struct virtio_device *vdev)
         int err;
         struct virtio_chan *chan;
 
+        if (!vdev->config->get) {
+                dev_err(&vdev->dev, "%s failure: config access disabled\n",
+                        __func__);
+                return -EINVAL;
+        }
+
         chan = kmalloc(sizeof(struct virtio_chan), GFP_KERNEL);
         if (!chan) {
                 pr_err("Failed to allocate virtio 9P channel\n");
diff --git a/tools/lguest/Makefile b/tools/lguest/Makefile
index 97bca4871ea3..a107b5e4da13 100644
--- a/tools/lguest/Makefile
+++ b/tools/lguest/Makefile
@@ -1,7 +1,13 @@
 # This creates the demonstration utility "lguest" which runs a Linux guest.
-CFLAGS:=-m32 -Wall -Wmissing-declarations -Wmissing-prototypes -O3 -U_FORTIFY_SOURCE
+CFLAGS:=-m32 -Wall -Wmissing-declarations -Wmissing-prototypes -O3 -U_FORTIFY_SOURCE -Iinclude
 
 all: lguest
 
+include/linux/virtio_types.h: ../../include/uapi/linux/virtio_types.h
+        mkdir -p include/linux 2>&1 || true
+        ln -sf ../../../../include/uapi/linux/virtio_types.h $@
+
+lguest: include/linux/virtio_types.h
+
 clean:
         rm -f lguest
diff --git a/tools/lguest/lguest.c b/tools/lguest/lguest.c
index 32cf2ce15d69..e44052483ed9 100644
--- a/tools/lguest/lguest.c
+++ b/tools/lguest/lguest.c
@@ -41,6 +41,8 @@
 #include <signal.h>
 #include <pwd.h>
 #include <grp.h>
+#include <sys/user.h>
+#include <linux/pci_regs.h>
 
 #ifndef VIRTIO_F_ANY_LAYOUT
 #define VIRTIO_F_ANY_LAYOUT             27
@@ -61,12 +63,19 @@ typedef uint16_t u16;
 typedef uint8_t u8;
 /*:*/
 
-#include <linux/virtio_config.h>
-#include <linux/virtio_net.h>
-#include <linux/virtio_blk.h>
-#include <linux/virtio_console.h>
-#include <linux/virtio_rng.h>
+#define VIRTIO_CONFIG_NO_LEGACY
+#define VIRTIO_PCI_NO_LEGACY
+#define VIRTIO_BLK_NO_LEGACY
+#define VIRTIO_NET_NO_LEGACY
+
+/* Use in-kernel ones, which defines VIRTIO_F_VERSION_1 */
+#include "../../include/uapi/linux/virtio_config.h"
+#include "../../include/uapi/linux/virtio_net.h"
+#include "../../include/uapi/linux/virtio_blk.h"
+#include "../../include/uapi/linux/virtio_console.h"
+#include "../../include/uapi/linux/virtio_rng.h"
 #include <linux/virtio_ring.h>
+#include "../../include/uapi/linux/virtio_pci.h"
 #include <asm/bootparam.h>
 #include "../../include/linux/lguest_launcher.h"
 
@@ -91,13 +100,16 @@ static bool verbose;
 /* The pointer to the start of guest memory. */
 static void *guest_base;
 /* The maximum guest physical address allowed, and maximum possible. */
-static unsigned long guest_limit, guest_max;
+static unsigned long guest_limit, guest_max, guest_mmio;
 /* The /dev/lguest file descriptor. */
 static int lguest_fd;
 
 /* a per-cpu variable indicating whose vcpu is currently running */
 static unsigned int __thread cpu_id;
 
+/* 5 bit device number in the PCI_CONFIG_ADDR => 32 only */
+#define MAX_PCI_DEVICES 32
+
 /* This is our list of devices. */
 struct device_list {
         /* Counter to assign interrupt numbers. */
@@ -106,30 +118,50 @@ struct device_list {
         /* Counter to print out convenient device numbers. */
         unsigned int device_num;
 
-        /* The descriptor page for the devices. */
-        u8 *descpage;
-
-        /* A single linked list of devices. */
-        struct device *dev;
-        /* And a pointer to the last device for easy append. */
-        struct device *lastdev;
+        /* PCI devices. */
+        struct device *pci[MAX_PCI_DEVICES];
 };
 
 /* The list of Guest devices, based on command line arguments. */
 static struct device_list devices;
 
-/* The device structure describes a single device. */
-struct device {
-        /* The linked-list pointer. */
-        struct device *next;
+struct virtio_pci_cfg_cap {
+        struct virtio_pci_cap cap;
+        u32 pci_cfg_data; /* Data for BAR access. */
+};
 
-        /* The device's descriptor, as mapped into the Guest. */
-        struct lguest_device_desc *desc;
+struct virtio_pci_mmio {
+        struct virtio_pci_common_cfg cfg;
+        u16 notify;
+        u8 isr;
+        u8 padding;
+        /* Device-specific configuration follows this. */
+};
 
-        /* We can't trust desc values once Guest has booted: we use these. */
-        unsigned int feature_len;
-        unsigned int num_vq;
+/* This is the layout (little-endian) of the PCI config space. */
+struct pci_config {
+        u16 vendor_id, device_id;
+        u16 command, status;
+        u8 revid, prog_if, subclass, class;
+        u8 cacheline_size, lat_timer, header_type, bist;
+        u32 bar[6];
+        u32 cardbus_cis_ptr;
+        u16 subsystem_vendor_id, subsystem_device_id;
+        u32 expansion_rom_addr;
+        u8 capabilities, reserved1[3];
+        u32 reserved2;
+        u8 irq_line, irq_pin, min_grant, max_latency;
+
+        /* Now, this is the linked capability list. */
+        struct virtio_pci_cap common;
+        struct virtio_pci_notify_cap notify;
+        struct virtio_pci_cap isr;
+        struct virtio_pci_cap device;
+        struct virtio_pci_cfg_cap cfg_access;
+};
 
+/* The device structure describes a single device. */
+struct device {
         /* The name of this device, for --verbose. */
         const char *name;
 
@@ -139,6 +171,25 @@ struct device {
         /* Is it operational */
         bool running;
 
+        /* Has it written FEATURES_OK but not re-checked it? */
+        bool wrote_features_ok;
+
+        /* PCI configuration */
+        union {
+                struct pci_config config;
+                u32 config_words[sizeof(struct pci_config) / sizeof(u32)];
+        };
+
+        /* Features we offer, and those accepted. */
+        u64 features, features_accepted;
+
+        /* Device-specific config hangs off the end of this. */
+        struct virtio_pci_mmio *mmio;
+
+        /* PCI MMIO resources (all in BAR0) */
+        size_t mmio_size;
+        u32 mmio_addr;
+
         /* Device-specific data. */
         void *priv;
 };
@@ -150,12 +201,15 @@ struct virtqueue {
         /* Which device owns me. */
         struct device *dev;
 
-        /* The configuration for this queue. */
-        struct lguest_vqconfig config;
+        /* Name for printing errors. */
+        const char *name;
 
         /* The actual ring of buffers. */
         struct vring vring;
 
+        /* The information about this virtqueue (we only use queue_size on) */
+        struct virtio_pci_common_cfg pci_config;
+
         /* Last available index we saw. */
         u16 last_avail_idx;
 
@@ -199,6 +253,16 @@ static struct termios orig_term;
 #define le32_to_cpu(v32) (v32)
 #define le64_to_cpu(v64) (v64)
 
+/*
+ * A real device would ignore weird/non-compliant driver behaviour.  We
+ * stop and flag it, to help debugging Linux problems.
+ */
+#define bad_driver(d, fmt, ...) \
+        errx(1, "%s: bad driver: " fmt, (d)->name, ## __VA_ARGS__)
+#define bad_driver_vq(vq, fmt, ...)                            \
+        errx(1, "%s vq %s: bad driver: " fmt, (vq)->dev->name, \
+             vq->name, ## __VA_ARGS__)
+
 /* Is this iovec empty? */
 static bool iov_empty(const struct iovec iov[], unsigned int num_iov)
 {
@@ -211,7 +275,8 @@ static bool iov_empty(const struct iovec iov[], unsigned int num_iov)
 }
 
 /* Take len bytes from the front of this iovec. */
-static void iov_consume(struct iovec iov[], unsigned num_iov,
+static void iov_consume(struct device *d,
+                        struct iovec iov[], unsigned num_iov,
                         void *dest, unsigned len)
 {
         unsigned int i;
@@ -229,14 +294,7 @@ static void iov_consume(struct iovec iov[], unsigned num_iov,
                 len -= used;
         }
         if (len != 0)
-                errx(1, "iovec too short!");
-}
-
-/* The device virtqueue descriptors are followed by feature bitmasks. */
-static u8 *get_feature_bits(struct device *dev)
-{
-        return (u8 *)(dev->desc + 1)
-                + dev->num_vq * sizeof(struct lguest_vqconfig);
+                bad_driver(d, "iovec too short!");
 }
 
 /*L:100
@@ -309,14 +367,20 @@ static void *map_zeroed_pages(unsigned int num)
         return addr + getpagesize();
 }
 
-/* Get some more pages for a device. */
-static void *get_pages(unsigned int num)
+/* Get some bytes which won't be mapped into the guest. */
+static unsigned long get_mmio_region(size_t size)
 {
-        void *addr = from_guest_phys(guest_limit);
+        unsigned long addr = guest_mmio;
+        size_t i;
+
+        if (!size)
+                return addr;
+
+        /* Size has to be a power of 2 (and multiple of 16) */
+        for (i = 1; i < size; i <<= 1);
+
+        guest_mmio += i;
 
-        guest_limit += num * getpagesize();
-        if (guest_limit > guest_max)
-                errx(1, "Not enough memory for devices");
         return addr;
 }
 
@@ -547,9 +611,11 @@ static void tell_kernel(unsigned long start)
 {
         unsigned long args[] = { LHREQ_INITIALIZE,
                                  (unsigned long)guest_base,
-                                 guest_limit / getpagesize(), start };
-        verbose("Guest: %p - %p (%#lx)\n",
-                guest_base, guest_base + guest_limit, guest_limit);
+                                 guest_limit / getpagesize(), start,
+                                 (guest_mmio+getpagesize()-1) / getpagesize() };
+        verbose("Guest: %p - %p (%#lx, MMIO %#lx)\n",
+                guest_base, guest_base + guest_limit,
+                guest_limit, guest_mmio);
         lguest_fd = open_or_die("/dev/lguest", O_RDWR);
         if (write(lguest_fd, args, sizeof(args)) < 0)
                 err(1, "Writing to /dev/lguest");
@@ -564,7 +630,8 @@ static void tell_kernel(unsigned long start)
  * we have a convenient routine which checks it and exits with an error message
  * if something funny is going on:
  */
-static void *_check_pointer(unsigned long addr, unsigned int size,
+static void *_check_pointer(struct device *d,
+                            unsigned long addr, unsigned int size,
                             unsigned int line)
 {
         /*
@@ -572,7 +639,8 @@ static void *_check_pointer(unsigned long addr, unsigned int size,
          * or addr + size wraps around.
          */
         if ((addr + size) > guest_limit || (addr + size) < addr)
-                errx(1, "%s:%i: Invalid address %#lx", __FILE__, line, addr);
+                bad_driver(d, "%s:%i: Invalid address %#lx",
+                           __FILE__, line, addr);
         /*
          * We return a pointer for the caller's convenience, now we know it's
          * safe to use.
@@ -580,14 +648,14 @@ static void *_check_pointer(unsigned long addr, unsigned int size,
         return from_guest_phys(addr);
 }
 /* A macro which transparently hands the line number to the real function. */
-#define check_pointer(addr,size) _check_pointer(addr, size, __LINE__)
+#define check_pointer(d,addr,size) _check_pointer(d, addr, size, __LINE__)
 
 /*
  * Each buffer in the virtqueues is actually a chain of descriptors.  This
  * function returns the next descriptor in the chain, or vq->vring.num if we're
  * at the end.
  */
-static unsigned next_desc(struct vring_desc *desc,
+static unsigned next_desc(struct device *d, struct vring_desc *desc,
                           unsigned int i, unsigned int max)
 {
         unsigned int next;
@@ -602,7 +670,7 @@ static unsigned next_desc(struct vring_desc *desc,
         wmb();
 
         if (next >= max)
-                errx(1, "Desc next is %u", next);
+                bad_driver(d, "Desc next is %u", next);
 
         return next;
 }
@@ -613,21 +681,48 @@ static unsigned next_desc(struct vring_desc *desc,
  */
 static void trigger_irq(struct virtqueue *vq)
 {
-        unsigned long buf[] = { LHREQ_IRQ, vq->config.irq };
+        unsigned long buf[] = { LHREQ_IRQ, vq->dev->config.irq_line };
 
         /* Don't inform them if nothing used. */
         if (!vq->pending_used)
                 return;
         vq->pending_used = 0;
 
-        /* If they don't want an interrupt, don't send one... */
+        /*
+         * 2.4.7.1:
+         *
+         *  If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
+         *    The driver MUST set flags to 0 or 1. 
+         */
+        if (vq->vring.avail->flags > 1)
+                bad_driver_vq(vq, "avail->flags = %u\n", vq->vring.avail->flags);
+
+        /*
+         * 2.4.7.2:
+         *
+         *  If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
+         *
+         *     - The device MUST ignore the used_event value.
+         *     - After the device writes a descriptor index into the used ring:
+         *         - If flags is 1, the device SHOULD NOT send an interrupt.
+         *         - If flags is 0, the device MUST send an interrupt.
+         */
         if (vq->vring.avail->flags & VRING_AVAIL_F_NO_INTERRUPT) {
                 return;
         }
 
+        /*
+         * 4.1.4.5.1:
+         *
+         *  If MSI-X capability is disabled, the device MUST set the Queue
+         *  Interrupt bit in ISR status before sending a virtqueue notification
+         *  to the driver.
+         */
+        vq->dev->mmio->isr = 0x1;
+
         /* Send the Guest an interrupt tell them we used something up. */
         if (write(lguest_fd, buf, sizeof(buf)) != 0)
-                err(1, "Triggering irq %i", vq->config.irq);
+                err(1, "Triggering irq %i", vq->dev->config.irq_line);
 }
 
 /*
@@ -646,6 +741,14 @@ static unsigned wait_for_vq_desc(struct virtqueue *vq,
         struct vring_desc *desc;
         u16 last_avail = lg_last_avail(vq);
 
+        /*
+         * 2.4.7.1:
+         *
+         *   The driver MUST handle spurious interrupts from the device.
+         *
+         * That's why this is a while loop.
+         */
+
         /* There's nothing available? */
         while (last_avail == vq->vring.avail->idx) {
                 u64 event;
@@ -679,8 +782,8 @@ static unsigned wait_for_vq_desc(struct virtqueue *vq,
 
         /* Check it isn't doing very strange things with descriptor numbers. */
         if ((u16)(vq->vring.avail->idx - last_avail) > vq->vring.num)
-                errx(1, "Guest moved used index from %u to %u",
-                     last_avail, vq->vring.avail->idx);
+                bad_driver_vq(vq, "Guest moved used index from %u to %u",
+                              last_avail, vq->vring.avail->idx);
 
         /* 
          * Make sure we read the descriptor number *after* we read the ring
@@ -697,7 +800,7 @@ static unsigned wait_for_vq_desc(struct virtqueue *vq,
 
         /* If their number is silly, that's a fatal mistake. */
         if (head >= vq->vring.num)
-                errx(1, "Guest says index %u is available", head);
+                bad_driver_vq(vq, "Guest says index %u is available", head);
 
         /* When we start there are none of either input nor output. */
         *out_num = *in_num = 0;
@@ -712,24 +815,73 @@ static unsigned wait_for_vq_desc(struct virtqueue *vq,
          * that: no rmb() required.
          */
 
-        /*
-         * If this is an indirect entry, then this buffer contains a descriptor
-         * table which we handle as if it's any normal descriptor chain.
-         */
-        if (desc[i].flags & VRING_DESC_F_INDIRECT) {
-                if (desc[i].len % sizeof(struct vring_desc))
-                        errx(1, "Invalid size for indirect buffer table");
+        do {
+                /*
+                 * If this is an indirect entry, then this buffer contains a
+                 * descriptor table which we handle as if it's any normal
+                 * descriptor chain.
+                 */
+                if (desc[i].flags & VRING_DESC_F_INDIRECT) {
+                        /* 2.4.5.3.1:
+                         *
+                         *  The driver MUST NOT set the VIRTQ_DESC_F_INDIRECT
+                         *  flag unless the VIRTIO_F_INDIRECT_DESC feature was
+                         *  negotiated.
+                         */
+                        if (!(vq->dev->features_accepted &
+                              (1<<VIRTIO_RING_F_INDIRECT_DESC)))
+                                bad_driver_vq(vq, "vq indirect not negotiated");
 
-                max = desc[i].len / sizeof(struct vring_desc);
-                desc = check_pointer(desc[i].addr, desc[i].len);
-                i = 0;
-        }
+                        /*
+                         * 2.4.5.3.1:
+                         *
+                         *   The driver MUST NOT set the VIRTQ_DESC_F_INDIRECT
+                         *   flag within an indirect descriptor (ie. only one
+                         *   table per descriptor).
+                         */
+                        if (desc != vq->vring.desc)
+                                bad_driver_vq(vq, "Indirect within indirect");
+
+                        /*
+                         * Proposed update VIRTIO-134 spells this out:
+                         *
+                         *   A driver MUST NOT set both VIRTQ_DESC_F_INDIRECT
+                         *   and VIRTQ_DESC_F_NEXT in flags.
+                         */
+                        if (desc[i].flags & VRING_DESC_F_NEXT)
+                                bad_driver_vq(vq, "indirect and next together");
+
+                        if (desc[i].len % sizeof(struct vring_desc))
+                                bad_driver_vq(vq,
+                                              "Invalid size for indirect table");
+                        /*
+                         * 2.4.5.3.2:
+                         *
+                         *  The device MUST ignore the write-only flag
+                         *  (flags&VIRTQ_DESC_F_WRITE) in the descriptor that
+                         *  refers to an indirect table.
+                         *
+                         * We ignore it here: :)
+                         */
+
+                        max = desc[i].len / sizeof(struct vring_desc);
+                        desc = check_pointer(vq->dev, desc[i].addr, desc[i].len);
+                        i = 0;
+
+                        /* 2.4.5.3.1:
+                         *
+                         *  A driver MUST NOT create a descriptor chain longer
+                         *  than the Queue Size of the device.
+                         */
+                        if (max > vq->pci_config.queue_size)
+                                bad_driver_vq(vq,
+                                              "indirect has too many entries");
+                }
 
-        do {
                 /* Grab the first descriptor, and check it's OK. */
                 iov[*out_num + *in_num].iov_len = desc[i].len;
                 iov[*out_num + *in_num].iov_base
-                        = check_pointer(desc[i].addr, desc[i].len);
+                        = check_pointer(vq->dev, desc[i].addr, desc[i].len);
                 /* If this is an input descriptor, increment that count. */
                 if (desc[i].flags & VRING_DESC_F_WRITE)
                         (*in_num)++;
@@ -739,14 +891,15 @@ static unsigned wait_for_vq_desc(struct virtqueue *vq,
                          * to come before any input descriptors.
                          */
                         if (*in_num)
-                                errx(1, "Descriptor has out after in");
+                                bad_driver_vq(vq,
+                                              "Descriptor has out after in");
                         (*out_num)++;
                 }
 
                 /* If we've got too many, that implies a descriptor loop. */
                 if (*out_num + *in_num > max)
-                        errx(1, "Looped descriptor");
-        } while ((i = next_desc(desc, i, max)) != max);
+                        bad_driver_vq(vq, "Looped descriptor");
+        } while ((i = next_desc(vq->dev, desc, i, max)) != max);
 
         return head;
 }
@@ -803,7 +956,7 @@ static void console_input(struct virtqueue *vq)
         /* Make sure there's a descriptor available. */
         head = wait_for_vq_desc(vq, iov, &out_num, &in_num);
         if (out_num)
-                errx(1, "Output buffers in console in queue?");
+                bad_driver_vq(vq, "Output buffers in console in queue?");
 
         /* Read into it.  This is where we usually wait. */
         len = readv(STDIN_FILENO, iov, in_num);
@@ -856,7 +1009,7 @@ static void console_output(struct virtqueue *vq)
         /* We usually wait in here, for the Guest to give us something. */
         head = wait_for_vq_desc(vq, iov, &out, &in);
         if (in)
-                errx(1, "Input buffers in console output queue?");
+                bad_driver_vq(vq, "Input buffers in console output queue?");
 
         /* writev can return a partial write, so we loop here. */
         while (!iov_empty(iov, out)) {
@@ -865,7 +1018,7 @@ static void console_output(struct virtqueue *vq)
                         warn("Write to stdout gave %i (%d)", len, errno);
                         break;
                 }
-                iov_consume(iov, out, NULL, len);
+                iov_consume(vq->dev, iov, out, NULL, len);
         }
 
         /*
@@ -894,7 +1047,7 @@ static void net_output(struct virtqueue *vq)
         /* We usually wait in here for the Guest to give us a packet. */
         head = wait_for_vq_desc(vq, iov, &out, &in);
         if (in)
-                errx(1, "Input buffers in net output queue?");
+                bad_driver_vq(vq, "Input buffers in net output queue?");
         /*
          * Send the whole thing through to /dev/net/tun.  It expects the exact
          * same format: what a coincidence!
@@ -942,7 +1095,7 @@ static void net_input(struct virtqueue *vq)
          */
         head = wait_for_vq_desc(vq, iov, &out, &in);
         if (out)
-                errx(1, "Output buffers in net input queue?");
+                bad_driver_vq(vq, "Output buffers in net input queue?");
 
         /*
          * If it looks like we'll block reading from the tun device, send them
@@ -986,6 +1139,12 @@ static void kill_launcher(int signal)
         kill(0, SIGTERM);
 }
 
+static void reset_vq_pci_config(struct virtqueue *vq)
+{
+        vq->pci_config.queue_size = VIRTQUEUE_NUM;
+        vq->pci_config.queue_enable = 0;
+}
+
 static void reset_device(struct device *dev)
 {
         struct virtqueue *vq;
@@ -993,53 +1152,705 @@ static void reset_device(struct device *dev)
         verbose("Resetting device %s\n", dev->name);
 
         /* Clear any features they've acked. */
-        memset(get_feature_bits(dev) + dev->feature_len, 0, dev->feature_len);
+        dev->features_accepted = 0;
 
         /* We're going to be explicitly killing threads, so ignore them. */
         signal(SIGCHLD, SIG_IGN);
 
-        /* Zero out the virtqueues, get rid of their threads */
+        /*
+         * 4.1.4.3.1:
+         *
+         *   The device MUST present a 0 in queue_enable on reset. 
+         *
+         * This means we set it here, and reset the saved ones in every vq.
+         */
+        dev->mmio->cfg.queue_enable = 0;
+
+        /* Get rid of the virtqueue threads */
         for (vq = dev->vq; vq; vq = vq->next) {
+                vq->last_avail_idx = 0;
+                reset_vq_pci_config(vq);
                 if (vq->thread != (pid_t)-1) {
                         kill(vq->thread, SIGTERM);
                         waitpid(vq->thread, NULL, 0);
                         vq->thread = (pid_t)-1;
                 }
-                memset(vq->vring.desc, 0,
-                       vring_size(vq->config.num, LGUEST_VRING_ALIGN));
-                lg_last_avail(vq) = 0;
         }
         dev->running = false;
+        dev->wrote_features_ok = false;
 
         /* Now we care if threads die. */
         signal(SIGCHLD, (void *)kill_launcher);
 }
 
+static void cleanup_devices(void)
+{
+        unsigned int i;
+
+        for (i = 1; i < MAX_PCI_DEVICES; i++) {
+                struct device *d = devices.pci[i];
+                if (!d)
+                        continue;
+                reset_device(d);
+        }
+
+        /* If we saved off the original terminal settings, restore them now. */
+        if (orig_term.c_lflag & (ISIG|ICANON|ECHO))
+                tcsetattr(STDIN_FILENO, TCSANOW, &orig_term);
+}
+
+/*L:217
+ * We do PCI.  This is mainly done to let us test the kernel virtio PCI
+ * code.
+ */
+
+/* Linux expects a PCI host bridge: ours is a dummy, and first on the bus. */
+static struct device pci_host_bridge;
+
+static void init_pci_host_bridge(void)
+{
+        pci_host_bridge.name = "PCI Host Bridge";
+        pci_host_bridge.config.class = 0x06; /* bridge */
+        pci_host_bridge.config.subclass = 0; /* host bridge */
+        devices.pci[0] = &pci_host_bridge;
+}
+
+/* The IO ports used to read the PCI config space. */
+#define PCI_CONFIG_ADDR 0xCF8
+#define PCI_CONFIG_DATA 0xCFC
+
+/*
+ * Not really portable, but does help readability: this is what the Guest
+ * writes to the PCI_CONFIG_ADDR IO port.
+ */
+union pci_config_addr {
+        struct {
+                unsigned mbz: 2;
+                unsigned offset: 6;
+                unsigned funcnum: 3;
+                unsigned devnum: 5;
+                unsigned busnum: 8;
+                unsigned reserved: 7;
+                unsigned enabled : 1;
+        } bits;
+        u32 val;
+};
+
+/*
+ * We cache what they wrote to the address port, so we know what they're
+ * talking about when they access the data port.
+ */
+static union pci_config_addr pci_config_addr;
+
+static struct device *find_pci_device(unsigned int index)
+{
+        return devices.pci[index];
+}
+
+/* PCI can do 1, 2 and 4 byte reads; we handle that here. */
+static void ioread(u16 off, u32 v, u32 mask, u32 *val)
+{
+        assert(off < 4);
+        assert(mask == 0xFF || mask == 0xFFFF || mask == 0xFFFFFFFF);
+        *val = (v >> (off * 8)) & mask;
+}
+
+/* PCI can do 1, 2 and 4 byte writes; we handle that here. */
+static void iowrite(u16 off, u32 v, u32 mask, u32 *dst)
+{
+        assert(off < 4);
+        assert(mask == 0xFF || mask == 0xFFFF || mask == 0xFFFFFFFF);
+        *dst &= ~(mask << (off * 8));
+        *dst |= (v & mask) << (off * 8);
+}
+
+/*
+ * Where PCI_CONFIG_DATA accesses depends on the previous write to
+ * PCI_CONFIG_ADDR.
+ */
+static struct device *dev_and_reg(u32 *reg)
+{
+        if (!pci_config_addr.bits.enabled)
+                return NULL;
+
+        if (pci_config_addr.bits.funcnum != 0)
+                return NULL;
+
+        if (pci_config_addr.bits.busnum != 0)
+                return NULL;
+
+        if (pci_config_addr.bits.offset * 4 >= sizeof(struct pci_config))
+                return NULL;
+
+        *reg = pci_config_addr.bits.offset;
+        return find_pci_device(pci_config_addr.bits.devnum);
+}
+
+/*
+ * We can get invalid combinations of values while they're writing, so we
+ * only fault if they try to write with some invalid bar/offset/length.
+ */
+static bool valid_bar_access(struct device *d,
+                             struct virtio_pci_cfg_cap *cfg_access)
+{
+        /* We only have 1 bar (BAR0) */
+        if (cfg_access->cap.bar != 0)
+                return false;
+
+        /* Check it's within BAR0. */
+        if (cfg_access->cap.offset >= d->mmio_size
+            || cfg_access->cap.offset + cfg_access->cap.length > d->mmio_size)
+                return false;
+
+        /* Check length is 1, 2 or 4. */
+        if (cfg_access->cap.length != 1
+            && cfg_access->cap.length != 2
+            && cfg_access->cap.length != 4)
+                return false;
+
+        /*
+         * 4.1.4.7.2:
+         *
+         *  The driver MUST NOT write a cap.offset which is not a multiple of
+         *  cap.length (ie. all accesses MUST be aligned).
+         */
+        if (cfg_access->cap.offset % cfg_access->cap.length != 0)
+                return false;
+
+        /* Return pointer into word in BAR0. */
+        return true;
+}
+
+/* Is this accessing the PCI config address port?. */
+static bool is_pci_addr_port(u16 port)
+{
+        return port >= PCI_CONFIG_ADDR && port < PCI_CONFIG_ADDR + 4;
+}
+
+static bool pci_addr_iowrite(u16 port, u32 mask, u32 val)
+{
+        iowrite(port - PCI_CONFIG_ADDR, val, mask,
+                &pci_config_addr.val);
+        verbose("PCI%s: %#x/%x: bus %u dev %u func %u reg %u\n",
+                pci_config_addr.bits.enabled ? "" : " DISABLED",
+                val, mask,
+                pci_config_addr.bits.busnum,
+                pci_config_addr.bits.devnum,
+                pci_config_addr.bits.funcnum,
+                pci_config_addr.bits.offset);
+        return true;
+}
+
+static void pci_addr_ioread(u16 port, u32 mask, u32 *val)
+{
+        ioread(port - PCI_CONFIG_ADDR, pci_config_addr.val, mask, val);
+}
+
+/* Is this accessing the PCI config data port?. */
+static bool is_pci_data_port(u16 port)
+{
+        return port >= PCI_CONFIG_DATA && port < PCI_CONFIG_DATA + 4;
+}
+
+static void emulate_mmio_write(struct device *d, u32 off, u32 val, u32 mask);
+
+static bool pci_data_iowrite(u16 port, u32 mask, u32 val)
+{
+        u32 reg, portoff;
+        struct device *d = dev_and_reg(&reg);
+
+        /* Complain if they don't belong to a device. */
+        if (!d)
+                return false;
+
+        /* They can do 1 byte writes, etc. */
+        portoff = port - PCI_CONFIG_DATA;
+
+        /*
+         * PCI uses a weird way to determine the BAR size: the OS
+         * writes all 1's, and sees which ones stick.
+         */
+        if (&d->config_words[reg] == &d->config.bar[0]) {
+                int i;
+
+                iowrite(portoff, val, mask, &d->config.bar[0]);
+                for (i = 0; (1 << i) < d->mmio_size; i++)
+                        d->config.bar[0] &= ~(1 << i);
+                return true;
+        } else if ((&d->config_words[reg] > &d->config.bar[0]
+                    && &d->config_words[reg] <= &d->config.bar[6])
+                   || &d->config_words[reg] == &d->config.expansion_rom_addr) {
+                /* Allow writing to any other BAR, or expansion ROM */
+                iowrite(portoff, val, mask, &d->config_words[reg]);
+                return true;
+                /* We let them overide latency timer and cacheline size */
+        } else if (&d->config_words[reg] == (void *)&d->config.cacheline_size) {
+                /* Only let them change the first two fields. */
+                if (mask == 0xFFFFFFFF)
+                        mask = 0xFFFF;
+                iowrite(portoff, val, mask, &d->config_words[reg]);
+                return true;
+        } else if (&d->config_words[reg] == (void *)&d->config.command
+                   && mask == 0xFFFF) {
+                /* Ignore command writes. */
+                return true;
+        } else if (&d->config_words[reg]
+                   == (void *)&d->config.cfg_access.cap.bar
+                   || &d->config_words[reg]
+                   == &d->config.cfg_access.cap.length
+                   || &d->config_words[reg]
+                   == &d->config.cfg_access.cap.offset) {
+
+                /*
+                 * The VIRTIO_PCI_CAP_PCI_CFG capability
+                 * provides a backdoor to access the MMIO
+                 * regions without mapping them.  Weird, but
+                 * useful.
+                 */
+                iowrite(portoff, val, mask, &d->config_words[reg]);
+                return true;
+        } else if (&d->config_words[reg] == &d->config.cfg_access.pci_cfg_data) {
+                u32 write_mask;
+
+                /*
+                 * 4.1.4.7.1:
+                 *
+                 *  Upon detecting driver write access to pci_cfg_data, the
+                 *  device MUST execute a write access at offset cap.offset at
+                 *  BAR selected by cap.bar using the first cap.length bytes
+                 *  from pci_cfg_data.
+                 */
+
+                /* Must be bar 0 */
+                if (!valid_bar_access(d, &d->config.cfg_access))
+                        return false;
+
+                iowrite(portoff, val, mask, &d->config.cfg_access.pci_cfg_data);
+
+                /*
+                 * Now emulate a write.  The mask we use is set by
+                 * len, *not* this write!
+                 */
+                write_mask = (1ULL<<(8*d->config.cfg_access.cap.length)) - 1;
+                verbose("Window writing %#x/%#x to bar %u, offset %u len %u\n",
+                        d->config.cfg_access.pci_cfg_data, write_mask,
+                        d->config.cfg_access.cap.bar,
+                        d->config.cfg_access.cap.offset,
+                        d->config.cfg_access.cap.length);
+
+                emulate_mmio_write(d, d->config.cfg_access.cap.offset,
+                                   d->config.cfg_access.pci_cfg_data,
+                                   write_mask);
+                return true;
+        }
+
+        /*
+         * 4.1.4.1:
+         *
+         *  The driver MUST NOT write into any field of the capability
+         *  structure, with the exception of those with cap_type
+         *  VIRTIO_PCI_CAP_PCI_CFG...
+         */
+        return false;
+}
+
+static u32 emulate_mmio_read(struct device *d, u32 off, u32 mask);
+
+static void pci_data_ioread(u16 port, u32 mask, u32 *val)
+{
+        u32 reg;
+        struct device *d = dev_and_reg(&reg);
+
+        if (!d)
+                return;
+
+        /* Read through the PCI MMIO access window is special */
+        if (&d->config_words[reg] == &d->config.cfg_access.pci_cfg_data) {
+                u32 read_mask;
+
+                /*
+                 * 4.1.4.7.1:
+                 *
+                 *  Upon detecting driver read access to pci_cfg_data, the
+                 *  device MUST execute a read access of length cap.length at
+                 *  offset cap.offset at BAR selected by cap.bar and store the
+                 *  first cap.length bytes in pci_cfg_data.
+                 */
+                /* Must be bar 0 */
+                if (!valid_bar_access(d, &d->config.cfg_access))
+                        bad_driver(d,
+                             "Invalid cfg_access to bar%u, offset %u len %u",
+                             d->config.cfg_access.cap.bar,
+                             d->config.cfg_access.cap.offset,
+                             d->config.cfg_access.cap.length);
+
+                /*
+                 * Read into the window.  The mask we use is set by
+                 * len, *not* this read!
+                 */
+                read_mask = (1ULL<<(8*d->config.cfg_access.cap.length))-1;
+                d->config.cfg_access.pci_cfg_data
+                        = emulate_mmio_read(d,
+                                            d->config.cfg_access.cap.offset,
+                                            read_mask);
+                verbose("Window read %#x/%#x from bar %u, offset %u len %u\n",
+                        d->config.cfg_access.pci_cfg_data, read_mask,
+                        d->config.cfg_access.cap.bar,
+                        d->config.cfg_access.cap.offset,
+                        d->config.cfg_access.cap.length);
+        }
+        ioread(port - PCI_CONFIG_DATA, d->config_words[reg], mask, val);
+}
+
 /*L:216
- * This actually creates the thread which services the virtqueue for a device.
+ * This is where we emulate a handful of Guest instructions.  It's ugly
+ * and we used to do it in the kernel but it grew over time.
+ */
+
+/*
+ * We use the ptrace syscall's pt_regs struct to talk about registers
+ * to lguest: these macros convert the names to the offsets.
+ */
+#define getreg(name) getreg_off(offsetof(struct user_regs_struct, name))
+#define setreg(name, val) \
+        setreg_off(offsetof(struct user_regs_struct, name), (val))
+
+static u32 getreg_off(size_t offset)
+{
+        u32 r;
+        unsigned long args[] = { LHREQ_GETREG, offset };
+
+        if (pwrite(lguest_fd, args, sizeof(args), cpu_id) < 0)
+                err(1, "Getting register %u", offset);
+        if (pread(lguest_fd, &r, sizeof(r), cpu_id) != sizeof(r))
+                err(1, "Reading register %u", offset);
+
+        return r;
+}
+
+static void setreg_off(size_t offset, u32 val)
+{
+        unsigned long args[] = { LHREQ_SETREG, offset, val };
+
+        if (pwrite(lguest_fd, args, sizeof(args), cpu_id) < 0)
+                err(1, "Setting register %u", offset);
+}
+
+/* Get register by instruction encoding */
+static u32 getreg_num(unsigned regnum, u32 mask)
+{
+        /* 8 bit ops use regnums 4-7 for high parts of word */
+        if (mask == 0xFF && (regnum & 0x4))
+                return getreg_num(regnum & 0x3, 0xFFFF) >> 8;
+
+        switch (regnum) {
+        case 0: return getreg(eax) & mask;
+        case 1: return getreg(ecx) & mask;
+        case 2: return getreg(edx) & mask;
+        case 3: return getreg(ebx) & mask;
+        case 4: return getreg(esp) & mask;
+        case 5: return getreg(ebp) & mask;
+        case 6: return getreg(esi) & mask;
+        case 7: return getreg(edi) & mask;
+        }
+        abort();
+}
+
+/* Set register by instruction encoding */
+static void setreg_num(unsigned regnum, u32 val, u32 mask)
+{
+        /* Don't try to set bits out of range */
+        assert(~(val & ~mask));
+
+        /* 8 bit ops use regnums 4-7 for high parts of word */
+        if (mask == 0xFF && (regnum & 0x4)) {
+                /* Construct the 16 bits we want. */
+                val = (val << 8) | getreg_num(regnum & 0x3, 0xFF);
+                setreg_num(regnum & 0x3, val, 0xFFFF);
+                return;
+        }
+
+        switch (regnum) {
+        case 0: setreg(eax, val | (getreg(eax) & ~mask)); return;
+        case 1: setreg(ecx, val | (getreg(ecx) & ~mask)); return;
+        case 2: setreg(edx, val | (getreg(edx) & ~mask)); return;
+        case 3: setreg(ebx, val | (getreg(ebx) & ~mask)); return;
+        case 4: setreg(esp, val | (getreg(esp) & ~mask)); return;
+        case 5: setreg(ebp, val | (getreg(ebp) & ~mask)); return;
+        case 6: setreg(esi, val | (getreg(esi) & ~mask)); return;
+        case 7: setreg(edi, val | (getreg(edi) & ~mask)); return;
+        }
+        abort();
+}
+
+/* Get bytes of displacement appended to instruction, from r/m encoding */
+static u32 insn_displacement_len(u8 mod_reg_rm)
+{
+        /* Switch on the mod bits */
+        switch (mod_reg_rm >> 6) {
+        case 0:
+                /* If mod == 0, and r/m == 101, 16-bit displacement follows */
+                if ((mod_reg_rm & 0x7) == 0x5)
+                        return 2;
+                /* Normally, mod == 0 means no literal displacement */
+                return 0;
+        case 1:
+                /* One byte displacement */
+                return 1;
+        case 2:
+                /* Four byte displacement */
+                return 4;
+        case 3:
+                /* Register mode */
+                return 0;
+        }
+        abort();
+}
+
+static void emulate_insn(const u8 insn[])
+{
+        unsigned long args[] = { LHREQ_TRAP, 13 };
+        unsigned int insnlen = 0, in = 0, small_operand = 0, byte_access;
+        unsigned int eax, port, mask;
+        /*
+         * Default is to return all-ones on IO port reads, which traditionally
+         * means "there's nothing there".
+         */
+        u32 val = 0xFFFFFFFF;
+
+        /*
+         * This must be the Guest kernel trying to do something, not userspace!
+         * The bottom two bits of the CS segment register are the privilege
+         * level.
+         */
+        if ((getreg(xcs) & 3) != 0x1)
+                goto no_emulate;
+
+        /* Decoding x86 instructions is icky. */
+
+        /*
+         * Around 2.6.33, the kernel started using an emulation for the
+         * cmpxchg8b instruction in early boot on many configurations.  This
+         * code isn't paravirtualized, and it tries to disable interrupts.
+         * Ignore it, which will Mostly Work.
+         */
+        if (insn[insnlen] == 0xfa) {
+                /* "cli", or Clear Interrupt Enable instruction.  Skip it. */
+                insnlen = 1;
+                goto skip_insn;
+        }
+
+        /*
+         * 0x66 is an "operand prefix".  It means a 16, not 32 bit in/out.
+         */
+        if (insn[insnlen] == 0x66) {
+                small_operand = 1;
+                /* The instruction is 1 byte so far, read the next byte. */
+                insnlen = 1;
+        }
+
+        /* If the lower bit isn't set, it's a single byte access */
+        byte_access = !(insn[insnlen] & 1);
+
+        /*
+         * Now we can ignore the lower bit and decode the 4 opcodes
+         * we need to emulate.
+         */
+        switch (insn[insnlen] & 0xFE) {
+        case 0xE4: /* in     <next byte>,%al */
+                port = insn[insnlen+1];
+                insnlen += 2;
+                in = 1;
+                break;
+        case 0xEC: /* in     (%dx),%al */
+                port = getreg(edx) & 0xFFFF;
+                insnlen += 1;
+                in = 1;
+                break;
+        case 0xE6: /* out    %al,<next byte> */
+                port = insn[insnlen+1];
+                insnlen += 2;
+                break;
+        case 0xEE: /* out    %al,(%dx) */
+                port = getreg(edx) & 0xFFFF;
+                insnlen += 1;
+                break;
+        default:
+                /* OK, we don't know what this is, can't emulate. */
+                goto no_emulate;
+        }
+
+        /* Set a mask of the 1, 2 or 4 bytes, depending on size of IO */
+        if (byte_access)
+                mask = 0xFF;
+        else if (small_operand)
+                mask = 0xFFFF;
+        else
+                mask = 0xFFFFFFFF;
+
+        /*
+         * If it was an "IN" instruction, they expect the result to be read
+         * into %eax, so we change %eax.
+         */
+        eax = getreg(eax);
+
+        if (in) {
+                /* This is the PS/2 keyboard status; 1 means ready for output */
+                if (port == 0x64)
+                        val = 1;
+                else if (is_pci_addr_port(port))
+                        pci_addr_ioread(port, mask, &val);
+                else if (is_pci_data_port(port))
+                        pci_data_ioread(port, mask, &val);
+
+                /* Clear the bits we're about to read */
+                eax &= ~mask;
+                /* Copy bits in from val. */
+                eax |= val & mask;
+                /* Now update the register. */
+                setreg(eax, eax);
+        } else {
+                if (is_pci_addr_port(port)) {
+                        if (!pci_addr_iowrite(port, mask, eax))
+                                goto bad_io;
+                } else if (is_pci_data_port(port)) {
+                        if (!pci_data_iowrite(port, mask, eax))
+                                goto bad_io;
+                }
+                /* There are many other ports, eg. CMOS clock, serial
+                 * and parallel ports, so we ignore them all. */
+        }
+
+        verbose("IO %s of %x to %u: %#08x\n",
+                in ? "IN" : "OUT", mask, port, eax);
+skip_insn:
+        /* Finally, we've "done" the instruction, so move past it. */
+        setreg(eip, getreg(eip) + insnlen);
+        return;
+
+bad_io:
+        warnx("Attempt to %s port %u (%#x mask)",
+              in ? "read from" : "write to", port, mask);
+
+no_emulate:
+        /* Inject trap into Guest. */
+        if (write(lguest_fd, args, sizeof(args)) < 0)
+                err(1, "Reinjecting trap 13 for fault at %#x", getreg(eip));
+}
+
+static struct device *find_mmio_region(unsigned long paddr, u32 *off)
+{
+        unsigned int i;
+
+        for (i = 1; i < MAX_PCI_DEVICES; i++) {
+                struct device *d = devices.pci[i];
+
+                if (!d)
+                        continue;
+                if (paddr < d->mmio_addr)
+                        continue;
+                if (paddr >= d->mmio_addr + d->mmio_size)
+                        continue;
+                *off = paddr - d->mmio_addr;
+                return d;
+        }
+        return NULL;
+}
+
+/* FIXME: Use vq array. */
+static struct virtqueue *vq_by_num(struct device *d, u32 num)
+{
+        struct virtqueue *vq = d->vq;
+
+        while (num-- && vq)
+                vq = vq->next;
+
+        return vq;
+}
+
+static void save_vq_config(const struct virtio_pci_common_cfg *cfg,
+                           struct virtqueue *vq)
+{
+        vq->pci_config = *cfg;
+}
+
+static void restore_vq_config(struct virtio_pci_common_cfg *cfg,
+                              struct virtqueue *vq)
+{
+        /* Only restore the per-vq part */
+        size_t off = offsetof(struct virtio_pci_common_cfg, queue_size);
+
+        memcpy((void *)cfg + off, (void *)&vq->pci_config + off,
+               sizeof(*cfg) - off);
+}
+
+/*
+ * 4.1.4.3.2:
+ *
+ *  The driver MUST configure the other virtqueue fields before
+ *  enabling the virtqueue with queue_enable.
+ *
+ * When they enable the virtqueue, we check that their setup is valid.
  */
-static void create_thread(struct virtqueue *vq)
+static void check_virtqueue(struct device *d, struct virtqueue *vq)
+{
+        /* Because lguest is 32 bit, all the descriptor high bits must be 0 */
+        if (vq->pci_config.queue_desc_hi
+            || vq->pci_config.queue_avail_hi
+            || vq->pci_config.queue_used_hi)
+                bad_driver_vq(vq, "invalid 64-bit queue address");
+
+        /*
+         * 2.4.1:
+         *
+         *  The driver MUST ensure that the physical address of the first byte
+         *  of each virtqueue part is a multiple of the specified alignment
+         *  value in the above table.
+         */
+        if (vq->pci_config.queue_desc_lo % 16
+            || vq->pci_config.queue_avail_lo % 2
+            || vq->pci_config.queue_used_lo % 4)
+                bad_driver_vq(vq, "invalid alignment in queue addresses");
+
+        /* Initialize the virtqueue and check they're all in range. */
+        vq->vring.num = vq->pci_config.queue_size;
+        vq->vring.desc = check_pointer(vq->dev,
+                                       vq->pci_config.queue_desc_lo,
+                                       sizeof(*vq->vring.desc) * vq->vring.num);
+        vq->vring.avail = check_pointer(vq->dev,
+                                        vq->pci_config.queue_avail_lo,
+                                        sizeof(*vq->vring.avail)
+                                        + (sizeof(vq->vring.avail->ring[0])
+                                           * vq->vring.num));
+        vq->vring.used = check_pointer(vq->dev,
+                                       vq->pci_config.queue_used_lo,
+                                       sizeof(*vq->vring.used)
+                                       + (sizeof(vq->vring.used->ring[0])
+                                          * vq->vring.num));
+
+        /*
+         * 2.4.9.1:
+         *
+         *   The driver MUST initialize flags in the used ring to 0
+         *   when allocating the used ring.
+         */
+        if (vq->vring.used->flags != 0)
+                bad_driver_vq(vq, "invalid initial used.flags %#x",
+                              vq->vring.used->flags);
+}
+
+static void start_virtqueue(struct virtqueue *vq)
 {
         /*
          * Create stack for thread.  Since the stack grows upwards, we point
          * the stack pointer to the end of this region.
          */
         char *stack = malloc(32768);
-        unsigned long args[] = { LHREQ_EVENTFD,
-                                 vq->config.pfn*getpagesize(), 0 };
 
         /* Create a zero-initialized eventfd. */
         vq->eventfd = eventfd(0, 0);
         if (vq->eventfd < 0)
                 err(1, "Creating eventfd");
-        args[2] = vq->eventfd;
-
-        /*
-         * Attach an eventfd to this virtqueue: it will go off when the Guest
-         * does an LHCALL_NOTIFY for this vq.
-         */
-        if (write(lguest_fd, &args, sizeof(args)) != 0)
-                err(1, "Attaching eventfd");
 
         /*
          * CLONE_VM: because it has to access the Guest memory, and SIGCHLD so
@@ -1048,167 +1859,531 @@ static void create_thread(struct virtqueue *vq)
         vq->thread = clone(do_thread, stack + 32768, CLONE_VM | SIGCHLD, vq);
         if (vq->thread == (pid_t)-1)
                 err(1, "Creating clone");
-
-        /* We close our local copy now the child has it. */
-        close(vq->eventfd);
 }
 
-static void start_device(struct device *dev)
+static void start_virtqueues(struct device *d)
 {
-        unsigned int i;
         struct virtqueue *vq;
 
-        verbose("Device %s OK: offered", dev->name);
-        for (i = 0; i < dev->feature_len; i++)
-                verbose(" %02x", get_feature_bits(dev)[i]);
-        verbose(", accepted");
-        for (i = 0; i < dev->feature_len; i++)
-                verbose(" %02x", get_feature_bits(dev)
-                        [dev->feature_len+i]);
-
-        for (vq = dev->vq; vq; vq = vq->next) {
-                if (vq->service)
-                        create_thread(vq);
+        for (vq = d->vq; vq; vq = vq->next) {
+                if (vq->pci_config.queue_enable)
+                        start_virtqueue(vq);
         }
-        dev->running = true;
 }
 
-static void cleanup_devices(void)
+static void emulate_mmio_write(struct device *d, u32 off, u32 val, u32 mask)
 {
-        struct device *dev;
+        struct virtqueue *vq;
 
-        for (dev = devices.dev; dev; dev = dev->next)
-                reset_device(dev);
+        switch (off) {
+        case offsetof(struct virtio_pci_mmio, cfg.device_feature_select):
+                /*
+                 * 4.1.4.3.1:
+                 *
+                 * The device MUST present the feature bits it is offering in
+                 * device_feature, starting at bit device_feature_select ∗ 32
+                 * for any device_feature_select written by the driver
+                 */
+                if (val == 0)
+                        d->mmio->cfg.device_feature = d->features;
+                else if (val == 1)
+                        d->mmio->cfg.device_feature = (d->features >> 32);
+                else
+                        d->mmio->cfg.device_feature = 0;
+                goto feature_write_through32;
+        case offsetof(struct virtio_pci_mmio, cfg.guest_feature_select):
+                if (val > 1)
+                        bad_driver(d, "Unexpected driver select %u", val);
+                goto feature_write_through32;
+        case offsetof(struct virtio_pci_mmio, cfg.guest_feature):
+                if (d->mmio->cfg.guest_feature_select == 0) {
+                        d->features_accepted &= ~((u64)0xFFFFFFFF);
+                        d->features_accepted |= val;
+                } else {
+                        assert(d->mmio->cfg.guest_feature_select == 1);
+                        d->features_accepted &= 0xFFFFFFFF;
+                        d->features_accepted |= ((u64)val) << 32;
+                }
+                /*
+                 * 2.2.1:
+                 *
+                 *   The driver MUST NOT accept a feature which the device did
+                 *   not offer
+                 */
+                if (d->features_accepted & ~d->features)
+                        bad_driver(d, "over-accepted features %#llx of %#llx",
+                                   d->features_accepted, d->features);
+                goto feature_write_through32;
+        case offsetof(struct virtio_pci_mmio, cfg.device_status): {
+                u8 prev;
+
+                verbose("%s: device status -> %#x\n", d->name, val);
+                /*
+                 * 4.1.4.3.1:
+                 * 
+                 *  The device MUST reset when 0 is written to device_status,
+                 *  and present a 0 in device_status once that is done.
+                 */
+                if (val == 0) {
+                        reset_device(d);
+                        goto write_through8;
+                }
 
-        /* If we saved off the original terminal settings, restore them now. */
-        if (orig_term.c_lflag & (ISIG|ICANON|ECHO))
-                tcsetattr(STDIN_FILENO, TCSANOW, &orig_term);
-}
+                /* 2.1.1: The driver MUST NOT clear a device status bit. */
+                if (d->mmio->cfg.device_status & ~val)
+                        bad_driver(d, "unset of device status bit %#x -> %#x",
+                                   d->mmio->cfg.device_status, val);
 
-/* When the Guest tells us they updated the status field, we handle it. */
-static void update_device_status(struct device *dev)
-{
-        /* A zero status is a reset, otherwise it's a set of flags. */
-        if (dev->desc->status == 0)
-                reset_device(dev);
-        else if (dev->desc->status & VIRTIO_CONFIG_S_FAILED) {
-                warnx("Device %s configuration FAILED", dev->name);
-                if (dev->running)
-                        reset_device(dev);
-        } else {
-                if (dev->running)
-                        err(1, "Device %s features finalized twice", dev->name);
-                start_device(dev);
+                /*
+                 * 2.1.2:
+                 *
+                 *  The device MUST NOT consume buffers or notify the driver
+                 *  before DRIVER_OK.
+                 */
+                if (val & VIRTIO_CONFIG_S_DRIVER_OK
+                    && !(d->mmio->cfg.device_status & VIRTIO_CONFIG_S_DRIVER_OK))
+                        start_virtqueues(d);
+
+                /*
+                 * 3.1.1:
+                 *
+                 *   The driver MUST follow this sequence to initialize a device:
+                 *   - Reset the device.
+                 *   - Set the ACKNOWLEDGE status bit: the guest OS has
+                 *     notice the device.
+                 *   - Set the DRIVER status bit: the guest OS knows how
+                 *     to drive the device.
+                 *   - Read device feature bits, and write the subset
+                 *     of feature bits understood by the OS and driver
+                 *     to the device. During this step the driver MAY
+                 *     read (but MUST NOT write) the device-specific
+                 *     configuration fields to check that it can
+                 *     support the device before accepting it.
+                 *   - Set the FEATURES_OK status bit.  The driver
+                 *     MUST not accept new feature bits after this
+                 *     step.
+                 *   - Re-read device status to ensure the FEATURES_OK
+                 *     bit is still set: otherwise, the device does
+                 *     not support our subset of features and the
+                 *     device is unusable.
+                 *   - Perform device-specific setup, including
+                 *     discovery of virtqueues for the device,
+                 *     optional per-bus setup, reading and possibly
+                 *     writing the device’s virtio configuration
+                 *     space, and population of virtqueues.
+                 *   - Set the DRIVER_OK status bit. At this point the
+                 *     device is “live”.
+                 */
+                prev = 0;
+                switch (val & ~d->mmio->cfg.device_status) {
+                case VIRTIO_CONFIG_S_DRIVER_OK:
+                        prev |= VIRTIO_CONFIG_S_FEATURES_OK; /* fall thru */
+                case VIRTIO_CONFIG_S_FEATURES_OK:
+                        prev |= VIRTIO_CONFIG_S_DRIVER; /* fall thru */
+                case VIRTIO_CONFIG_S_DRIVER:
+                        prev |= VIRTIO_CONFIG_S_ACKNOWLEDGE; /* fall thru */
+                case VIRTIO_CONFIG_S_ACKNOWLEDGE:
+                        break;
+                default:
+                        bad_driver(d, "unknown device status bit %#x -> %#x",
+                                   d->mmio->cfg.device_status, val);
+                }
+                if (d->mmio->cfg.device_status != prev)
+                        bad_driver(d, "unexpected status transition %#x -> %#x",
+                                   d->mmio->cfg.device_status, val);
+
+                /* If they just wrote FEATURES_OK, we make sure they read */
+                switch (val & ~d->mmio->cfg.device_status) {
+                case VIRTIO_CONFIG_S_FEATURES_OK:
+                        d->wrote_features_ok = true;
+                        break;
+                case VIRTIO_CONFIG_S_DRIVER_OK:
+                        if (d->wrote_features_ok)
+                                bad_driver(d, "did not re-read FEATURES_OK");
+                        break;
+                }
+                goto write_through8;
         }
-}
+        case offsetof(struct virtio_pci_mmio, cfg.queue_select):
+                vq = vq_by_num(d, val);
+                /*
+                 * 4.1.4.3.1:
+                 *
+                 *  The device MUST present a 0 in queue_size if the virtqueue
+                 *  corresponding to the current queue_select is unavailable.
+                 */
+                if (!vq) {
+                        d->mmio->cfg.queue_size = 0;
+                        goto write_through16;
+                }
+                /* Save registers for old vq, if it was a valid vq */
+                if (d->mmio->cfg.queue_size)
+                        save_vq_config(&d->mmio->cfg,
+                                       vq_by_num(d, d->mmio->cfg.queue_select));
+                /* Restore the registers for the queue they asked for */
+                restore_vq_config(&d->mmio->cfg, vq);
+                goto write_through16;
+        case offsetof(struct virtio_pci_mmio, cfg.queue_size):
+                /*
+                 * 4.1.4.3.2:
+                 *
+                 *  The driver MUST NOT write a value which is not a power of 2
+                 *  to queue_size.
+                 */
+                if (val & (val-1))
+                        bad_driver(d, "invalid queue size %u", val);
+                if (d->mmio->cfg.queue_enable)
+                        bad_driver(d, "changing queue size on live device");
+                goto write_through16;
+        case offsetof(struct virtio_pci_mmio, cfg.queue_msix_vector):
+                bad_driver(d, "attempt to set MSIX vector to %u", val);
+        case offsetof(struct virtio_pci_mmio, cfg.queue_enable): {
+                struct virtqueue *vq = vq_by_num(d, d->mmio->cfg.queue_select);
 
-/*L:215
- * This is the generic routine we call when the Guest uses LHCALL_NOTIFY.  In
- * particular, it's used to notify us of device status changes during boot.
- */
-static void handle_output(unsigned long addr)
-{
-        struct device *i;
+                /*
+                 * 4.1.4.3.2:
+                 *
+                 *  The driver MUST NOT write a 0 to queue_enable.
+                 */
+                if (val != 1)
+                        bad_driver(d, "setting queue_enable to %u", val);
 
-        /* Check each device. */
-        for (i = devices.dev; i; i = i->next) {
-                struct virtqueue *vq;
+                /*
+                 * 3.1.1:
+                 *
+                 *  7. Perform device-specific setup, including discovery of
+                 *     virtqueues for the device, optional per-bus setup,
+                 *     reading and possibly writing the device’s virtio
+                 *     configuration space, and population of virtqueues.
+                 *  8. Set the DRIVER_OK status bit.
+                 *
+                 * All our devices require all virtqueues to be enabled, so
+                 * they should have done that before setting DRIVER_OK.
+                 */
+                if (d->mmio->cfg.device_status & VIRTIO_CONFIG_S_DRIVER_OK)
+                        bad_driver(d, "enabling vq after DRIVER_OK");
 
+                d->mmio->cfg.queue_enable = val;
+                save_vq_config(&d->mmio->cfg, vq);
+                check_virtqueue(d, vq);
+                goto write_through16;
+        }
+        case offsetof(struct virtio_pci_mmio, cfg.queue_notify_off):
+                bad_driver(d, "attempt to write to queue_notify_off");
+        case offsetof(struct virtio_pci_mmio, cfg.queue_desc_lo):
+        case offsetof(struct virtio_pci_mmio, cfg.queue_desc_hi):
+        case offsetof(struct virtio_pci_mmio, cfg.queue_avail_lo):
+        case offsetof(struct virtio_pci_mmio, cfg.queue_avail_hi):
+        case offsetof(struct virtio_pci_mmio, cfg.queue_used_lo):
+        case offsetof(struct virtio_pci_mmio, cfg.queue_used_hi):
                 /*
-                 * Notifications to device descriptors mean they updated the
-                 * device status.
+                 * 4.1.4.3.2:
+                 *
+                 *  The driver MUST configure the other virtqueue fields before
+                 *  enabling the virtqueue with queue_enable.
                  */
-                if (from_guest_phys(addr) == i->desc) {
-                        update_device_status(i);
-                        return;
-                }
+                if (d->mmio->cfg.queue_enable)
+                        bad_driver(d, "changing queue on live device");
+
+                /*
+                 * 3.1.1:
+                 *
+                 *  The driver MUST follow this sequence to initialize a device:
+                 *...
+                 *  5. Set the FEATURES_OK status bit. The driver MUST not
+                 *  accept new feature bits after this step.
+                 */
+                if (!(d->mmio->cfg.device_status & VIRTIO_CONFIG_S_FEATURES_OK))
+                        bad_driver(d, "setting up vq before FEATURES_OK");
 
-                /* Devices should not be used before features are finalized. */
-                for (vq = i->vq; vq; vq = vq->next) {
-                        if (addr != vq->config.pfn*getpagesize())
-                                continue;
-                        errx(1, "Notification on %s before setup!", i->name);
+                /*
+                 *  6. Re-read device status to ensure the FEATURES_OK bit is
+                 *     still set...
+                 */
+                if (d->wrote_features_ok)
+                        bad_driver(d, "didn't re-read FEATURES_OK before setup");
+
+                goto write_through32;
+        case offsetof(struct virtio_pci_mmio, notify):
+                vq = vq_by_num(d, val);
+                if (!vq)
+                        bad_driver(d, "Invalid vq notification on %u", val);
+                /* Notify the process handling this vq by adding 1 to eventfd */
+                write(vq->eventfd, "\1\0\0\0\0\0\0\0", 8);
+                goto write_through16;
+        case offsetof(struct virtio_pci_mmio, isr):
+                bad_driver(d, "Unexpected write to isr");
+        /* Weird corner case: write to emerg_wr of console */
+        case sizeof(struct virtio_pci_mmio)
+                + offsetof(struct virtio_console_config, emerg_wr):
+                if (strcmp(d->name, "console") == 0) {
+                        char c = val;
+                        write(STDOUT_FILENO, &c, 1);
+                        goto write_through32;
                 }
+                /* Fall through... */
+        default:
+                /*
+                 * 4.1.4.3.2:
+                 *
+                 *   The driver MUST NOT write to device_feature, num_queues,
+                 *   config_generation or queue_notify_off.
+                 */
+                bad_driver(d, "Unexpected write to offset %u", off);
         }
 
+feature_write_through32:
         /*
-         * Early console write is done using notify on a nul-terminated string
-         * in Guest memory.  It's also great for hacking debugging messages
-         * into a Guest.
+         * 3.1.1:
+         *
+         *   The driver MUST follow this sequence to initialize a device:
+         *...
+         *   - Set the DRIVER status bit: the guest OS knows how
+         *     to drive the device.
+         *   - Read device feature bits, and write the subset
+         *     of feature bits understood by the OS and driver
+         *     to the device.
+         *...
+         *   - Set the FEATURES_OK status bit. The driver MUST not
+         *     accept new feature bits after this step.
          */
-        if (addr >= guest_limit)
-                errx(1, "Bad NOTIFY %#lx", addr);
+        if (!(d->mmio->cfg.device_status & VIRTIO_CONFIG_S_DRIVER))
+                bad_driver(d, "feature write before VIRTIO_CONFIG_S_DRIVER");
+        if (d->mmio->cfg.device_status & VIRTIO_CONFIG_S_FEATURES_OK)
+                bad_driver(d, "feature write after VIRTIO_CONFIG_S_FEATURES_OK");
 
-        write(STDOUT_FILENO, from_guest_phys(addr),
-              strnlen(from_guest_phys(addr), guest_limit - addr));
+        /*
+         * 4.1.3.1:
+         *
+         *  The driver MUST access each field using the “natural” access
+         *  method, i.e. 32-bit accesses for 32-bit fields, 16-bit accesses for
+         *  16-bit fields and 8-bit accesses for 8-bit fields.
+         */
+write_through32:
+        if (mask != 0xFFFFFFFF) {
+                bad_driver(d, "non-32-bit write to offset %u (%#x)",
+                           off, getreg(eip));
+                return;
+        }
+        memcpy((char *)d->mmio + off, &val, 4);
+        return;
+
+write_through16:
+        if (mask != 0xFFFF)
+                bad_driver(d, "non-16-bit write to offset %u (%#x)",
+                           off, getreg(eip));
+        memcpy((char *)d->mmio + off, &val, 2);
+        return;
+
+write_through8:
+        if (mask != 0xFF)
+                bad_driver(d, "non-8-bit write to offset %u (%#x)",
+                           off, getreg(eip));
+        memcpy((char *)d->mmio + off, &val, 1);
+        return;
 }
 
-/*L:190
- * Device Setup
- *
- * All devices need a descriptor so the Guest knows it exists, and a "struct
- * device" so the Launcher can keep track of it.  We have common helper
- * routines to allocate and manage them.
- */
-
-/*
- * The layout of the device page is a "struct lguest_device_desc" followed by a
- * number of virtqueue descriptors, then two sets of feature bits, then an
- * array of configuration bytes.  This routine returns the configuration
- * pointer.
- */
-static u8 *device_config(const struct device *dev)
+static u32 emulate_mmio_read(struct device *d, u32 off, u32 mask)
 {
-        return (void *)(dev->desc + 1)
-                + dev->num_vq * sizeof(struct lguest_vqconfig)
-                + dev->feature_len * 2;
+        u8 isr;
+        u32 val = 0;
+
+        switch (off) {
+        case offsetof(struct virtio_pci_mmio, cfg.device_feature_select):
+        case offsetof(struct virtio_pci_mmio, cfg.device_feature):
+        case offsetof(struct virtio_pci_mmio, cfg.guest_feature_select):
+        case offsetof(struct virtio_pci_mmio, cfg.guest_feature):
+                /*
+                 * 3.1.1:
+                 *
+                 *   The driver MUST follow this sequence to initialize a device:
+                 *...
+                 *   - Set the DRIVER status bit: the guest OS knows how
+                 *     to drive the device.
+                 *   - Read device feature bits, and write the subset
+                 *     of feature bits understood by the OS and driver
+                 *     to the device.
+                 */
+                if (!(d->mmio->cfg.device_status & VIRTIO_CONFIG_S_DRIVER))
+                        bad_driver(d,
+                                   "feature read before VIRTIO_CONFIG_S_DRIVER");
+                goto read_through32;
+        case offsetof(struct virtio_pci_mmio, cfg.msix_config):
+                bad_driver(d, "read of msix_config");
+        case offsetof(struct virtio_pci_mmio, cfg.num_queues):
+                goto read_through16;
+        case offsetof(struct virtio_pci_mmio, cfg.device_status):
+                /* As they did read, any write of FEATURES_OK is now fine. */
+                d->wrote_features_ok = false;
+                goto read_through8;
+        case offsetof(struct virtio_pci_mmio, cfg.config_generation):
+                /*
+                 * 4.1.4.3.1:
+                 *
+                 *  The device MUST present a changed config_generation after
+                 *  the driver has read a device-specific configuration value
+                 *  which has changed since any part of the device-specific
+                 *  configuration was last read.
+                 *
+                 * This is simple: none of our devices change config, so this
+                 * is always 0.
+                 */
+                goto read_through8;
+        case offsetof(struct virtio_pci_mmio, notify):
+                /*
+                 * 3.1.1:
+                 *
+                 *   The driver MUST NOT notify the device before setting
+                 *   DRIVER_OK.
+                 */
+                if (!(d->mmio->cfg.device_status & VIRTIO_CONFIG_S_DRIVER_OK))
+                        bad_driver(d, "notify before VIRTIO_CONFIG_S_DRIVER_OK");
+                goto read_through16;
+        case offsetof(struct virtio_pci_mmio, isr):
+                if (mask != 0xFF)
+                        bad_driver(d, "non-8-bit read from offset %u (%#x)",
+                                   off, getreg(eip));
+                isr = d->mmio->isr;
+                /*
+                 * 4.1.4.5.1:
+                 *
+                 *  The device MUST reset ISR status to 0 on driver read. 
+                 */
+                d->mmio->isr = 0;
+                return isr;
+        case offsetof(struct virtio_pci_mmio, padding):
+                bad_driver(d, "read from padding (%#x)", getreg(eip));
+        default:
+                /* Read from device config space, beware unaligned overflow */
+                if (off > d->mmio_size - 4)
+                        bad_driver(d, "read past end (%#x)", getreg(eip));
+
+                /*
+                 * 3.1.1:
+                 *  The driver MUST follow this sequence to initialize a device:
+                 *...
+                 *  3. Set the DRIVER status bit: the guest OS knows how to
+                 *  drive the device.
+                 *  4. Read device feature bits, and write the subset of
+                 *  feature bits understood by the OS and driver to the
+                 *  device. During this step the driver MAY read (but MUST NOT
+                 *  write) the device-specific configuration fields to check
+                 *  that it can support the device before accepting it.
+                 */
+                if (!(d->mmio->cfg.device_status & VIRTIO_CONFIG_S_DRIVER))
+                        bad_driver(d,
+                                   "config read before VIRTIO_CONFIG_S_DRIVER");
+
+                if (mask == 0xFFFFFFFF)
+                        goto read_through32;
+                else if (mask == 0xFFFF)
+                        goto read_through16;
+                else
+                        goto read_through8;
+        }
+
+        /*
+         * 4.1.3.1:
+         *
+         *  The driver MUST access each field using the “natural” access
+         *  method, i.e. 32-bit accesses for 32-bit fields, 16-bit accesses for
+         *  16-bit fields and 8-bit accesses for 8-bit fields.
+         */
+read_through32:
+        if (mask != 0xFFFFFFFF)
+                bad_driver(d, "non-32-bit read to offset %u (%#x)",
+                           off, getreg(eip));
+        memcpy(&val, (char *)d->mmio + off, 4);
+        return val;
+
+read_through16:
+        if (mask != 0xFFFF)
+                bad_driver(d, "non-16-bit read to offset %u (%#x)",
+                           off, getreg(eip));
+        memcpy(&val, (char *)d->mmio + off, 2);
+        return val;
+
+read_through8:
+        if (mask != 0xFF)
+                bad_driver(d, "non-8-bit read to offset %u (%#x)",
+                           off, getreg(eip));
+        memcpy(&val, (char *)d->mmio + off, 1);
+        return val;
 }
 
-/*
- * This routine allocates a new "struct lguest_device_desc" from descriptor
- * table page just above the Guest's normal memory.  It returns a pointer to
- * that descriptor.
- */
-static struct lguest_device_desc *new_dev_desc(u16 type)
+static void emulate_mmio(unsigned long paddr, const u8 *insn)
 {
-        struct lguest_device_desc d = { .type = type };
-        void *p;
+        u32 val, off, mask = 0xFFFFFFFF, insnlen = 0;
+        struct device *d = find_mmio_region(paddr, &off);
+        unsigned long args[] = { LHREQ_TRAP, 14 };
 
-        /* Figure out where the next device config is, based on the last one. */
-        if (devices.lastdev)
-                p = device_config(devices.lastdev)
-                        + devices.lastdev->desc->config_len;
-        else
-                p = devices.descpage;
+        if (!d) {
+                warnx("MMIO touching %#08lx (not a device)", paddr);
+                goto reinject;
+        }
+
+        /* Prefix makes it a 16 bit op */
+        if (insn[0] == 0x66) {
+                mask = 0xFFFF;
+                insnlen++;
+        }
 
-        /* We only have one page for all the descriptors. */
-        if (p + sizeof(d) > (void *)devices.descpage + getpagesize())
-                errx(1, "Too many devices");
+        /* iowrite */
+        if (insn[insnlen] == 0x89) {
+                /* Next byte is r/m byte: bits 3-5 are register. */
+                val = getreg_num((insn[insnlen+1] >> 3) & 0x7, mask);
+                emulate_mmio_write(d, off, val, mask);
+                insnlen += 2 + insn_displacement_len(insn[insnlen+1]);
+        } else if (insn[insnlen] == 0x8b) { /* ioread */
+                /* Next byte is r/m byte: bits 3-5 are register. */
+                val = emulate_mmio_read(d, off, mask);
+                setreg_num((insn[insnlen+1] >> 3) & 0x7, val, mask);
+                insnlen += 2 + insn_displacement_len(insn[insnlen+1]);
+        } else if (insn[0] == 0x88) { /* 8-bit iowrite */
+                mask = 0xff;
+                /* Next byte is r/m byte: bits 3-5 are register. */
+                val = getreg_num((insn[1] >> 3) & 0x7, mask);
+                emulate_mmio_write(d, off, val, mask);
+                insnlen = 2 + insn_displacement_len(insn[1]);
+        } else if (insn[0] == 0x8a) { /* 8-bit ioread */
+                mask = 0xff;
+                val = emulate_mmio_read(d, off, mask);
+                setreg_num((insn[1] >> 3) & 0x7, val, mask);
+                insnlen = 2 + insn_displacement_len(insn[1]);
+        } else {
+                warnx("Unknown MMIO instruction touching %#08lx:"
+                     " %02x %02x %02x %02x at %u",
+                     paddr, insn[0], insn[1], insn[2], insn[3], getreg(eip));
+        reinject:
+                /* Inject trap into Guest. */
+                if (write(lguest_fd, args, sizeof(args)) < 0)
+                        err(1, "Reinjecting trap 14 for fault at %#x",
+                            getreg(eip));
+                return;
+        }
 
-        /* p might not be aligned, so we memcpy in. */
-        return memcpy(p, &d, sizeof(d));
+        /* Finally, we've "done" the instruction, so move past it. */
+        setreg(eip, getreg(eip) + insnlen);
 }
 
-/*
- * Each device descriptor is followed by the description of its virtqueues.  We
- * specify how many descriptors the virtqueue is to have.
+/*L:190
+ * Device Setup
+ *
+ * All devices need a descriptor so the Guest knows it exists, and a "struct
+ * device" so the Launcher can keep track of it.  We have common helper
+ * routines to allocate and manage them.
  */
-static void add_virtqueue(struct device *dev, unsigned int num_descs,
-                          void (*service)(struct virtqueue *))
+static void add_pci_virtqueue(struct device *dev,
+                              void (*service)(struct virtqueue *),
+                              const char *name)
 {
-        unsigned int pages;
         struct virtqueue **i, *vq = malloc(sizeof(*vq));
-        void *p;
-
-        /* First we need some memory for this virtqueue. */
-        pages = (vring_size(num_descs, LGUEST_VRING_ALIGN) + getpagesize() - 1)
-                / getpagesize();
-        p = get_pages(pages);
 
         /* Initialize the virtqueue */
         vq->next = NULL;
         vq->last_avail_idx = 0;
         vq->dev = dev;
+        vq->name = name;
 
         /*
          * This is the routine the service thread will run, and its Process ID
@@ -1218,25 +2393,11 @@ static void add_virtqueue(struct device *dev, unsigned int num_descs,
         vq->thread = (pid_t)-1;
 
         /* Initialize the configuration. */
-        vq->config.num = num_descs;
-        vq->config.irq = devices.next_irq++;
-        vq->config.pfn = to_guest_phys(p) / getpagesize();
-
-        /* Initialize the vring. */
-        vring_init(&vq->vring, num_descs, p, LGUEST_VRING_ALIGN);
-
-        /*
-         * Append virtqueue to this device's descriptor.  We use
-         * device_config() to get the end of the device's current virtqueues;
-         * we check that we haven't added any config or feature information
-         * yet, otherwise we'd be overwriting them.
-         */
-        assert(dev->desc->config_len == 0 && dev->desc->feature_len == 0);
-        memcpy(device_config(dev), &vq->config, sizeof(vq->config));
-        dev->num_vq++;
-        dev->desc->num_vq++;
+        reset_vq_pci_config(vq);
+        vq->pci_config.queue_notify_off = 0;
 
-        verbose("Virtqueue page %#lx\n", to_guest_phys(p));
+        /* Add one to the number of queues */
+        vq->dev->mmio->cfg.num_queues++;
 
         /*
          * Add to tail of list, so dev->vq is first vq, dev->vq->next is
@@ -1246,73 +2407,239 @@ static void add_virtqueue(struct device *dev, unsigned int num_descs,
         *i = vq;
 }
 
-/*
- * The first half of the feature bitmask is for us to advertise features.  The
- * second half is for the Guest to accept features.
- */
-static void add_feature(struct device *dev, unsigned bit)
+/* The Guest accesses the feature bits via the PCI common config MMIO region */
+static void add_pci_feature(struct device *dev, unsigned bit)
 {
-        u8 *features = get_feature_bits(dev);
+        dev->features |= (1ULL << bit);
+}
 
-        /* We can't extend the feature bits once we've added config bytes */
-        if (dev->desc->feature_len <= bit / CHAR_BIT) {
-                assert(dev->desc->config_len == 0);
-                dev->feature_len = dev->desc->feature_len = (bit/CHAR_BIT) + 1;
-        }
+/* For devices with no config. */
+static void no_device_config(struct device *dev)
+{
+        dev->mmio_addr = get_mmio_region(dev->mmio_size);
 
-        features[bit / CHAR_BIT] |= (1 << (bit % CHAR_BIT));
+        dev->config.bar[0] = dev->mmio_addr;
+        /* Bottom 4 bits must be zero */
+        assert(~(dev->config.bar[0] & 0xF));
+}
+
+/* This puts the device config into BAR0 */
+static void set_device_config(struct device *dev, const void *conf, size_t len)
+{
+        /* Set up BAR 0 */
+        dev->mmio_size += len;
+        dev->mmio = realloc(dev->mmio, dev->mmio_size);
+        memcpy(dev->mmio + 1, conf, len);
+
+        /*
+         * 4.1.4.6:
+         *
+         *  The device MUST present at least one VIRTIO_PCI_CAP_DEVICE_CFG
+         *  capability for any device type which has a device-specific
+         *  configuration.
+         */
+        /* Hook up device cfg */
+        dev->config.cfg_access.cap.cap_next
+                = offsetof(struct pci_config, device);
+
+        /*
+         * 4.1.4.6.1:
+         *
+         *  The offset for the device-specific configuration MUST be 4-byte
+         *  aligned.
+         */
+        assert(dev->config.cfg_access.cap.cap_next % 4 == 0);
+
+        /* Fix up device cfg field length. */
+        dev->config.device.length = len;
+
+        /* The rest is the same as the no-config case */
+        no_device_config(dev);
+}
+
+static void init_cap(struct virtio_pci_cap *cap, size_t caplen, int type,
+                     size_t bar_offset, size_t bar_bytes, u8 next)
+{
+        cap->cap_vndr = PCI_CAP_ID_VNDR;
+        cap->cap_next = next;
+        cap->cap_len = caplen;
+        cap->cfg_type = type;
+        cap->bar = 0;
+        memset(cap->padding, 0, sizeof(cap->padding));
+        cap->offset = bar_offset;
+        cap->length = bar_bytes;
 }
 
 /*
- * This routine sets the configuration fields for an existing device's
- * descriptor.  It only works for the last device, but that's OK because that's
- * how we use it.
+ * This sets up the pci_config structure, as defined in the virtio 1.0
+ * standard (and PCI standard).
  */
-static void set_config(struct device *dev, unsigned len, const void *conf)
+static void init_pci_config(struct pci_config *pci, u16 type,
+                            u8 class, u8 subclass)
 {
-        /* Check we haven't overflowed our single page. */
-        if (device_config(dev) + len > devices.descpage + getpagesize())
-                errx(1, "Too many devices");
+        size_t bar_offset, bar_len;
+
+        /*
+         * 4.1.4.4.1:
+         *
+         *  The device MUST either present notify_off_multiplier as an even
+         *  power of 2, or present notify_off_multiplier as 0.
+         *
+         * 2.1.2:
+         *
+         *   The device MUST initialize device status to 0 upon reset. 
+         */
+        memset(pci, 0, sizeof(*pci));
+
+        /* 4.1.2.1: Devices MUST have the PCI Vendor ID 0x1AF4 */
+        pci->vendor_id = 0x1AF4;
+        /* 4.1.2.1: ... PCI Device ID calculated by adding 0x1040 ... */
+        pci->device_id = 0x1040 + type;
+
+        /*
+         * PCI have specific codes for different types of devices.
+         * Linux doesn't care, but it's a good clue for people looking
+         * at the device.
+         */
+        pci->class = class;
+        pci->subclass = subclass;
+
+        /*
+         * 4.1.2.1:
+         *
+         *  Non-transitional devices SHOULD have a PCI Revision ID of 1 or
+         *  higher
+         */
+        pci->revid = 1;
+
+        /*
+         * 4.1.2.1:
+         *
+         *  Non-transitional devices SHOULD have a PCI Subsystem Device ID of
+         *  0x40 or higher.
+         */
+        pci->subsystem_device_id = 0x40;
+
+        /* We use our dummy interrupt controller, and irq_line is the irq */
+        pci->irq_line = devices.next_irq++;
+        pci->irq_pin = 0;
+
+        /* Support for extended capabilities. */
+        pci->status = (1 << 4);
+
+        /* Link them in. */
+        /*
+         * 4.1.4.3.1:
+         *
+         *  The device MUST present at least one common configuration
+         *  capability.
+         */
+        pci->capabilities = offsetof(struct pci_config, common);
+
+        /* 4.1.4.3.1 ... offset MUST be 4-byte aligned. */
+        assert(pci->capabilities % 4 == 0);
+
+        bar_offset = offsetof(struct virtio_pci_mmio, cfg);
+        bar_len = sizeof(((struct virtio_pci_mmio *)0)->cfg);
+        init_cap(&pci->common, sizeof(pci->common), VIRTIO_PCI_CAP_COMMON_CFG,
+                 bar_offset, bar_len,
+                 offsetof(struct pci_config, notify));
+
+        /*
+         * 4.1.4.4.1:
+         *
+         *  The device MUST present at least one notification capability.
+         */
+        bar_offset += bar_len;
+        bar_len = sizeof(((struct virtio_pci_mmio *)0)->notify);
+
+        /*
+         * 4.1.4.4.1:
+         *
+         *  The cap.offset MUST be 2-byte aligned.
+         */
+        assert(pci->common.cap_next % 2 == 0);
+
+        /* FIXME: Use a non-zero notify_off, for per-queue notification? */
+        /*
+         * 4.1.4.4.1:
+         *
+         *  The value cap.length presented by the device MUST be at least 2 and
+         *  MUST be large enough to support queue notification offsets for all
+         *  supported queues in all possible configurations.
+         */
+        assert(bar_len >= 2);
+
+        init_cap(&pci->notify.cap, sizeof(pci->notify),
+                 VIRTIO_PCI_CAP_NOTIFY_CFG,
+                 bar_offset, bar_len,
+                 offsetof(struct pci_config, isr));
+
+        bar_offset += bar_len;
+        bar_len = sizeof(((struct virtio_pci_mmio *)0)->isr);
+        /*
+         * 4.1.4.5.1:
+         *
+         *  The device MUST present at least one VIRTIO_PCI_CAP_ISR_CFG
+         *  capability.
+         */
+        init_cap(&pci->isr, sizeof(pci->isr),
+                 VIRTIO_PCI_CAP_ISR_CFG,
+                 bar_offset, bar_len,
+                 offsetof(struct pci_config, cfg_access));
+
+        /*
+         * 4.1.4.7.1:
+         *
+         * The device MUST present at least one VIRTIO_PCI_CAP_PCI_CFG
+         * capability.
+         */
+        /* This doesn't have any presence in the BAR */
+        init_cap(&pci->cfg_access.cap, sizeof(pci->cfg_access),
+                 VIRTIO_PCI_CAP_PCI_CFG,
+                 0, 0, 0);
 
-        /* Copy in the config information, and store the length. */
-        memcpy(device_config(dev), conf, len);
-        dev->desc->config_len = len;
+        bar_offset += bar_len + sizeof(((struct virtio_pci_mmio *)0)->padding);
+        assert(bar_offset == sizeof(struct virtio_pci_mmio));
 
-        /* Size must fit in config_len field (8 bits)! */
-        assert(dev->desc->config_len == len);
+        /*
+         * This gets sewn in and length set in set_device_config().
+         * Some devices don't have a device configuration interface, so
+         * we never expose this if we don't call set_device_config().
+         */
+        init_cap(&pci->device, sizeof(pci->device), VIRTIO_PCI_CAP_DEVICE_CFG,
+                 bar_offset, 0, 0);
 }
 
 /*
- * This routine does all the creation and setup of a new device, including
- * calling new_dev_desc() to allocate the descriptor and device memory.  We
- * don't actually start the service threads until later.
+ * This routine does all the creation and setup of a new device, but we don't
+ * actually place the MMIO region until we know the size (if any) of the
+ * device-specific config.  And we don't actually start the service threads
+ * until later.
  *
  * See what I mean about userspace being boring?
  */
-static struct device *new_device(const char *name, u16 type)
+static struct device *new_pci_device(const char *name, u16 type,
+                                     u8 class, u8 subclass)
 {
         struct device *dev = malloc(sizeof(*dev));
 
         /* Now we populate the fields one at a time. */
-        dev->desc = new_dev_desc(type);
         dev->name = name;
         dev->vq = NULL;
-        dev->feature_len = 0;
-        dev->num_vq = 0;
         dev->running = false;
-        dev->next = NULL;
+        dev->wrote_features_ok = false;
+        dev->mmio_size = sizeof(struct virtio_pci_mmio);
+        dev->mmio = calloc(1, dev->mmio_size);
+        dev->features = (u64)1 << VIRTIO_F_VERSION_1;
+        dev->features_accepted = 0;
 
-        /*
-         * Append to device list.  Prepending to a single-linked list is
-         * easier, but the user expects the devices to be arranged on the bus
-         * in command-line order.  The first network device on the command line
-         * is eth0, the first block device /dev/vda, etc.
-         */
-        if (devices.lastdev)
-                devices.lastdev->next = dev;
-        else
-                devices.dev = dev;
-        devices.lastdev = dev;
+        if (devices.device_num + 1 >= MAX_PCI_DEVICES)
+                errx(1, "Can only handle 31 PCI devices");
+
+        init_pci_config(&dev->config, type, class, subclass);
+        assert(!devices.pci[devices.device_num+1]);
+        devices.pci[++devices.device_num] = dev;
 
         return dev;
 }
@@ -1324,6 +2651,7 @@ static struct device *new_device(const char *name, u16 type)
 static void setup_console(void)
 {
         struct device *dev;
+        struct virtio_console_config conf;
 
         /* If we can save the initial standard input settings... */
         if (tcgetattr(STDIN_FILENO, &orig_term) == 0) {
@@ -1336,7 +2664,7 @@ static void setup_console(void)
                 tcsetattr(STDIN_FILENO, TCSANOW, &term);
         }
 
-        dev = new_device("console", VIRTIO_ID_CONSOLE);
+        dev = new_pci_device("console", VIRTIO_ID_CONSOLE, 0x07, 0x00);
 
         /* We store the console state in dev->priv, and initialize it. */
         dev->priv = malloc(sizeof(struct console_abort));
@@ -1348,10 +2676,14 @@ static void setup_console(void)
          * stdin.  When they put something in the output queue, we write it to
          * stdout.
          */
-        add_virtqueue(dev, VIRTQUEUE_NUM, console_input);
-        add_virtqueue(dev, VIRTQUEUE_NUM, console_output);
+        add_pci_virtqueue(dev, console_input, "input");
+        add_pci_virtqueue(dev, console_output, "output");
+
+        /* We need a configuration area for the emerg_wr early writes. */
+        add_pci_feature(dev, VIRTIO_CONSOLE_F_EMERG_WRITE);
+        set_device_config(dev, &conf, sizeof(conf));
 
-        verbose("device %u: console\n", ++devices.device_num);
+        verbose("device %u: console\n", devices.device_num);
 }
 /*:*/
 
@@ -1449,6 +2781,7 @@ static void configure_device(int fd, const char *tapif, u32 ipaddr)
 static int get_tun_device(char tapif[IFNAMSIZ])
 {
         struct ifreq ifr;
+        int vnet_hdr_sz;
         int netfd;
 
         /* Start with this zeroed.  Messy but sure. */
@@ -1476,6 +2809,18 @@ static int get_tun_device(char tapif[IFNAMSIZ])
          */
         ioctl(netfd, TUNSETNOCSUM, 1);
 
+        /*
+         * In virtio before 1.0 (aka legacy virtio), we added a 16-bit
+         * field at the end of the network header iff
+         * VIRTIO_NET_F_MRG_RXBUF was negotiated.  For virtio 1.0,
+         * that became the norm, but we need to tell the tun device
+         * about our expanded header (which is called
+         * virtio_net_hdr_mrg_rxbuf in the legacy system).
+         */
+        vnet_hdr_sz = sizeof(struct virtio_net_hdr_v1);
+        if (ioctl(netfd, TUNSETVNETHDRSZ, &vnet_hdr_sz) != 0)
+                err(1, "Setting tun header size to %u", vnet_hdr_sz);
+
         memcpy(tapif, ifr.ifr_name, IFNAMSIZ);
         return netfd;
 }
@@ -1499,12 +2844,12 @@ static void setup_tun_net(char *arg)
         net_info->tunfd = get_tun_device(tapif);
 
         /* First we create a new network device. */
-        dev = new_device("net", VIRTIO_ID_NET);
+        dev = new_pci_device("net", VIRTIO_ID_NET, 0x02, 0x00);
         dev->priv = net_info;
 
         /* Network devices need a recv and a send queue, just like console. */
-        add_virtqueue(dev, VIRTQUEUE_NUM, net_input);
-        add_virtqueue(dev, VIRTQUEUE_NUM, net_output);
+        add_pci_virtqueue(dev, net_input, "rx");
+        add_pci_virtqueue(dev, net_output, "tx");
 
         /*
          * We need a socket to perform the magic network ioctls to bring up the
@@ -1524,7 +2869,7 @@ static void setup_tun_net(char *arg)
         p = strchr(arg, ':');
         if (p) {
                 str2mac(p+1, conf.mac);
-                add_feature(dev, VIRTIO_NET_F_MAC);
+                add_pci_feature(dev, VIRTIO_NET_F_MAC);
                 *p = '\0';
         }
 
@@ -1538,25 +2883,21 @@ static void setup_tun_net(char *arg)
         configure_device(ipfd, tapif, ip);
 
         /* Expect Guest to handle everything except UFO */
-        add_feature(dev, VIRTIO_NET_F_CSUM);
-        add_feature(dev, VIRTIO_NET_F_GUEST_CSUM);
-        add_feature(dev, VIRTIO_NET_F_GUEST_TSO4);
-        add_feature(dev, VIRTIO_NET_F_GUEST_TSO6);
-        add_feature(dev, VIRTIO_NET_F_GUEST_ECN);
-        add_feature(dev, VIRTIO_NET_F_HOST_TSO4);
-        add_feature(dev, VIRTIO_NET_F_HOST_TSO6);
-        add_feature(dev, VIRTIO_NET_F_HOST_ECN);
+        add_pci_feature(dev, VIRTIO_NET_F_CSUM);
+        add_pci_feature(dev, VIRTIO_NET_F_GUEST_CSUM);
+        add_pci_feature(dev, VIRTIO_NET_F_GUEST_TSO4);
+        add_pci_feature(dev, VIRTIO_NET_F_GUEST_TSO6);
+        add_pci_feature(dev, VIRTIO_NET_F_GUEST_ECN);
+        add_pci_feature(dev, VIRTIO_NET_F_HOST_TSO4);
+        add_pci_feature(dev, VIRTIO_NET_F_HOST_TSO6);
+        add_pci_feature(dev, VIRTIO_NET_F_HOST_ECN);
         /* We handle indirect ring entries */
-        add_feature(dev, VIRTIO_RING_F_INDIRECT_DESC);
-        /* We're compliant with the damn spec. */
-        add_feature(dev, VIRTIO_F_ANY_LAYOUT);
-        set_config(dev, sizeof(conf), &conf);
+        add_pci_feature(dev, VIRTIO_RING_F_INDIRECT_DESC);
+        set_device_config(dev, &conf, sizeof(conf));
 
         /* We don't need the socket any more; setup is done. */
         close(ipfd);
 
-        devices.device_num++;
-
         if (bridging)
                 verbose("device %u: tun %s attached to bridge: %s\n",
                         devices.device_num, tapif, arg);
@@ -1607,7 +2948,7 @@ static void blk_request(struct virtqueue *vq)
         head = wait_for_vq_desc(vq, iov, &out_num, &in_num);
 
         /* Copy the output header from the front of the iov (adjusts iov) */
-        iov_consume(iov, out_num, &out, sizeof(out));
+        iov_consume(vq->dev, iov, out_num, &out, sizeof(out));
 
         /* Find and trim end of iov input array, for our status byte. */
         in = NULL;
@@ -1619,7 +2960,7 @@ static void blk_request(struct virtqueue *vq)
                 }
         }
         if (!in)
-                errx(1, "Bad virtblk cmd with no room for status");
+                bad_driver_vq(vq, "Bad virtblk cmd with no room for status");
 
         /*
          * For historical reasons, block operations are expressed in 512 byte
@@ -1627,15 +2968,7 @@ static void blk_request(struct virtqueue *vq)
          */
         off = out.sector * 512;
 
-        /*
-         * In general the virtio block driver is allowed to try SCSI commands.
-         * It'd be nice if we supported eject, for example, but we don't.
-         */
-        if (out.type & VIRTIO_BLK_T_SCSI_CMD) {
-                fprintf(stderr, "Scsi commands unsupported\n");
-                *in = VIRTIO_BLK_S_UNSUPP;
-                wlen = sizeof(*in);
-        } else if (out.type & VIRTIO_BLK_T_OUT) {
+        if (out.type & VIRTIO_BLK_T_OUT) {
                 /*
                  * Write
                  *
@@ -1657,7 +2990,7 @@ static void blk_request(struct virtqueue *vq)
                         /* Trim it back to the correct length */
                         ftruncate64(vblk->fd, vblk->len);
                         /* Die, bad Guest, die. */
-                        errx(1, "Write past end %llu+%u", off, ret);
+                        bad_driver_vq(vq, "Write past end %llu+%u", off, ret);
                 }
 
                 wlen = sizeof(*in);
@@ -1699,11 +3032,11 @@ static void setup_block_file(const char *filename)
         struct vblk_info *vblk;
         struct virtio_blk_config conf;
 
-        /* Creat the device. */
-        dev = new_device("block", VIRTIO_ID_BLOCK);
+        /* Create the device. */
+        dev = new_pci_device("block", VIRTIO_ID_BLOCK, 0x01, 0x80);
 
         /* The device has one virtqueue, where the Guest places requests. */
-        add_virtqueue(dev, VIRTQUEUE_NUM, blk_request);
+        add_pci_virtqueue(dev, blk_request, "request");
 
         /* Allocate the room for our own bookkeeping */
         vblk = dev->priv = malloc(sizeof(*vblk));
@@ -1712,9 +3045,6 @@ static void setup_block_file(const char *filename)
         vblk->fd = open_or_die(filename, O_RDWR|O_LARGEFILE);
         vblk->len = lseek64(vblk->fd, 0, SEEK_END);
 
-        /* We support FLUSH. */
-        add_feature(dev, VIRTIO_BLK_F_FLUSH);
-
         /* Tell Guest how many sectors this device has. */
         conf.capacity = cpu_to_le64(vblk->len / 512);
 
@@ -1722,20 +3052,19 @@ static void setup_block_file(const char *filename)
          * Tell Guest not to put in too many descriptors at once: two are used
          * for the in and out elements.
          */
-        add_feature(dev, VIRTIO_BLK_F_SEG_MAX);
+        add_pci_feature(dev, VIRTIO_BLK_F_SEG_MAX);
         conf.seg_max = cpu_to_le32(VIRTQUEUE_NUM - 2);
 
-        /* Don't try to put whole struct: we have 8 bit limit. */
-        set_config(dev, offsetof(struct virtio_blk_config, geometry), &conf);
+        set_device_config(dev, &conf, sizeof(struct virtio_blk_config));
 
         verbose("device %u: virtblock %llu sectors\n",
-                ++devices.device_num, le64_to_cpu(conf.capacity));
+                devices.device_num, le64_to_cpu(conf.capacity));
 }
 
 /*L:211
- * Our random number generator device reads from /dev/random into the Guest's
+ * Our random number generator device reads from /dev/urandom into the Guest's
  * input buffers.  The usual case is that the Guest doesn't want random numbers
- * and so has no buffers although /dev/random is still readable, whereas
+ * and so has no buffers although /dev/urandom is still readable, whereas
  * console is the reverse.
  *
  * The same logic applies, however.
@@ -1754,7 +3083,7 @@ static void rng_input(struct virtqueue *vq)
         /* First we need a buffer from the Guests's virtqueue. */
         head = wait_for_vq_desc(vq, iov, &out_num, &in_num);
         if (out_num)
-                errx(1, "Output buffers in rng?");
+                bad_driver_vq(vq, "Output buffers in rng?");
 
         /*
          * Just like the console write, we loop to cover the whole iovec.
@@ -1763,8 +3092,8 @@ static void rng_input(struct virtqueue *vq)
         while (!iov_empty(iov, in_num)) {
                 len = readv(rng_info->rfd, iov, in_num);
                 if (len <= 0)
-                        err(1, "Read from /dev/random gave %i", len);
-                iov_consume(iov, in_num, NULL, len);
+                        err(1, "Read from /dev/urandom gave %i", len);
+                iov_consume(vq->dev, iov, in_num, NULL, len);
                 totlen += len;
         }
 
@@ -1780,17 +3109,20 @@ static void setup_rng(void)
         struct device *dev;
         struct rng_info *rng_info = malloc(sizeof(*rng_info));
 
-        /* Our device's privat info simply contains the /dev/random fd. */
-        rng_info->rfd = open_or_die("/dev/random", O_RDONLY);
+        /* Our device's private info simply contains the /dev/urandom fd. */
+        rng_info->rfd = open_or_die("/dev/urandom", O_RDONLY);
 
         /* Create the new device. */
-        dev = new_device("rng", VIRTIO_ID_RNG);
+        dev = new_pci_device("rng", VIRTIO_ID_RNG, 0xff, 0);
         dev->priv = rng_info;
 
         /* The device has one virtqueue, where the Guest places inbufs. */
-        add_virtqueue(dev, VIRTQUEUE_NUM, rng_input);
+        add_pci_virtqueue(dev, rng_input, "input");
 
-        verbose("device %u: rng\n", devices.device_num++);
+        /* We don't have any configuration space */
+        no_device_config(dev);
+
+        verbose("device %u: rng\n", devices.device_num);
 }
 /* That's the end of device setup. */
 
@@ -1820,17 +3152,23 @@ static void __attribute__((noreturn)) restart_guest(void)
 static void __attribute__((noreturn)) run_guest(void)
 {
         for (;;) {
-                unsigned long notify_addr;
+                struct lguest_pending notify;
                 int readval;
 
                 /* We read from the /dev/lguest device to run the Guest. */
-                readval = pread(lguest_fd, &notify_addr,
-                                sizeof(notify_addr), cpu_id);
-
-                /* One unsigned long means the Guest did HCALL_NOTIFY */
-                if (readval == sizeof(notify_addr)) {
-                        verbose("Notify on address %#lx\n", notify_addr);
-                        handle_output(notify_addr);
+                readval = pread(lguest_fd, &notify, sizeof(notify), cpu_id);
+                if (readval == sizeof(notify)) {
+                        if (notify.trap == 13) {
+                                verbose("Emulating instruction at %#x\n",
+                                        getreg(eip));
+                                emulate_insn(notify.insn);
+                        } else if (notify.trap == 14) {
+                                verbose("Emulating MMIO at %#x\n",
+                                        getreg(eip));
+                                emulate_mmio(notify.addr, notify.insn);
+                        } else
+                                errx(1, "Unknown trap %i addr %#08x\n",
+                                     notify.trap, notify.addr);
                 /* ENOENT means the Guest died.  Reading tells us why. */
                 } else if (errno == ENOENT) {
                         char reason[1024] = { 0 };
@@ -1893,11 +3231,9 @@ int main(int argc, char *argv[])
         main_args = argv;
 
         /*
-         * First we initialize the device list.  We keep a pointer to the last
-         * device, and the next interrupt number to use for devices (1:
-         * remember that 0 is used by the timer).
+         * First we initialize the device list.  We remember next interrupt
+         * number to use for devices (1: remember that 0 is used by the timer).
          */
-        devices.lastdev = NULL;
         devices.next_irq = 1;
 
         /* We're CPU 0.  In fact, that's the only CPU possible right now. */
@@ -1921,12 +3257,14 @@ int main(int argc, char *argv[])
                         guest_base = map_zeroed_pages(mem / getpagesize()
                                                       + DEVICE_PAGES);
                         guest_limit = mem;
-                        guest_max = mem + DEVICE_PAGES*getpagesize();
-                        devices.descpage = get_pages(1);
+                        guest_max = guest_mmio = mem + DEVICE_PAGES*getpagesize();
                         break;
                 }
         }
 
+        /* We always have a console device, and it's always device 1. */
+        setup_console();
+
         /* The options are fairly straight-forward */
         while ((c = getopt_long(argc, argv, "v", opts, NULL)) != EOF) {
                 switch (c) {
@@ -1967,8 +3305,8 @@ int main(int argc, char *argv[])
 
         verbose("Guest base is at %p\n", guest_base);
 
-        /* We always have a console device */
-        setup_console();
+        /* Initialize the (fake) PCI host bridge device. */
+        init_pci_host_bridge();
 
         /* Now we load the kernel */
         start = load_kernel(open_or_die(argv[optind+1], O_RDONLY));