User Mode Linux HOWTO User Mode Linux Core Team Mon Nov 18 14:16:16 EST 2002 This document describes the use and abuse of Jeff Dike's User Mode Linux: a port of the Linux kernel as a normal Intel Linux process. ______________________________________________________________________ Table of Contents 1. Introduction 1.1 How is User Mode Linux Different? 1.2 Why Would I Want User Mode Linux? 2. Compiling the kernel and modules 2.1 Compiling the kernel 2.2 Compiling and installing kernel modules 2.3 Compiling and installing uml_utilities 3. Running UML and logging in 3.1 Running UML 3.2 Logging in 3.3 Examples 4. UML on 2G/2G hosts 4.1 Introduction 4.2 The problem 4.3 The solution 5. Setting up serial lines and consoles 5.1 Specifying the device 5.2 Specifying the channel 5.3 Examples 6. Setting up the network 6.1 General setup 6.2 Userspace daemons 6.3 Specifying ethernet addresses 6.4 UML interface setup 6.5 Multicast 6.6 TUN/TAP with the uml_net helper 6.7 TUN/TAP with a preconfigured tap device 6.8 Ethertap 6.9 The switch daemon 6.10 Slip 6.11 Slirp 6.12 pcap 6.13 Setting up the host yourself 7. Sharing Filesystems between Virtual Machines 7.1 A warning 7.2 Using layered block devices 7.3 Note! 7.4 Another warning 7.5 uml_moo : Merging a COW file with its backing file 8. Creating filesystems 8.1 Create the filesystem file 8.2 Assign the file to a UML device 8.3 Creating and mounting the filesystem 9. Host file access 9.1 Using hostfs 9.2 hostfs as the root filesystem 9.3 Building hostfs 10. The Management Console 10.1 version 10.2 halt and reboot 10.3 config 10.4 remove 10.5 sysrq 10.6 help 10.7 cad 10.8 stop 10.9 go 11. Kernel debugging 11.1 Starting the kernel under gdb 11.2 Examining sleeping processes 11.3 Running ddd on UML 11.4 Debugging modules 11.5 Attaching gdb to the kernel 11.6 Using alternate debuggers 12. Kernel debugging examples 12.1 The case of the hung fsck 12.2 Episode 2: The case of the hung fsck 13. What to do when UML doesn't work 13.1 Strange compilation errors when you build from source 13.2 (obsolete) 13.3 A variety of panics and hangs with /tmp on a reiserfs filesystem 13.4 The compile fails with errors about conflicting types for 'open', 'dup', and 'waitpid' 13.5 UML doesn't work when /tmp is an NFS filesystem 13.6 UML hangs on boot when compiled with gprof support 13.7 syslogd dies with a SIGTERM on startup 13.8 TUN/TAP networking doesn't work on a 2.4 host 13.9 You can network to the host but not to other machines on the net 13.10 I have no root and I want to scream 13.11 UML build conflict between ptrace.h and ucontext.h 13.12 The UML BogoMips is exactly half the host's BogoMips 13.13 When you run UML, it immediately segfaults 13.14 xterms appear, then immediately disappear 13.15 Any other panic, hang, or strange behavior 14. Diagnosing Problems 14.1 Case 1 : Normal kernel panics 14.2 Case 2 : Tracing thread panics 14.3 Case 3 : Tracing thread panics caused by other threads 14.4 Case 4 : Hangs 15. Thanks 15.1 Code and Documentation 15.2 Flushing out bugs 15.3 Buglets and clean-ups 15.4 Case Studies 15.5 Other contributions ______________________________________________________________________ 1. Introduction Welcome to User Mode Linux. It's going to be fun. 1.1. How is User Mode Linux Different? Normally, the Linux Kernel talks straight to your hardware (video card, keyboard, hard drives, etc), and any programs which run ask the kernel to operate the hardware, like so: +-----------+-----------+----+ | Process 1 | Process 2 | ...| +-----------+-----------+----+ | Linux Kernel | +----------------------------+ | Hardware | +----------------------------+ The User Mode Linux Kernel is different; instead of talking to the hardware, it talks to a `real' Linux kernel (called the `host kernel' from now on), like any other program. Programs can then run inside User-Mode Linux as if they were running under a normal kernel, like so: +----------------+ | Process 2 | ...| +-----------+----------------+ | Process 1 | User-Mode Linux| +----------------------------+ | Linux Kernel | +----------------------------+ | Hardware | +----------------------------+ 1.2. Why Would I Want User Mode Linux? 1. If User Mode Linux crashes, your host kernel is still fine. 2. You can run a usermode kernel as a non-root user. 3. You can debug the User Mode Linux like any normal process. 4. You can run gprof (profiling) and gcov (coverage testing). 5. You can play with your kernel without breaking things. 6. You can use it as a sandbox for testing new apps. 7. You can try new development kernels safely. 8. You can run different distributions simultaneously. 9. It's extremely fun. 2. Compiling the kernel and modules 2.1. Compiling the kernel Compiling the user mode kernel is just like compiling any other kernel. Let's go through the steps, using 2.4.0-prerelease (current as of this writing) as an example: 1. Download the latest UML patch from the download page . 3. Make a directory and unpack the kernel into it. host% mkdir ~/uml host% cd ~/uml host% tar -xzvf linux-2.4.0-prerelease.tar.bz2 4. Apply the patch using host% cd ~/uml/linux host% bzcat uml-patch-2.4.0-prerelease.bz2 | patch -p1 5. Run your favorite config; `make xconfig ARCH=um' is the most convenient. `make config ARCH=um' and 'make menuconfig ARCH=um' will work as well. The defaults will give you a useful kernel. If you want to change something, go ahead, it probably won't hurt anything. Note: If the host is configured with a 2G/2G address space split rather than the usual 3G/1G split, then the packaged UML binaries will not run. They will immediately segfault. See ``UML on 2G/2G hosts'' for the scoop on running UML on your system. 6. Finish with `make linux ARCH=um': the result is a file called `linux' in the top directory of your source tree. Make sure that you don't build this kernel in /usr/src/linux. On some distributions, /usr/include/asm is a link into this pool. The user- mode build changes the other end of that link, and things that include stop compiling. The sources are also available from cvs at the project's cvs page, which has directions on getting the sources. You can also browse the CVS pool from there. If you get the CVS sources, you will have to check them out into an empty directory. You will then have to copy each file into the corresponding directory in the appropriate kernel pool. If you don't have the latest kernel pool, you can get the corresponding user-mode sources with host% cvs co -r v_2_3_x linux where 'x' is the version in your pool. Note that you will not get the bug fixes and enhancements that have gone into subsequent releases. 2.2. Compiling and installing kernel modules UML modules are built in the same way as the native kernel (with the exception of the 'ARCH=um' that you always need for UML): host% make modules ARCH=um Any modules that you want to load into this kernel need to be built in the user-mode pool. Modules from the native kernel won't work. You can install them by using ftp or something to copy them into the virtual machine and dropping them into /lib/modules/`uname -r`. You can also get the kernel build process to install them as follows: 1. with the kernel not booted, mount the root filesystem in the top level of the kernel pool: host% mount root_fs mnt -o loop 2. run host% make modules_install INSTALL_MOD_PATH=`pwd`/mnt ARCH=um 3. unmount the filesystem host% umount mnt 4. boot the kernel on it When the system is booted, you can use insmod as usual to get the modules into the kernel. A number of things have been loaded into UML as modules, especially filesystems and network protocols and filters, so most symbols which need to be exported probably already are. However, if you do find symbols that need exporting, let us know, and they'll be "taken care of". 2.3. Compiling and installing uml_utilities Many features of the UML kernel require a user-space helper program, so a uml_utilities package is distributed separately from the kernel patch which provides these helpers. Included within this is: o port-helper - Used by consoles which connect to xterms or ports o tunctl - Configuration tool to create and delete tap devices o uml_net - Setuid binary for automatic tap device configuration o uml_switch - User-space virtual switch required for daemon transport The uml_utilities tree is compiled with: host# make && make install Note that UML kernel patches may require a specific version of the uml_utilities distribution. If you don't keep up with the mailing lists, ensure that you have the latest release of uml_utilities if you are experiencing problems with your UML kernel, particularly when dealing with consoles or command-line switches to the helper programs 3. Running UML and logging in 3.1. Running UML It runs on 2.2.15 or later, and all 2.4 kernels. Booting UML is straightforward. Simply run 'linux': it will try to mount the file `root_fs' in the current directory. You do not need to run it as root. If your root filesystem is not named `root_fs', then you need to put a `ubd0=root_fs_whatever' switch on the linux command line. You will need a filesystem to boot UML from. There are a number available for download from here . There are also several tools which can be used to generate UML-compatible filesystem images from media. The kernel will boot up and present you with a login prompt. Note: If the host is configured with a 2G/2G address space split rather than the usual 3G/1G split, then the packaged UML binaries will not run. They will immediately segfault. See ``UML on 2G/2G hosts'' for the scoop on running UML on your system. 3.2. Logging in The prepackaged filesystems have a root account with password 'root' and a user account with password 'user'. The login banner will generally tell you how to log in. So, you log in and you will find yourself inside a little virtual machine. Our filesystems have a variety of commands and utilities installed (and it is fairly easy to add more), so you will have a lot of tools with which to poke around the system. There are a couple of other ways to log in: o On a virtual console Each virtual console that is configured (i.e. the device exists in /dev and /etc/inittab runs a getty on it) will come up in its own xterm. If you get tired of the xterms, read ``Setting up serial lines and consoles'' to see how to attach the consoles to something else, like host ptys. o Over the serial line In the boot output, find a line that looks like: serial line 0 assigned pty /dev/ptyp1 Attach your favorite terminal program to the corresponding tty. I.e. for minicom, the command would be host% minicom -o -p /dev/ttyp1 o Over the net If the network is running, then you can telnet to the virtual machine and log in to it. See ``Setting up the network'' to learn about setting up a virtual network. When you're done using it, run halt, and the kernel will bring itself down and the process will exit. 3.3. Examples Here are some examples of UML in action: o A login session o A virtual network 4. UML on 2G/2G hosts 4.1. Introduction Most Linux machines are configured so that the kernel occupies the upper 1G (0xc0000000 - 0xffffffff) of the 4G address space and processes use the lower 3G (0x00000000 - 0xbfffffff). However, some machine are configured with a 2G/2G split, with the kernel occupying the upper 2G (0x80000000 - 0xffffffff) and processes using the lower 2G (0x00000000 - 0x7fffffff). 4.2. The problem The prebuilt UML binaries on this site will not run on 2G/2G hosts because UML occupies the upper .5G of the 3G process address space (0xa0000000 - 0xbfffffff). Obviously, on 2G/2G hosts, this is right in the middle of the kernel address space, so UML won't even load - it will immediately segfault. 4.3. The solution The fix for this is to rebuild UML from source after enabling CONFIG_HOST_2G_2G (under 'General Setup'). This will cause UML to load itself in the top .5G of that smaller process address space, where it will run fine. See ``Compiling the kernel and modules'' if you need help building UML from source. 5. Setting up serial lines and consoles It is possible to attach UML serial lines and consoles to many types of host I/O channels by specifying them on the command line. You can attach them to host ptys, ttys, file descriptors, and ports. This allows you to do things like o have a UML console appear on an unused host console, o hook two virtual machines together by having one attach to a pty and having the other attach to the corresponding tty o make a virtual machine accessible from the net by attaching a console to a port on the host. The general format of the command line option is device=channel. 5.1. Specifying the device Devices are specified with "con" or "ssl" (console or serial line, respectively), optionally with a device number if you are talking about a specific device. Using just "con" or "ssl" describes all of the consoles or serial lines. If you want to talk about console #3 or serial line #10, they would be "con3" and "ssl10", respectively. A specific device name will override a less general "con=" or "ssl=". So, for example, you can assign a pty to each of the serial lines except for the first two like this: ssl=pty ssl0=tty:/dev/tty0 ssl1=tty:/dev/tty1 The specificity of the device name is all that matters; order on the command line is irrelevant. 5.2. Specifying the channel There are a number of different types of channels to attach a UML device to, each with a different way of specifying exactly what to attach to. o pseudo-terminals - device=pty pts terminals - device=pts This will cause UML to allocate a free host pseudo-terminal for the device. The terminal that it got will be announced in the boot log. You access it by attaching a terminal program to the corresponding tty: o screen /dev/pts/n o screen /dev/ttyxx o minicom -o -p /dev/ttyxx - minicom seems not able to handle pts devices o kermit - start it up, 'open' the device, then 'connect' o terminals - device=tty:tty device file This will make UML attach the device to the specified tty (i.e con1=tty:/dev/tty3 will attach UML's console 1 to the host's /dev/tty3). If the tty that you specify is the slave end of a tty/pty pair, something else must have already opened the corresponding pty in order for this to work. o xterms - device=xterm UML will run an xterm and the device will be attached to it. o Port - device=port:port number This will attach the UML devices to the specified host port. Attaching console 1 to the host's port 9000 would be done like this: con1=port:9000 Attaching all the serial lines to that port would be done similarly: ssl=port:9000 You access these devices by telnetting to that port. Each active tel- net session gets a different device. If there are more telnets to a port than UML devices attached to it, then the extra telnet sessions will block until an existing telnet detaches, or until another device becomes active (i.e. by being activated in /etc/inittab). This channel has the advantage that you can both attach multiple UML devices to it and know how to access them without reading the UML boot log. It is also unique in allowing access to a UML from remote machines without requiring that the UML be networked. This could be useful in allowing public access to UMLs because they would be accessible from the net, but wouldn't need any kind of network filtering or access control because they would have no network access. If you attach the main console to a portal, then the UML boot will appear to hang. In reality, it's waiting for a telnet to connect, at which point the boot will proceed. o already-existing file descriptors - device=file descriptor If you set up a file descriptor on the UML command line, you can attach a UML device to it. This is most commonly used to put the main console back on stdin and stdout after assigning all the other consoles to something else: con0=fd:0,fd:1 con=pts o Nothing - device=null This allows the device to be opened, in contrast to 'none', but reads will block, and writes will succeed and the data will be thrown out. o None - device=none This causes the device to disappear. You can also specify different input and output channels for a device by putting a comma between them: ssl3=tty:/dev/tty2,xterm will cause serial line 3 to accept input on the host's /dev/tty2 and display output on an xterm. That's a silly example - the most common use of this syntax is to reattach the main console to stdin and stdout as shown above. If you decide to move the main console away from stdin/stdout, the initial boot output will appear in the terminal that you're running UML in. However, once the console driver has been officially initialized, then the boot output will start appearing wherever you specified that console 0 should be. That device will receive all subsequent output. 5.3. Examples There are a number of interesting things you can do with this capability. First, this is how you get rid of those bleeding console xterms by attaching them to host ptys: con=pty con0=fd:0,fd:1 This will make a UML console take over an unused host virtual console, so that when you switch to it, you will see the UML login prompt rather than the host login prompt: con1=tty:/dev/tty6 You can attach two virtual machines together with what amounts to a serial line as follows: Run one UML with a serial line attached to a pty - ssl1=pty Look at the boot log to see what pty it got (this example will assume that it got /dev/ptyp1). Boot the other UML with a serial line attached to the corresponding tty - ssl1=tty:/dev/ttyp1 Log in, make sure that it has no getty on that serial line, attach a terminal program like minicom to it, and you should see the login prompt of the other virtual machine. 6. Setting up the network This page describes how to set up the various transports and to provide a UML instance with network access to the host, other machines on the local net, and the rest of the net. As of 2.4.5, UML networking has been completely redone to make it much easier to set up, fix bugs, and add new features. There is a new helper, uml_net, which does the host setup that requires root privileges. There are currently five transport types available for a UML virtual machine to exchange packets with other hosts: o ethertap o TUN/TAP o Multicast o a switch daemon o slip o slirp o pcap The TUN/TAP, ethertap, slip, and slirp transports allow a UML instance to exchange packets with the host. They may be directed to the host or the host may just act as a router to provide access to other physical or virtual machines. The pcap transport is a synthetic read-only interface, using the libpcap binary to collect packets from interfaces on the host and filter them. This is useful for building preconfigured traffic monitors or sniffers. The daemon and multicast transports provide a completely virtual network to other virtual machines. This network is completely disconnected from the physical network unless one of the virtual machines on it is acting as a gateway. With so many host transports, which one should you use? Here's when you should use each one: o ethertap - if you want access to the host networking and it is running 2.2 o TUN/TAP - if you want access to the host networking and it is running 2.4. Also, the TUN/TAP transport is able to use a preconfigured device, allowing it to avoid using the setuid uml_net helper, which is a security advantage. o Multicast - if you want a purely virtual network and you don't want to set up anything but the UML o a switch daemon - if you want a purely virtual network and you don't mind running the daemon in order to get somewhat better performance o slip - there is no particular reason to run the slip backend unless ethertap and TUN/TAP are just not available for some reason o slirp - if you don't have root access on the host to setup networking, or if you don't want to allocate an IP to your UML o pcap - not much use for actual network connectivity, but great for monitoring traffic on the host Ethertap is available on 2.4 and works fine. TUN/TAP is preferred to it because it has better performance and ethertap is officially considered obsolete in 2.4. Also, the root helper only needs to run occasionally for TUN/TAP, rather than handling every packet, as it does with ethertap. This is a slight security advantage since it provides fewer opportunities for a nasty UML user to somehow exploit the helper's root privileges. 6.1. General setup First, you must have the virtual network enabled in your UML. If are running a prebuilt kernel from this site, everything is already enabled. If you build the kernel yourself, under the "Network device support" menu, enable "Network device support", and then the three transports. The next step is to provide a network device to the virtual machine. This is done by describing it on the kernel command line. The general format is eth = , For example, a virtual ethernet device may be attached to a host ethertap device as follows: eth0=ethertap,tap0,fe:fd:0:0:0:1,192.168.0.254 This sets up eth0 inside the virtual machine to attach itself to the host /dev/tap0, assigns it an ethernet address, and assigns the host tap0 interface an IP address. Note that the IP address you assign to the host end of the tap device must be different than the IP you assign to the eth device inside UML. If you are short on IPs and don't want to consume two per UML, then you can reuse the host's eth IP address for the host ends of the tap devices. Internally, the UMLs must still get unique IPs for their eth devices. You can also give the UMLs non-routable IPs (192.168.x.x or 10.x.x.x) and have the host masquerade them. This will let outgoing connections work, but incoming connections won't without more work, such as port forwarding from the host. Also note that when you configure the host side of an interface, it is only acting as a gateway. It will respond to pings sent to it locally, but is not useful to do that since it's a host interface. You are not talking to the UML when you ping that interface and get a response. You can also add devices to a UML and remove them at runtime. See the ``The Management Console'' page for details. The sections below describe this in more detail. Once you've decided how you're going to set up the devices, you boot UML, log in, configure the UML side of the devices, and set up routes to the outside world. At that point, you will be able to talk to any other machines, physical or virtual, on the net. If ifconfig inside UML fails and the network refuses to come up, run tell you what went wrong. 6.2. Userspace daemons You will likely need the setuid helper, or the switch daemon, or both. They are both installed with the RPM and deb, so if you've installed either, you can skip the rest of this section. If not, then you need to check them out of CVS, build them, and install them. The helper is uml_net, in CVS /tools/uml_net, and the daemon is uml_switch, in CVS /tools/uml_router. They are both built with a plain 'make'. Both need to be installed in a directory that's in your path - /usr/bin is recommend. On top of that, uml_net needs to be setuid root. 6.3. Specifying ethernet addresses Below, you will see that the TUN/TAP, ethertap, and daemon interfaces allow you to specify hardware addresses for the virtual ethernet devices. This is generally not necessary. If you don't have a specific reason to do it, you probably shouldn't. If one is not specified on the command line, the driver will assign one based on the device IP address. It will provide the address fe:fd:nn:nn:nn:nn where nn.nn.nn.nn is the device IP address. This is nearly always sufficient to guarantee a unique hardware address for the device. A couple of exceptions are: o Another set of virtual ethernet devices are on the same network and they are assigned hardware addresses using a different scheme which may conflict with the UML IP address-based scheme o You aren't going to use the device for IP networking, so you don't assign the device an IP address If you let the driver provide the hardware address, you should make sure that the device IP address is known before the interface is brought up. So, inside UML, this will guarantee that: UML# ifconfig eth0 192.168.0.250 up If you decide to assign the hardware address yourself, make sure that the first byte of the address is even. Addresses with an odd first byte are broadcast addresses, which you don't want assigned to a device. 6.4. UML interface setup Once the network devices have been described on the command line, you should boot UML and log in. The first thing to do is bring the interface up: UML# ifconfig ethn ip-address up You should be able to ping the host at this point. To reach the rest of the world, you should set a default route to the host: UML# route add default gw host ip Again, with host ip of 192.168.0.4: UML# route add default gw 192.168.0.4 This page used to recommend setting a network route to your local net. This is wrong, because it will cause UML to try to figure out hardware addresses of the local machines by arping on the interface to the host. Since that interface is basically a single strand of ethernet with two nodes on it (UML and the host) and arp requests don't cross networks, they will fail to elicit any responses. So, what you want is for UML to just blindly throw all packets at the host and let it figure out what to do with them, which is what leaving out the network route and adding the default route does. Note: If you can't communicate with other hosts on your physical ethernet, it's probably because of a network route that's automatically set up. If you run 'route -n' and see a route that looks like this: Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 with a mask that's not 255.255.255.255, then replace it with a route to your host: UML# route del -net 192.168.0.0 dev eth0 netmask 255.255.255.0 UML# route add -host 192.168.0.4 dev eth0 This, plus the default route to the host, will allow UML to exchange packets with any machine on your ethernet. 6.5. Multicast The simplest way to set up a virtual network between multiple UMLs is to use the mcast transport. This was written by Harald Welte and is present in UML version 2.4.5-5um and later. Your system must have multicast enabled in the kernel and there must be a multicast-capable network device on the host. Normally, this is eth0, but if there is no ethernet card on the host, then you will likely get strange error messages when you bring the device up inside UML. To use it, run two UMLs with eth0=mcast on their command lines. Log in, configure the ethernet device in each machine with different IP addresses: UML1# ifconfig eth0 192.168.0.254 UML2# ifconfig eth0 192.168.0.253 and they should be able to talk to each other. The full set of command line options for this transport are ethn=mcast,ethernet address,multicast address,multicast port,ttl Harald's original README is here and explains these in detail, as well as some other issues. There is also a related point-to-point only "ucast" transport. This is useful when your network does not support multicast, and all network connections are simple point to point links. The full set of command line options for this transport are ethn=ucast,ethernet address,remote address,listen port,remote port 6.6. TUN/TAP with the uml_net helper TUN/TAP is the preferred mechanism on 2.4 to exchange packets with the host. The TUN/TAP backend has been in UML since 2.4.9-3um. The easiest way to get up and running is to let the setuid uml_net helper do the host setup for you. This involves insmod-ing the tun.o module if necessary, configuring the device, and setting up IP forwarding, routing, and proxy arp. If you are new to UML networking, do this first. If you're concerned about the security implications of the setuid helper, use it to get up and running, then read the next section to see how to have UML use a preconfigured tap device, which avoids the use of uml_net. If you specify an IP address for the host side of the device, the uml_net helper will do all necessary setup on the host - the only requirement is that TUN/TAP be available, either built in to the host kernel or as the tun.o module. The format of the command line switch to attach a device to a TUN/TAP device is eth =tuntap,,, For example, this argument will attach the UML's eth0 to the next available tap device and assign an ethernet address to it based on its IP address eth0=tuntap,,,192.168.0.254 Note that the IP address that must be used for the eth device inside UML is fixed by the routing and proxy arp that is set up on the TUN/TAP device on the host. You can use a different one, but it won't work because reply packets won't reach the UML. This is a feature. It prevents a nasty UML user from doing things like setting the UML IP to the same as the network's nameserver or mail server. There are a couple potential problems with running the TUN/TAP transport on a 2.4 host kernel o TUN/TAP seems not to work on 2.4.3 and earlier. Upgrade the host kernel or use the ethertap transport. o With an upgraded kernel, TUN/TAP may fail with File descriptor in bad state This is due to a header mismatch between the upgraded kernel and the kernel that was originally installed on the machine. The fix is to make sure that /usr/src/linux points to the headers for the running kernel. These were pointed out by Tim Robinson in name="this uml- user post"> . 6.7. TUN/TAP with a preconfigured tap device If you prefer not to have UML use uml_net (which is somewhat insecure), with UML 2.4.17-11, you can set up a TUN/TAP device beforehand. The setup needs to be done as root, but once that's done, there is no need for root assistance. Setting up the device is done as follows: o Create the device with tunctl (available from the UML utilities tarball) host# tunctl -u uid where uid is the user id or username that UML will be run as. This will tell you what device was created. o Configure the device IP (change IP addresses and device name to suit) host# ifconfig tap0 192.168.0.254 up o Set up routing and arping if desired - this is my recipe, there are other ways of doing the same thing host# bash -c 'echo 1 > /proc/sys/net/ipv4/ip_forward' host# route add -host 192.168.0.253 dev tap0 host# bash -c 'echo 1 > /proc/sys/net/ipv4/conf/tap0/proxy_arp' host# arp -Ds 192.168.0.253 eth0 pub Note that this must be done every time the host boots - this configu- ration is not stored across host reboots. So, it's probably a good idea to stick it in an rc file. An even better idea would be a little utility which reads the information from a config file and sets up devices at boot time. o Rather than using up two IPs and ARPing for one of them, you can also provide direct access to your LAN by the UML by using a bridge. host# brctl addbr br0 host# ifconfig eth0 0.0.0.0 promisc up host# ifconfig tap0 0.0.0.0 promisc up host# ifconfig br0 192.168.0.1 netmask 255.255.255.0 up host# brctl stp br0 off host# brctl setfd br0 1 host# brctl sethello br0 1 host# brctl addif br0 eth0 host# brctl addif br0 tap0 Note that 'br0' should be setup using ifconfig with the existing IP address of eth0, as eth0 no longer has its own IP. o Also, the /dev/net/tun device must be writable by the user running UML in order for the UML to use the device that's been configured for it. The simplest thing to do is host# chmod 666 /dev/net/tun Making it world-writable looks bad, but it seems not to be exploitable as a security hole. However, it does allow anyone to cre- ate useless tap devices (useless because they can't configure them), which is a DOS attack. A somewhat more secure alternative would to be to create a group containing all the users who have preconfigured tap devices and chgrp /dev/net/tun to that group with mode 664 or 660. o Once the device is set up, run UML with 'eth0=tuntap,device name' (i.e. 'eth0=tuntap,tap0') on the command line (or do it with the mconsole config command). o Bring the eth device up in UML and you're in business. If you don't want that tap device any more, you can make it non- persistent with host# tunctl -d tap device Finally, tunctl has a -b (for brief mode) switch which causes it to output only the name of the tap device it created. This makes it suitable for capture by a script: host# TAP=`tunctl -u 1000 -b` 6.8. Ethertap Ethertap is the general mechanism on 2.2 for userspace processes to exchange packets with the kernel. To use this transport, you need to describe the virtual network device on the UML command line. The general format for this is eth =ethertap, , , So, the previous example eth0=ethertap,tap0,fe:fd:0:0:0:1,192.168.0.254 attaches the UML eth0 device to the host /dev/tap0, assigns it the ethernet address fe:fd:0:0:0:1, and assigns the IP address 192.168.0.254 to the tap device. The tap device is mandatory, but the others are optional. If the ethernet address is omitted, one will be assigned to it. The presence of the tap IP address will cause the helper to run and do whatever host setup is needed to allow the virtual machine to communicate with the outside world. If you're not sure you know what you're doing, this is the way to go. If it is absent, then you must configure the tap device and whatever arping and routing you will need on the host. However, even in this case, the uml_net helper still needs to be in your path and it must be setuid root if you're not running UML as root. This is because the tap device doesn't support SIGIO, which UML needs in order to use something as a source of input. So, the helper is used as a convenient asynchronous IO thread. If you're using the uml_net helper, you can ignore the following host setup - uml_net will do it for you. You just need to make sure you have ethertap available, either built in to the host kernel or available as a module. If you want to set things up yourself, you need to make sure that the appropriate /dev entry exists. If it doesn't, become root and create it as follows: mknod /dev/tap c 36 + 16 For example, this is how to create /dev/tap0: mknod /dev/tap0 c 36 0 + 16 You also need to make sure that the host kernel has ethertap support. If ethertap is enabled as a module, you apparently need to insmod ethertap once for each ethertap device you want to enable. So, host# insmod ethertap will give you the tap0 interface. To get the tap1 interface, you need to run host# insmod ethertap unit=1 -o ethertap1 6.9. The switch daemon Note: This is the daemon formerly known as uml_router, but which was renamed so the network weenies of the world would stop growling at me. The switch daemon, uml_switch, provides a mechanism for creating a totally virtual network. By default, it provides no connection to the host network (but see -tap, below). The first thing you need to do is run the daemon. Running it with no arguments will make it listen on a default pair of unix domain sockets. If you want it to listen on a different pair of sockets, use -unix control socket data socket If you want it to act as a hub rather than a switch, use -hub If you want the switch to be connected to host networking (allowing the umls to get access to the outside world through the host), use -tap tap0 Note that the tap device must be preconfigured (see "TUN/TAP with a preconfigured tap device", above). If you're using a different tap device than tap0, specify that instead of tap0. uml_switch can be backgrounded as follows host% uml_switch [ options ] < /dev/null > /dev/null The reason it doesn't background by default is that it listens to stdin for EOF. When it sees that, it exits. The general format of the kernel command line switch is ethn=daemon,ethernet address,socket type,control socket,data socket You can leave off everything except the 'daemon'. You only need to specify the ethernet address if the one that will be assigned to it isn't acceptable for some reason. The rest of the arguments describe how to communicate with the daemon. You should only specify them if you told the daemon to use different sockets than the default. So, if you ran the daemon with no arguments, running the UML on the same machine with eth0=daemon will cause the eth0 driver to attach itself to the daemon correctly. 6.10. Slip Slip is another, less general, mechanism for a process to communicate with the host networking. In contrast to the ethertap interface, which exchanges ethernet frames with the host and can be used to transport any higher-level protocol, it can only be used to transport IP. The general format of the command line switch is ethn=slip,slip IP The slip IP argument is the IP address that will be assigned to the host end of the slip device. If it is specified, the helper will run and will set up the host so that the virtual machine can reach it and the rest of the network. There are some oddities with this interface that you should be aware of. You should only specify one slip device on a given virtual machine, and its name inside UML will be 'umn', not 'eth0' or whatever you specified on the command line. These problems will be fixed at some point. 6.11. Slirp slirp uses an external program, usually /usr/bin/slirp, to provide IP only networking connectivity through the host. This is similar to IP masquerading with a firewall, although the translation is performed in user-space, rather than by the kernel. As slirp does not set up any interfaces on the host, or changes routing, slirp does not require root access or setuid binaries on the host. The general format of the command line switch for slirp is: ethn=slirp,ethernet address,slirp path The ethernet address is optional, as UML will set up the interface with an ethernet address based upon the initial IP address of the interface. The slirp path is generally /usr/bin/slirp, although it will depend on distribution. The slirp program can have a number of options passed to the command line and we can't add them to the UML command line, as they will be parsed incorrectly. Instead, a wrapper shell script can be written or the options inserted into the /.slirprc file. More information on all of the slirp options can be found in its man pages. The eth0 interface on UML should be set up with the IP 10.2.0.15, although you can use anything as long as it is not used by a network you will be connecting to. The default route on UML should be set to use UML# route add default dev eth0 slirp provides a number of useful IP addresses which can be used by UML, such as 10.0.2.3 which is an alias for the DNS server specified in /etc/resolv.conf on the host or the IP given in the 'dns' option for slirp. Even with a baudrate setting higher than 115200, the slirp connection is limited to 115200. If you need it to go faster, the slirp binary needs to be compiled with FULL_BOLT defined in config.h. 6.12. pcap The pcap transport is attached to a UML ethernet device on the command line or with uml_mconsole with the following syntax: ethn=pcap,host interface,filter expression,option1,option2 The expression and options are optional. The interface is whatever network device on the host you want to sniff. The expression is a pcap filter expression, which is also what tcpdump uses, so if you know how to specify tcpdump filters, you will use the same expressions here. The options are up to two of 'promisc', control whether pcap puts the host interface into promiscuous mode. 'optimize' and 'nooptimize' control whether the pcap expression optimizer is used. Example: eth0=pcap,eth0,tcp eth1=pcap,eth0,!tcp will cause the UML eth0 to emit all tcp packets on the host eth0 and the UML eth1 to emit all non-tcp packets on the host eth0. 6.13. Setting up the host yourself If you don't specify an address for the host side of the ethertap or slip device, UML won't do any setup on the host. So this is what is needed to get things working (the examples use a host-side IP of 192.168.0.251 and a UML-side IP of 192.168.0.250 - adjust to suit your own network): o The device needs to be configured with its IP address. Tap devices are also configured with an mtu of 1484. Slip devices are configured with a point-to-point address pointing at the UML ip address. host# ifconfig tap0 arp mtu 1484 192.168.0.251 up host# ifconfig sl0 192.168.0.251 pointopoint 192.168.0.250 up o If a tap device is being set up, a route is set to the UML IP. UML# route add -host 192.168.0.250 gw 192.168.0.251 o To allow other hosts on your network to see the virtual machine, proxy arp is set up for it. host# arp -Ds 192.168.0.250 eth0 pub o Finally, the host is set up to route packets. host# echo 1 > /proc/sys/net/ipv4/ip_forward 7. Sharing Filesystems between Virtual Machines 7.1. A warning Don't attempt to share filesystems simply by booting two UMLs from the same file. That's the same thing as booting two physical machines from a shared disk. It will result in filesystem corruption. 7.2. Using layered block devices The way to share a filesystem between two virtual machines is to use the copy-on-write (COW) layering capability of the ubd block driver. As of 2.4.6-2um, the driver supports layering a read-write private device over a read-only shared device. A machine's writes are stored in the private device, while reads come from either device - the private one if the requested block is valid in it, the shared one if not. Using this scheme, the majority of data which is unchanged is shared between an arbitrary number of virtual machines, each of which has a much smaller file containing the changes that it has made. With a large number of UMLs booting from a large root filesystem, this leads to a huge disk space saving. It will also help performance, since the host will be able to cache the shared data using a much smaller amount of memory, so UML disk requests will be served from the host's memory rather than its disks. To add a copy-on-write layer to an existing block device file, simply add the name of the COW file to the appropriate ubd switch: ubd0=root_fs_cow,root_fs_debian_22 where 'root_fs_cow' is the private COW file and 'root_fs_debian_22' is the existing shared filesystem. The COW file need not exist. If it doesn't, the driver will create and initialize it. Once the COW file has been initialized, it can be used on its own on the command line: ubd0=root_fs_cow The name of the backing file is stored in the COW file header, so it would be redundant to continue specifying it on the command line. 7.3. Note! When checking the size of the COW file in order to see the gobs of space that you're saving, make sure you use 'ls -ls' to see the actual disk consumption rather than the length of the file. The COW file is sparse, so the length will be very different from the disk usage. Here is a 'ls -l' of a COW file and backing file from one boot and shutdown: host% ls -l cow.debian debian2.2 -rw-r--r-- 1 jdike jdike 492504064 Aug 6 21:16 cow.debian -rwxrw-rw- 1 jdike jdike 537919488 Aug 6 20:42 debian2.2 Doesn't look like much saved space, does it? Well, here's 'ls -ls': host% ls -ls cow.debian debian2.2 880 -rw-r--r-- 1 jdike jdike 492504064 Aug 6 21:16 cow.debian 525832 -rwxrw-rw- 1 jdike jdike 537919488 Aug 6 20:42 debian2.2 Now, you can see that the COW file has less than a meg of disk, rather than 492 meg. 7.4. Another warning Once a filesystem is being used as a readonly backing file for a COW file, do not boot directly from it or modify it in any way. Doing so will invalidate any COW files that are using it. The mtime and size of the backing file are stored in the COW file header at its creation, and they must continue to match. If they don't, the driver will refuse to use the COW file. If you attempt to evade this restriction by changing either the backing file or the COW header by hand, you will get a corrupted filesystem. Among other things, this means that upgrading the distribution in a backing file and expecting that all of the COW files using it will see the upgrade will not work. 7.5. uml_moo : Merging a COW file with its backing file Depending on how you use UML and COW devices, it may be advisable to merge the changes in the COW file into the backing file every once in a while. The utility that does this is uml_moo. Its usage is host% uml_moo COW file new backing file There's no need to specify the backing file since that information is already in the COW file header. If you're paranoid, boot the new merged file, and if you're happy with it, move it over the old backing file. uml_moo creates a new backing file by default as a safety measure. It also has a destructive merge option which will merge the COW file directly into its current backing file. This is really only usable when the backing file only has one COW file associated with it. If there are multiple COWs associated with a backing file, a -d merge of one of them will invalidate all of the others. However, it is convenient if you're short of disk space, and it should also be noticeably faster than a non-destructive merge. uml_moo is installed with the UML deb and RPM. If you didn't install UML from one of those packages, you can also get it from the UML utilities tar file in tools/moo. 8. Creating filesystems You may want to create and mount new UML filesystems, either because your root filesystem isn't large enough or because you want to use a filesystem other than ext2. This was written on the occasion of reiserfs being included in the 2.4.1 kernel pool, and therefore the 2.4.1 UML, so the examples will talk about reiserfs. This information is generic, and the examples should be easy to translate to the filesystem of your choice. 8.1. Create the filesystem file dd is your friend. All you need to do is tell dd to create an empty file of the appropriate size. I usually make it sparse to save time and to avoid allocating disk space until it's actually used. For example, the following command will create a sparse 100 meg file full of zeroes. host% dd if=/dev/zero of=new_filesystem seek=100 count=1 bs=1M 8.2. Assign the file to a UML device Add an argument like the following to the UML command line: ubd4=new_filesystem making sure that you use an unassigned ubd device number. 8.3. Creating and mounting the filesystem Make sure that the filesystem is available, either by being built into the kernel, or available as a module, then boot up UML and log in. If the root filesystem doesn't have the filesystem utilities (mkfs, fsck, etc), then get them into UML by way of the net or hostfs. Make the new filesystem on the device assigned to the new file: host# mkreiserfs /dev/ubd/4 <----------- MKREISERFSv2 -----------> ReiserFS version 3.6.25 Block size 4096 bytes Block count 25856 Used blocks 8212 Journal - 8192 blocks (18-8209), journal header is in block 8210 Bitmaps: 17 Root block 8211 Hash function "r5" ATTENTION: ALL DATA WILL BE LOST ON '/dev/ubd/4'! (y/n)y journal size 8192 (from 18) Initializing journal - 0%....20%....40%....60%....80%....100% Syncing..done. Now, mount it: UML# mount /dev/ubd/4 /mnt and you're in business. 9. Host file access If you want to access files on the host machine from inside UML, you can treat it as a separate machine and either nfs mount directories from the host or copy files into the virtual machine with scp or rcp. However, since UML is running on the host, it can access those files just like any other process and make them available inside the virtual machine without needing to use the network. This is now possible with the hostfs virtual filesystem. With it, you can mount a host directory into the UML filesystem and access the files contained in it just as you would on the host. 9.1. Using hostfs To begin with, make sure that hostfs is available inside the virtual machine with UML# cat /proc/filesystems . hostfs should be listed. If it's not, either rebuild the kernel with hostfs configured into it or make sure that hostfs is built as a module and available inside the virtual machine, and insmod it. Now all you need to do is run mount: UML# mount none /mnt/host -t hostfs will mount the host's / on the virtual machine's /mnt/host. If you don't want to mount the host root directory, then you can specify a subdirectory to mount with the -o switch to mount: UML# mount none /mnt/home -t hostfs -o /home will mount the hosts's /home on the virtual machine's /mnt/home. 9.2. hostfs as the root filesystem It's possible to boot from a directory hierarchy on the host using hostfs rather than using the standard filesystem in a file. To start, you need that hierarchy. The easiest way is to loop mount an existing root_fs file: host# mount root_fs uml_root_dir -o loop You need to change the filesystem type of / in etc/fstab to be 'hostfs', so that line looks like this: /dev/ubd/0 / hostfs defaults 1 1 Then you need to chown to yourself all the files in that directory that are owned by root. This worked for me: host# find . -uid 0 -exec chown jdike {} \; Next, make sure that your UML kernel has hostfs compiled in, not as a module. Then run UML with the boot device pointing at that directory: ubd0=/path/to/uml/root/directory UML should then boot as it does normally. 9.3. Building hostfs If you need to build hostfs because it's not in your kernel, you have two choices: o Compiling hostfs into the kernel: Reconfigure the kernel and set the 'Host filesystem' option under o Compiling hostfs as a module: Reconfigure the kernel and set the 'Host filesystem' option under be in arch/um/fs/hostfs/hostfs.o. Install that in /lib/modules/`uname -r`/fs in the virtual machine, boot it up, and UML# insmod hostfs 10. The Management Console The UML management console is a low-level interface to the kernel, somewhat like the i386 SysRq interface. Since there is a full-blown operating system under UML, there is much greater flexibility possible than with the SysRq mechanism. There are a number of things you can do with the mconsole interface: o get the kernel version o add and remove devices o halt or reboot the machine o Send SysRq commands o Pause and resume the UML You need the mconsole client (uml_mconsole) which is present in CVS (/tools/mconsole) in 2.4.5-9um and later, and will be in the RPM in 2.4.6. You also need CONFIG_MCONSOLE (under 'General Setup') enabled in UML. When you boot UML, you'll see a line like: mconsole initialized on /home/jdike/.uml/umlNJ32yL/mconsole If you specify a unique machine id one the UML command line, i.e. umid=debian you'll see this mconsole initialized on /home/jdike/.uml/debian/mconsole That file is the socket that uml_mconsole will use to communicate with UML. Run it with either the umid or the full path as its argument: host% uml_mconsole debian or host% uml_mconsole /home/jdike/.uml/debian/mconsole You'll get a prompt, at which you can run one of these commands: o version o halt o reboot o config o remove o sysrq o help o cad o stop o go 10.1. version This takes no arguments. It prints the UML version. (mconsole) version OK Linux usermode 2.4.5-9um #1 Wed Jun 20 22:47:08 EDT 2001 i686 There are a couple actual uses for this. It's a simple no-op which can be used to check that a UML is running. It's also a way of sending an interrupt to the UML. This is sometimes useful on SMP hosts, where there's a bug which causes signals to UML to be lost, often causing it to appear to hang. Sending such a UML the mconsole version command is a good way to 'wake it up' before networking has been enabled, as it does not do anything to the function of the UML. 10.2. halt and reboot These take no arguments. They shut the machine down immediately, with no syncing of disks and no clean shutdown of userspace. So, they are pretty close to crashing the machine. (mconsole) halt OK 10.3. config "config" adds a new device to the virtual machine. Currently the ubd and network drivers support this. It takes one argument, which is the device to add, with the same syntax as the kernel command line. (mconsole) config ubd3=/home/jdike/incoming/roots/root_fs_debian22 OK (mconsole) config eth1=mcast OK 10.4. remove "remove" deletes a device from the system. Its argument is just the name of the device to be removed. The device must be idle in whatever sense the driver considers necessary. In the case of the ubd driver, the removed block device must not be mounted, swapped on, or otherwise open, and in the case of the network driver, the device must be down. (mconsole) remove ubd3 OK (mconsole) remove eth1 OK 10.5. sysrq This takes one argument, which is a single letter. It calls the generic kernel's SysRq driver, which does whatever is called for by that argument. See the SysRq documentation in Documentation/sysrq.txt in your favorite kernel tree to see what letters are valid and what they do. 10.6. help "help" returns a string listing the valid commands and what each one does. 10.7. cad This invokes the Ctl-Alt-Del action on init. What exactly this ends up doing is up to /etc/inittab. Normally, it reboots the machine. With UML, this is usually not desired, so if a halt would be better, then find the section of inittab that looks like this # What to do when CTRL-ALT-DEL is pressed. ca:12345:ctrlaltdel:/sbin/shutdown -t1 -a -r now and change the command to halt. 10.8. stop This puts the UML in a loop reading mconsole requests until a 'go' mconsole command is received. This is very useful for making backups of UML filesystems, as the UML can be stopped, then synced via 'sysrq s', so that everything is written to the filesystem. You can then copy the filesystem and then send the UML 'go' via mconsole. Note that a UML running with more than one CPU will have problems after you send the 'stop' command, as only one CPU will be held in a mconsole loop and all others will continue as normal. This is a bug, and will be fixed. 10.9. go This resumes a UML after being paused by a 'stop' command. Note that when the UML has resumed, TCP connections may have timed out and if the UML is paused for a long period of time, crond might go a little crazy, running all the jobs it didn't do earlier. 11. Kernel debugging Note: The interface that makes debugging, as described here, possible is present in 2.4.0-test6 kernels and later. Since the user-mode kernel runs as a normal Linux process, it is possible to debug it with gdb almost like any other process. It is slightly different because the kernel's threads are already being ptraced for system call interception, so gdb can't ptrace them. However, a mechanism has been added to work around that problem. In order to debug the kernel, you need build it from source. See ``Compiling the kernel and modules'' for information on doing that. Make sure that you enable CONFIG_DEBUGSYM and CONFIG_PT_PROXY during the config. These will compile the kernel with -g, and enable the ptrace proxy so that gdb works with UML, respectively. 11.1. Starting the kernel under gdb You can have the kernel running under the control of gdb from the beginning by putting 'debug' on the command line. You will get an xterm with gdb running inside it. The kernel will send some commands to gdb which will leave it stopped at the beginning of start_kernel. At this point, you can get things going with 'next', 'step', or 'cont'. There is a transcript of a debugging session here , with breakpoints being set in the scheduler and in an interrupt handler. 11.2. Examining sleeping processes Not every bug is evident in the currently running process. Sometimes, processes hang in the kernel when they shouldn't because they've deadlocked on a semaphore or something similar. In this case, when you ^C gdb and get a backtrace, you will see the idle thread, which isn't very relevant. What you want is the stack of whatever process is sleeping when it shouldn't be. You need to figure out which process that is, which is generally fairly easy. Then you need to get its host process id, which you can do either by looking at ps on the host or at task.thread.extern_pid in gdb. Now what you do is this: o detach from the current thread (UML gdb) det o attach to the thread you are interested in (UML gdb) att o look at its stack and anything else of interest (UML gdb) bt Note that you can't do anything at this point that requires that a process execute, e.g. calling a function o when you're done looking at that process, reattach to the current thread and continue it (UML gdb) att 1 (UML gdb) c Here, specifying any pid which is not the process id of a UML thread will cause gdb to reattach to the current thread. I commonly use 1, but any other invalid pid would work. 11.3. Running ddd on UML ddd works on UML, but requires a special kludge. The process goes like this: o Start ddd host% ddd linux o With ps, get the pid of the gdb that ddd started. You can ask the gdb to tell you, but for some reason that confuses things and causes a hang. o run UML with 'debug=parent gdb-pid=' added to the command line - it will just sit there after you hit return o type 'att 1' to the ddd gdb and you will see something like 0xa013dc51 in __kill () (gdb) o At this point, type 'c', UML will boot up, and you can use ddd just as you do on any other process. 11.4. Debugging modules gdb has support for debugging code which is dynamically loaded into the process. This support is what is needed to debug kernel modules under UML. Using that support is somewhat complicated. You have to tell gdb what object file you just loaded into UML and where in memory it is. Then, it can read the symbol table, and figure out where all the symbols are from the load address that you provided. It gets more interesting when you load the module again (i.e. after an rmmod). You have to tell gdb to forget about all its symbols, including the main UML ones for some reason, then load then all back in again. There's an easy way and a hard way to do this. The easy way is to use the umlgdb expect script written by Chandan Kudige. It basically automates the process for you. First, you must tell it where your modules are. There is a list in the script that looks like this: set MODULE_PATHS { "fat" "/usr/src/uml/linux-2.4.18/fs/fat/fat.o" "isofs" "/usr/src/uml/linux-2.4.18/fs/isofs/isofs.o"