Xen-test-debian

Xen HOWTOS for Debian

Installing Xen basic packages in Debian

First of all, update the APT repository

 apt-get update

Xen Hypervisor

Install the Xen hypervisor. Note that the version may have changed: search for the current one using 'apt-cache search xen-hypervisor'.

 apt-get install xen-hypervisor-3.0.3-1-i386

In the case you get "CPU0 panic" errors when booting this hypervisor, try with the PAE version instead:

 apt-get install xen-hypervisor-3.0.3-1-i386-pae

Install the Xen kernel

 apt-get install linux-image-xen-686

Install Xen tools

 apt-get install xen-tools

Install the libc6-xen

It is not mandatory, but very recommendable (otherwise, you will get "4gb seg fixup" error messages in dom0):

 apt-get install libc6-xen

Tunning Xen

Grub boot configuration

Kernel boot configuration should be upgraded when installing linux-image-xen-686 package. However, this automatic upgrade doesn't seem to work properly, so manual editing of /boot/grub/menu.lst is required (Debian has shown this behaviour; however, other distributions, like Ubuntu, seems not being suffering it).

Considering the linux-image-xen-686 installed /boot/vmlinuz-2.6.18-4-xen-686 (exact version may change, so check it), you should configure the following booting option in your /boot/grub/menu.lst file:

  title           Debian GNU/Linux, kernel 2.6.18-4-xen-686
  root            (hd0,0)
  kernel          /xen-3.0.3-1-i386-pae.gz
  module          /vmlinuz-2.6.18-4-xen-686 root=/dev/sda2 ro
  module          /initrd.img-2.6.18-4-xen-686
  savedefault
  boot

Configure networking

I needed to edit /etc/xen/xend-config.sxp and configure the network-script option, in order virtual machines get directly bridged to the external interface (eth1 in my case, change it appropriately):

  (network-script 'network-bridge netdev=eth1')

Configure loop

If you want to run many virtual machines simultaneously (more than 8) you have to use max_loop option for loop module (http://lists.xensource.com/archives/html/xen-users/2006-06/msg01150.html). For example:

   rmmod loop
   modprobe loop max_loop=64

You can also configure as default option for modprobe, editing /etc/modprobe.d/options and adding the following line. If the file doesn't exist, create it. (Thanks to Ramon Casellas for the tip :)

  options loop max_loop=64

Creating a GNU/Linux (para)virtual machine in domU

Use the following configuration file (let's say test.cfg):

   kernel  = '/boot/vmlinuz-2.6.18-4-xen-686'
   ramdisk = '/boot/initrd.img-2.6.18-4-xen-686'
   memory  = '50'
   root    = '/dev/sda1 ro'
   disk    = [ "file:/usr/share/vnuml/filesystems/xen_fs,sda1,w" ]
   name    = 'occ0-v'
   vif     = [ '' ]
   on_poweroff = 'destroy'
   on_reboot   = 'restart'
   on_crash    = 'restart'

Comments:

• You can use the VNUML filesystems as xen_fs. It is highly recommendable to install kernel modules before. The commands are (exact version may change, so check it):

  mount /usr/share/vnuml/filesystems/xen_fs /mnt/loop -o loop
  cd /mnt/loop/lib/modules
  cp -va /lib/modules/2.6.18-4-xen-686 .
  cd /
  umount /mnt/loop

Warning:: differently from the usual configuration with VNUML, this filesystem doesn't work in COW mode, all modification performed during virtual machine operation will be stored in the xen_fs. In order to avoid changes in the root_fs file, I recommend you to use a copy ('cp root_fs xen_fs') and not to use directly the one you are using for your COW-ed based VNUML scenarios.

• The vif line configures a network interface in the virtual machine (named 'eth0'). I haven't already checked, but it is supposed that additional vif lines will add additional interfaces (eth1, eth2, etc.
• This configuration uses 50MB of memory
• This configuration does not use swap memory

In order to boot the machine (note that you don't have to specify the complete pathname of test.cfg if it is located in a default directory; it seems that xm looks in /etc/xen by default).

 xm create test.cfg -c

It also can take a while to boot the virtual machine. The '-c' is to get the booting output and loging in the same console where xm is invoked (this is very useful to know when the virtual machine has been booted). To come back to the host enviroment just press Ctrl+].

Creating a Windows XP (full)virtual machine in HVM

Based on: http://mediakey.dk/~cc/howto-install-windows-xp-vista-on-xen/

Before starting, check that your CPU supports virtualization extensions (otherwise, HVM can not work). In order to do so:

 egrep '^flags.*(vmx|svm)' /proc/cpuinfo

In addition, check that the virtualization extension is enabled in the BIOS (in my case, with a Dell Latitute D630, I needed to do it).

1. Install some additional packages (appart of the Xen based ones, described above)

 apt-get install xen-ioemu-3.0.3-1

2. Get the .iso image of the Windows XP cd (assuming your CD drive is /dev/hda, otherwise set the right device):

 dd if=/dev/hda of=/data/bigthings/WinxpSP2.iso

3. Create the virtual disk to hold the HVM (the example used 2GB):

 dd if=/dev/zero of=/data/bigthings/xentests/win_vm.img bs=1M count=2048

4. Prepare the Xen configuration file (let's say win.cfg) for the virtual machine:

 kernel = '/usr/lib/xen-3.0.3-1/boot/hvmloader'
 builder = 'hvm'
 memory = '256'
 device_model='/usr/lib/xen-3.0.3-1/bin/qemu-dm'
 
 # Disks
 disk = [ 'file:/data/bigthings/xentests/win_vm.img,ioemu:hda,w',
 'file:/data/bigthings/WinxpSP2.iso,ioemu:hdc:cdrom,r' ]
 
 # Hostname
 name = 'win01'
 
 # Networking
 vif = ['type=ioemu, bridge=xenbr0']
 
 # Behaviour
 boot='d'
 vnc=1
 vncviewer=1
 vncunused=0
 sdl=0

(Note: the name of the Xen bridge is not always xenbr0. For example, in my system is xenbr2. You need to change the vif parameter accordingly. In order to know which the name of the bridge for your system use 'brctl show').

5. Create the vm:

 xm create win.cfg

6. Open a VNC session. The port to use is 5900+n, where n is the domain ID for the virtual machine (that you can find out with 'xm list'). For example, if the domain ID is 12, use:

 krdc 127.0.0.1:5912

(krdc is just one of the many existing VNC clients, some others are TightVNC and RealVNC)

Alternatively, you can change to 'vncunused=1', in which case the VNC port is an arbitrary one above 5900 (usually is the 5900 itself). However, I don't recommend to use this, because I've found problems when creating several concurrent HVM machines.

7. Install Windows as usual using its installer! :)

If you have offset problems with the mouse (see http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=585), then use

 usbdevice="tablet"

in the configuration file.

Uselful management operations

• To list existing domains:

 xm list

• To connect to the virtual machine console (doesn't work for HVM, use VNC instead):

 xm console <domU>

or (combining with xterm, if you are using a graphical desktop in domO):

 xterm -T vm-name -e xm console <domU>

• To finish with the virtual machine:

 xm destroy <domU>

or (preferable)

 xm shutdown <domU>

• To get performance info:

 xm top

Troubleshooting

• If you have problems related with memory when booting virtual machines, you may find useful the dom0_mem kernel switch. Edit your GRUB configuration so your kernel line looks like this:

 kernel /boot/xen-3.0.3-1-i386.gz dom0_mem=512M

changing 512 for the desired memory in your case, of course. Note that if you use a dom0_mem value too close to your actual physical RAM memory you will get a panic error when booting the host: as a hint dom0_mem<= phy_mem-32M (assuming the Xen hypervisor consumes less than 32MB).

• If you run Xen in a VMware virtual machine using NAT, you should use an special MAC prefix for your Xen virtual machine: 00:0C:29. Edit the /etc/xen/test.cfg file and edit the vif line so looks (for example):

 vif  = [ 'mac=00:0C:29:11:22:33' ]

the problem is quite similar to the one described in the following post in the vnuml-users list: https://lists.dit.upm.es/pipermail/vnuml-users/2005-September/000204.html

• If you get the following error when creating a new virtual machine:

 Error: I need 65536 KiB, but dom0_min_mem is 200704 and shrinking to 200704 
 KiB would leave only 43236 KiB free.

This happens because of the sum of memory assigned to virtual machines is exceeding the limit reserve to dom0. To reduce that limit, edit the dom0-min-mem parameter in /etc/xen/xend-config.sxp file. For example, in order to use a 96MB limit:

  (dom0-min-mem 96)

You need to restart xend in order changes in /etc/xen/xend-config.sxp take effect:

  /etc/init.d/xend restart

• Virtual machines seem to work properly (i.e., you can use 'xm console' to access the domU and issue commands without problems), but after a while (even at the very beginning of create them) network start to fail and they loss connectivity. In addition, the following error message start to appear in /var/log/messages in dom0:

 xen_net: Memory squeeze in netback driver.

After some googling and dead ends (see the references gathered in the bug report at http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=762), the final solution was found in the following thread: http://lists.xensource.com/archives/html/xen-users/2007-01/msg00428.html, although it could not work for all cases (see comment #7 in the previously cited bugzilla report).

In conclusion, you have to use dom0_mem hypervisor boot switch (see the first bullet in this section). If you are using dom0-min-mem (see previous bullet), then make dom0-min-mem = dom0_mem.

• Strange messages appear in the virtual machine console:

 4gb seg fixup, process syslogd (pid 1107), cs:ip 73:b7efc5f0
 4gb seg fixup, process syslogd (pid 1107), cs:ip 73:b7efc570
 4gb seg fixup, process klogd (pid 1113), cs:ip 73:b7e2e5f0
 4gb seg fixup, process klogd (pid 1113), cs:ip 73:b7eafd10
 printk: 38 messages suppressed.
 4gb seg fixup, process named (pid 1128), cs:ip 73:b7bc5c1c

The problem is described in the Xen FAQ. It is not only a "aesthetic" issue: it impacts performance.

This seems to be due processes running in domU are using a wrong version of libc:

 domU:~# ldd /sbin/init | grep libc.so.6
         libc.so.6 => /lib/tls/libc.so.6 (0xb7e0f000)

The solution is to install a right version of libc in the domU filesystem (not only in dom0 filesystem, as described above), in particular the libc6-xen package.

After that, you can check that now processes are using the right version of libc in domU:

 domU:~# ldd /sbin/init  | grep libc.so.6
         libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7e36000)

• If you get the following error when trying to create a domU:

 Error: Device 0 (vif) could not be connected. Backend device not found.

and your /var/log/xen/xend-debug.log shows something complaining about brctl, like that:

 /etc/xen/scripts/network-bridge: line 114: brctl: command not found

it seems that bridge-utils package is not installed in your host, so:

 apt-get install bridge-utils
 /etc/init.d/xend restart

should solve the problem.

• If you are using VMware products, they may interfere with Xen networking. If you are having problems with xen bridge or vif devices setting up, try to stop VMware networking and restar the xend daemon before trying again:

 /etc/init.d/vmware stop
 /etc/init.d/xend restart

• If you are having problems setting the Xen bridge (usually named xenbr0 or xenbr1) in a host with two (or more) physical interfaces, try to set down all except one and restart the xend again.
• If you get the following error when trying to create a domU (note that it's similar to the former bullet, but different):

  Error: Device 2049 (vbd) could not be connected. Backend device not found.

you maybe have not load the loop module. Use 'lsmod | grep loop' to check it. In that case, load loop ('modprobe loop') and try again.

Uselful references

• http://www.debian-administration.org/articles/396
• http://wiki.xensource.com/xenwiki/XenNetworking