pervasive logo
Virtualization on a Server Farm

Server Setup (last update 02.12.2009,14:07)

Starting point, we got four new Sun Fire x2100 plates. The goal was to setup a system which is robust and stable and that can be used for a large field of applications. For reliability reasons the server should have RAID 1. To support different kind of applications, which means also different kind of operating systems, we must install a virtualization software. For that XEN and QEMU got installed. We need QEMU because our Dual Core AMD Opteron(tm) Processor 180 does not support the vanderpool extension, but we also want to have a Windows System working. Try this to find out if the Opteron support vanderpool:

 cat /proc/cpuinfo | grep svm
in our case it gave no result.

Server Host System

The Host System is a Slackware 12.0 [Slackware website]. Download the DVD iso image burn it on dvd and boot the server with it. The first step during the install process is to prepare the hard-disks. In our case we had two SATA Disks with 80GB. We used cfdisk and mdadm to configure our RAID 1. Attention the Kernel must have RAID support. The default installation Kernel (2.6.21.5-smp) is ok. Our configuration looks like:

 cfdisk /dev/sda
and this is the result:
                          Disk Drive: /dev/sda
                    Size: 80026361856 bytes, 80.0 GB
          Heads: 255   Sectors per Track: 63   Cylinders: 9729

   Name       Flags     Part Type FS Type        [Label]       Size (MB)
 ----------------------------------------------------------------------
   sda1                  Primary  Linux raid autodetect         12000.69
   sda2                  Primary  Linux raid autodetect         12000.69
   sda3                  Primary  Linux raid autodetect         12000.69
   sda5                  Logical  Linux raid autodetect         29997.60
   sda6                  Logical  Linux raid autodetect         10001.95
   sda7                  Logical  Linux swap                     4022.17
The same configuration we used for /dev/sdb. To save time we copy the disk layout with sfdisk to our second disk:
 sfdisk -d /dev/sda |sfdisk /dev/sdb
Now we have two identical disks. To setup our RAID 1 we use the mdadm command line tool. By using the following commands:
 mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sd[ab]1
 mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sd[ab]2
 mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/sd[ab]3
 mdadm --create /dev/md3 --level=1 --raid-devices=2 /dev/sd[ab]5
 mdadm --create /dev/md4 --level=1 --raid-devices=2 /dev/sd[ab]6
Details about our RAID device we can found with the following:
 mdadm -D /dev/md0 # or md[0-6]
Now we continue our installation. In the setup menu we have to choose our raid devices as install destinations, let the setup format the raid devices with a file system. We use reiserfs because it is faster when handling a lot of little files. Now we need to add our raid configuration to the mdadm.conf file:
 mdadm -D --scan >> /mnt/etc/mdadm.conf
Our raid device file system destination mapping looks like:
 /dev/md/0            /
 /dev/md/3            /data
 /dev/md/4            /home
With the md1 and md2 device we will create later a LVM volume.

....TODO....

Setup Boot loader

We use the slackware default boot loader which is LILO. After finishing setup we have to perform following steps:
 chroot /mnt
 vi /etc/lilo.conf
add following lines into the config file
 boot = /dev/md0
 raid-extra-boot = "/dev/sda, /dev/sdb"
and execute lilo. If our RAID breaks we can boot our system from both disks. Now we can reboot! Detail information to the slackware installation you can find here [Slackbook website]. Another howto for Slackware + RAID you can find here [userlocal.com website]

Prepare a LVM partition for XEN

We need several steps to setup a LVM Volume. First we initializes Physical Volumes for a later usage in the the Logical Volume Manager (LVM). For that we use our still undefined d devices md1 and md2.
 pvcreate /dev/md1
 pvcreate /dev/md2
With vgcreate we create new volume group called "xenlvm" and add our prepared devices
 vgcreate xenlvm /dev/md1 /dev/md2
Now we can create the Volume Group. we must define the size, the name of the Logical Volume and the Volume Group.
 lvcreate -n logv1 -L 22G xenlvm
At least we create a reiser file system on this partition:
 mkreiserfs /dev/xenlvm/logv1
If a test mount works successfully we can include this device in our /etc/fstab:
/dev/sda7        swap             swap        defaults         0   0
/dev/sdb7        swap             swap        defaults         0   0
/dev/md0         /                reiserfs    defaults         1   1
/dev/md3         /data            reiserfs    defaults         1   2
/dev/md4         /home            reiserfs    defaults         1   2
/dev/cdrom      /mnt/cdrom       auto        noauto,owner,ro  0   0
/dev/fd0         /mnt/floppy      auto        noauto,owner     0   0
devpts           /dev/pts         devpts      gid=5,mode=620   0   0
proc             /proc            proc        defaults         0   0
/dev/xenlvm/logv1       /mnt/xen        reiserfs    defaults   0   0

Network device bonding

Aggregating multiple network links into a single link, using a variety of methods. Before we can start to configure we have to build as a binary:
 cd /usr/src/linux/Documentation/networking
 gcc -Wall -O -I/usr/src/linux/include ifenslave.c -o ifenslave
 cp ifenslave /sbin/ifenslave
We need the kernel module bonding and the module for our network card in our case tg3. Now we can create a init script that make a bond device for our two GBit NICs, then place them under "/etc/rc.d/rc.bond":
 #!/bin/sh
 #
        case "$1" in
          'start')
            echo "start bond0"
            #modprobe bonding mode=balance-alb miimon=100
            modprobe bonding mode=balance-rr miimon=100
            modprobe tg3
            ifconfig bond0 up
            ifenslave bond0 eth0
            ifenslave bond0 eth1
            #TODO need to be changed
            ifconfig bond0 hw ether 00:16:3e:aa:aa:aa
          ;;
          'stop')
            ifconfig bond0 down
            rmmod bonding
            rmmod tg3
          ;;
          *)
            echo "Usage: $0 {start|stop}"
          ;;
        esac
The different kinds of bonding methods we can add as parameter to the bonding module. More under [Bonding wiki website]. Attention for the MAC Address! If we do not use the "hw ether" parameter the MAC Address will be automatically take from the first network card. We use the static method because we got problems with the Xen Bridge when we used the dynamic mode.

UPDATE: Bonding

We have to change the bonding mode otherwise we run into ARP network problems with the xen bridge. More about the the problem you can find [here]. The solution is disable the bonding or (i tried) use mode=1 (active-backup). But this is only a fault tolerance mode.

That we can use the standard network configuration files under slackware we add the following into "/etc/rc.d/rc.M" but before "rc.inet1" call:
 ..
 # NEW START
 # dominik say start bonding first
 if [ -x /etc/rc.d/rc.bond ]; then
   . /etc/rc.d/rc.bond start
 fi 
 # NEW END

 # Initialize the networking hardware.
 if [ -x /etc/rc.d/rc.inet1 ]; then
   . /etc/rc.d/rc.inet1
 fi
 ..
To configure our bond device we use the "/etc/rc.d/rc.inet1.conf" file:
 # Bond
 IFNAME[4]="bond0"
 IPADDR[4]="XXX.XX.XX.XX"
 NETMASK[4]="255.255.255.0"
 USE_DHCP[4]=""
 DHCP_HOSTNAME[4]=""

 # Default gateway IP address:
 GATEWAY="XXX.XX.XX.XX"
To get the status of the bond device use the following command:
 cat /proc/net/bonding/bond0
result:
 MII Status: up
 MII Polling Interval (ms): 100
 Up Delay (ms): 0
 Down Delay (ms): 0

 Slave Interface: eth0
 MII Status: up
 Link Failure Count: 0
 Permanent HW addr: 00:e0:81:5e:9e:c4

 Slave Interface: eth1
 MII Status: up
 Link Failure Count: 0
 Permanent HW addr: 00:e0:81:5e:9e:c5

Keep our system up to date

We use the Slackware package management tool slapt-get. Download the latest version from [slapt-get website]:
 wget http://software.jaos.org/slackpacks/12.0/slapt-get-0.9.12-i386-1.tgz
 installpkg slapt-get-0.9.12-i386-1.tgz
Add your preferred slackware mirror into "/etc/slapt-get/slapt-getrc":
 WORKINGDIR=/var/slapt-get
 EXCLUDE=^kernel-.*,^alsa-.*,^glibc.*,.*-[0-9]+dl$,^devs$,^udev$,aaa_elflibs,x86_64
 SOURCE=http://gd.tuwien.ac.at/opsys/linux/slackware/slackware-12.0/
 slapt-get --update
 slapt-get --upgrade
After installation we upgrade the latest changes, especially the security fixes. To keep our server also in future up to date we install a mail notification. We use the Perl script from Joran Kvalvaag:
 wget http://www.nerdworks.org/download/scripts/update-notifier/update-notifier
 cp /etc/cron.daily/rc.slackware-update-notifier
We set our Email address into the script. If new slackware packages are available we get a notification

Apache and MySQL

The following instruction; we can use in the host system (dom0) or inside our XEN guests! We use the slackware MySQL Server package. Follow the installation instructions:
 vim /etc/rc.d/rc.mysqld

 #modify 
 #SKIP="--skip-networking"
 
 su - mysql #as root
 mysql_install_db
 exit
 chmod +x /etc/rc.d/rc.mysqld
 /etc/rc.d/rc.mysqld start
 
 # look if mysql is running
 ps -aef | grep mysql

 /usr/bin/mysqladmin -u root password 'password'
After MySQL configuration make the init script executable and start them
 chmod +x /etc/rc.d/rc.mysqld
 /etc/rc.d/rc.mysqld start
The Apache web server we install from source. Download the latest version from the [Apache website]. We use httpd-2.2.4 with some modules:
 cd httpd-2.2.4
 ./configure --enable-proxy --enable-ssl --enable-rewrite --enable-dav --enable-info
 make 
 make install

 slapt-get --remove httpd-2.2.4-i486-6
 ln -s /usr/local/apache2/bin/apachectl /usr/sbin/apachectl
 chmod +x /etc/rc.d/rc.httpd
 /etc/rc.d/rc.httpd start
In the future i will use these servers for a Apache Web Cluster so i have to add some relevant things as modperl or embperl:
 cpan
 cpan[1]> install ModPerl::PerlRun
 cpan[1]> install Embperl
Add the Perl Modules to the apache configuration file and restart the web server:
 vim /usr/local/apache2/conf/httpd.conf
 # Example:
 # LoadModule foo_module modules/mod_foo.so
 #
 LoadModule perl_module modules/mod_perl.so
 LoadModule embperl_module /usr/lib/perl5/site_perl/5.8.8/i486-linux-thread-multi/auto/Embperl/Embperl.so

XEN Installation on Slackware

First we download the latest Sources from Xen [Xen website] and then we build the Xen Kernel:
 wget http://bits.xensource.com/oss-xen/release/3.1.0/src.tgz/xen-3.1.0-src.tgz
 tar xzf xen-3.1.0-src.tgz 
 cd xen-3.1.0-src
 make world
 make install
We need some extra options in our Xen Kernel so we re- configure the kernel and rebuild them in the following way:
 make linux-2.6-xen-config CONFIGMODE=menuconfig     #or xconfig
 make linux-2.6-xen-build
 make linux-2.6-xen-instal
In our case we need to rebuild the Kernel because we must add our SATA device drivers.
Before we can use the Xen Kernel we have to replace our bootloader because XEN is not working with LILO so we need to install GRUB from our slackware dvd:
 cd /mnt/[DVD]/extra/grub
 installpkg grub-0.97-i486-3.tgz 
We install now on both disks a MBR with the following:
 Grub>device (hd0) /dev/sda
 Grub>root (hd0,0)
 Grub>setup (hd0)
 
 Grub>device (hd0) /dev/sdb
 Grub>root (hd0,0) 
 Grub>setup (hd0)
Under "/boot/grub" we but the "menu.1st" file:
 # Boot automatically after 5 secs.
 timeout 5
 #
 # By default, boot the first entry.
 default 0
 #
 # Fallback to the second entry.
 fallback 1
 #
 # For booting with disc 0 kernel

 title Xen 3.1 on Slackware 12
 root (hd0,0)
 kernel /boot/xen.gz noreboot
 module /boot/vmlinuz-2.6-xen root=/dev/md0 ro console=tty0

 title Slackware 12 
 root (hd0,0)
 kernel /boot/vmlinuz ro root=/dev/md0
Now we can reboot our System and boot the Xen Kernel based Slackware. During the boot sequence the following message are seen:
 ***************************************************************
 ** WARNING: Currently emulating unsupported memory accesses **
 ** in /lib/tls glibc libraries. The emulation is **
 ** slow. To ensure full performance you should **
 ** install a 'xen-friendly' (nosegneg) version of **
 ** the library, or disable tls support by executing **
 ** the following as root: **
 ** mv /lib/tls /lib/tls.disabled **
 ** Offending process: init (pid=1) **
 ***************************************************************
Slackware does not have a "/lib/tls". The problem is we need a Xen compatible glibc otherwise we loose huge performance . In detail the Native POSIX Thread Library, Definition: [NPTL wikipedia website] need to be reconfigured. We should rebuild our glibc with the following "-mno-tls-direct-seg-refs" option. For that we copy as the glibc sources from the dvd and add the CFLAGS option in the "glibc.SlackBuild" file:
 # old
 # CFLAGS="-g $OPTIMIZ" \
 # new
 CFLAGS="-g $OPTIMIZ -mno-tls-direct-seg-refs" \
After that we execute the build file and install the created slackware package.
 ./glibc.SlackBuild
 installpkg glibc-2.5-i486-4.tgz
 reboot
With our new glibc we rebuild the sysvinit and the mdadm package. We copy the sources from the packages and call again the build script and install the new created packages.

UPDATE:Kernel headers

If the kernel headers version does not match your slackware glibc source package - install the kernel headers under a temp. destination like /tmp/kernel-headers
installpkg -root /tmp/kernel-headers kernel-headers-2.6.27.7_smp-x86-1.tgz
and add the following to the CFLAGS in the glibc.SlackBuild file
#  --with-headers=/usr/src/linux-${KERNEL_HEADERS}/include \
  --with-headers=/tmp/kernel-headers/usr/include \

UPDATE:Upgrade to slackware 12.2

Blacklist the coreutils if you use the 2.6.18 xen kernel. New version dont work together with a too old kernels.

You can download xen slackware 12.0 glibc package here: [xenbuild.tar.gz]
download unpack and:
 cd xenbuild/pkg
 installpkg glibc-2.5-i486-4.tgz
To configure xen daemon xend use "/etc/xen/xend-config.sxp". A good description you can find here [website](German). For the network configuration we decide to use bridging. Here we need to replace the "network-bridge" with the "network-bridge-bonding" you can find them in the Xen users Archives [link]. Copy them under "/etc/xen/scripts/" and replace in "/etc/xen/xend-config.sxp":
 #old
 #(network-script network-bridge)
 #new
 (network-script network-bridge-bonding)
Now we start the daemon with the command "xend". The next step is to prepare our Xen for automatic startup after reboot. First we add the xen domains starter script to our "/etc/rc.d/":
 cp /[XEN-SOURCE-DIR]/dist/install/etc/init.d/xendomains /etc/rc.d/rc.xendomains
 chmod +x /etc/rc.d/rc.xendomains
Now we modify our "/etc/rc.d/rc.local":
 # start xend
 if [ `uname -r | grep -c 'xen'` -eq "1" ]; then
        echo start xen
        xend start
        echo start autostarter for xendomains
        /etc/rc.d/rc.xendomains start
 else
        echo xen not started 
 fi

UPDATE: Xen Network via Proxy ARP (routing)

Because i used device bonding i get problems with xen-bridged-mode (read more at the xen forum). So i decided to change to routing mode via Proxy ARP. But here i run into problems because whenever i moved one xen guest from one xen server to the other, this xen guest was inaccessible from outside (network behinde the router). The reason was that the router in my network make an ARP Cache update every 300min. And i have no access to this router!

The solution is:

Shutdown the moved xen guest on his new xen server. But remember the IP address(es) which the xen guest use. And follow this procedure.
IP=X.X.X.X (the IP Address form the xen guest)
IP_ROUTER=X.X.X.X
ifconfig bond0:0 $IP netmask 255.255.255.0
arping -U -Ibond0:0 -s $IP  $IP_ROUTER

#output:
#Unicast reply from X.X.X.X [00:.....]  1.820ms
#.
#.
#.

ifconfig bond0:0 down
Boot the xen guest and the host will be accessible from outside!

Setup Xen guests

Under our lvm mount "/mnt/xen" we crete two directories
 mkdir /mnt/xen/images
 mkdir /mnt/xen/mount

Build a Debian guest

Build image files for our guest:
 dd if=/dev/zero of=/mnt/xen/images/guest-debian-4.img bs=1024k count=5000

 dd if=/dev/zero of=/mnt/xen/images/guest-debian-4-swap.img bs=1024k count=500

 mkreiserfs -f /mnt/xen/images/guest-debian-4.img
 mkswap /mnt/xen/images/guest-debian-4-swap.img
Mount the image file in our filesystem:
 mount -o loop /mnt/xen/images/guest-debian-4.img /mnt/xen/mount
Install debian with debootstrap. To build a debootstrap for Slackware we download the debian package debootstrap_0.3.3.2_all.deb from a debian mirror and prepare them for usage under slackware:
 ar x debootstrap_0.3.3.2_all.deb
 mv data.tar.gz debootstrap_0.3.3.2_all.tgz
 installpkg debootstrap-0.2.45-i486-1.tgz
Now use debootstrap with the following to install debian:
 debootstrap --verbose --arch i386 etch /mnt/xen/mount/debian-4/ http://ftp.de.debian.org/debian

Duplicate to the other servers

For that is used two Knoppix DVDs. I booted both servers with the knoppix dvds. Put both servers in a private class C network.
 ifconfig eth0 192.168.0.1 netmask 255.255.255.0 # installed one
 ifconfig eth0 192.168.0.2 netmask 255.255.255.0 # new one
We use dd to mirror the data. On the empty server machine we start a sshd deamon. On the already installed server we use the following command:
 dd if=/dev/sda | root@192.168.0.2 "dd of=/dev/sda"
After a successfull data copy we can boot our new server! To put the second sata disk into the RAID we do the following:
 sfdisk -d /dev/sda |sfdisk /dev/sdb
 mdadm /dev/md0 -a /dev/sdb1
 mdadm /dev/md1 -a /dev/sdb2
 mdadm /dev/md2 -a /dev/sdb3
 mdadm /dev/md3 -a /dev/sdb5
 mdadm /dev/md4 -a /dev/sdb6