Virtualization on a Server Farm
Server Setup (last update 02.12.2009,14:07)
Starting point, we got four new Sun Fire x2100 plates. The goal was to setup a system which is robust and stable and that can be used for a large field of applications. For reliability reasons the server should have RAID 1. To support different kind of applications, which means also different kind of operating systems, we must install a virtualization software. For that XEN and QEMU got installed. We need QEMU because our Dual Core AMD Opteron(tm) Processor 180 does not support the vanderpool extension, but we also want to have a Windows System working. Try this to find out if the Opteron support vanderpool:
cat /proc/cpuinfo | grep svmin our case it gave no result.
Server Host System
The Host System is a Slackware 12.0 [Slackware website]. Download the DVD iso image burn it on dvd and boot the server with it. The first step during the install process is to prepare the hard-disks. In our case we had two SATA Disks with 80GB. We used cfdisk and mdadm to configure our RAID 1. Attention the Kernel must have RAID support. The default installation Kernel (2.6.21.5-smp) is ok. Our configuration looks like:
cfdisk /dev/sdaand this is the result:
Disk Drive: /dev/sda
Size: 80026361856 bytes, 80.0 GB
Heads: 255 Sectors per Track: 63 Cylinders: 9729
Name Flags Part Type FS Type [Label] Size (MB)
----------------------------------------------------------------------
sda1 Primary Linux raid autodetect 12000.69
sda2 Primary Linux raid autodetect 12000.69
sda3 Primary Linux raid autodetect 12000.69
sda5 Logical Linux raid autodetect 29997.60
sda6 Logical Linux raid autodetect 10001.95
sda7 Logical Linux swap 4022.17
The same configuration we used for /dev/sdb. To save time we copy the disk layout with
sfdisk to our second disk:
sfdisk -d /dev/sda |sfdisk /dev/sdbNow we have two identical disks. To setup our RAID 1 we use the mdadm command line tool. By using the following commands:
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sd[ab]1 mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sd[ab]2 mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/sd[ab]3 mdadm --create /dev/md3 --level=1 --raid-devices=2 /dev/sd[ab]5 mdadm --create /dev/md4 --level=1 --raid-devices=2 /dev/sd[ab]6Details about our RAID device we can found with the following:
mdadm -D /dev/md0 # or md[0-6]Now we continue our installation. In the setup menu we have to choose our raid devices as install destinations, let the setup format the raid devices with a file system. We use reiserfs because it is faster when handling a lot of little files. Now we need to add our raid configuration to the mdadm.conf file:
mdadm -D --scan >> /mnt/etc/mdadm.confOur raid device file system destination mapping looks like:
/dev/md/0 / /dev/md/3 /data /dev/md/4 /homeWith the md1 and md2 device we will create later a LVM volume.
....TODO....
Setup Boot loader
We use the slackware default boot loader which is LILO. After finishing setup we have to perform following steps:chroot /mnt vi /etc/lilo.confadd following lines into the config file
boot = /dev/md0 raid-extra-boot = "/dev/sda, /dev/sdb"and execute lilo. If our RAID breaks we can boot our system from both disks. Now we can reboot! Detail information to the slackware installation you can find here [Slackbook website]. Another howto for Slackware + RAID you can find here [userlocal.com website]
Prepare a LVM partition for XEN
We need several steps to setup a LVM Volume. First we initializes Physical Volumes for a later usage in the the Logical Volume Manager (LVM). For that we use our still undefined d devices md1 and md2.pvcreate /dev/md1 pvcreate /dev/md2With vgcreate we create new volume group called "xenlvm" and add our prepared devices
vgcreate xenlvm /dev/md1 /dev/md2Now we can create the Volume Group. we must define the size, the name of the Logical Volume and the Volume Group.
lvcreate -n logv1 -L 22G xenlvmAt least we create a reiser file system on this partition:
mkreiserfs /dev/xenlvm/logv1If a test mount works successfully we can include this device in our /etc/fstab:
/dev/sda7 swap swap defaults 0 0 /dev/sdb7 swap swap defaults 0 0 /dev/md0 / reiserfs defaults 1 1 /dev/md3 /data reiserfs defaults 1 2 /dev/md4 /home reiserfs defaults 1 2 /dev/cdrom /mnt/cdrom auto noauto,owner,ro 0 0 /dev/fd0 /mnt/floppy auto noauto,owner 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 proc /proc proc defaults 0 0 /dev/xenlvm/logv1 /mnt/xen reiserfs defaults 0 0
Network device bonding
Aggregating multiple network links into a single link, using a variety of methods. Before we can start to configure we have to build as a binary:cd /usr/src/linux/Documentation/networking gcc -Wall -O -I/usr/src/linux/include ifenslave.c -o ifenslave cp ifenslave /sbin/ifenslaveWe need the kernel module bonding and the module for our network card in our case tg3. Now we can create a init script that make a bond device for our two GBit NICs, then place them under "/etc/rc.d/rc.bond":
#!/bin/sh
#
case "$1" in
'start')
echo "start bond0"
#modprobe bonding mode=balance-alb miimon=100
modprobe bonding mode=balance-rr miimon=100
modprobe tg3
ifconfig bond0 up
ifenslave bond0 eth0
ifenslave bond0 eth1
#TODO need to be changed
ifconfig bond0 hw ether 00:16:3e:aa:aa:aa
;;
'stop')
ifconfig bond0 down
rmmod bonding
rmmod tg3
;;
*)
echo "Usage: $0 {start|stop}"
;;
esac
The different kinds of bonding methods we can add as parameter to the bonding module. More
under [Bonding wiki website].
Attention for the MAC Address! If we do not use the "hw ether" parameter the MAC Address
will be automatically take from the first network card. We use the static method because
we got problems with the Xen Bridge when we used the dynamic mode.UPDATE: Bonding
We have to change the bonding mode otherwise we run into ARP network problems with the xen bridge. More about the the problem you can find [here]. The solution is disable the bonding or (i tried) use mode=1 (active-backup). But this is only a fault tolerance mode.That we can use the standard network configuration files under slackware we add the following into "/etc/rc.d/rc.M" but before "rc.inet1" call:
.. # NEW START # dominik say start bonding first if [ -x /etc/rc.d/rc.bond ]; then . /etc/rc.d/rc.bond start fi # NEW END # Initialize the networking hardware. if [ -x /etc/rc.d/rc.inet1 ]; then . /etc/rc.d/rc.inet1 fi ..To configure our bond device we use the "/etc/rc.d/rc.inet1.conf" file:
# Bond IFNAME[4]="bond0" IPADDR[4]="XXX.XX.XX.XX" NETMASK[4]="255.255.255.0" USE_DHCP[4]="" DHCP_HOSTNAME[4]="" # Default gateway IP address: GATEWAY="XXX.XX.XX.XX"To get the status of the bond device use the following command:
cat /proc/net/bonding/bond0result:
MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth0 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:e0:81:5e:9e:c4 Slave Interface: eth1 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:e0:81:5e:9e:c5
Keep our system up to date
We use the Slackware package management tool slapt-get. Download the latest version from [slapt-get website]:wget http://software.jaos.org/slackpacks/12.0/slapt-get-0.9.12-i386-1.tgz installpkg slapt-get-0.9.12-i386-1.tgzAdd your preferred slackware mirror into "/etc/slapt-get/slapt-getrc":
WORKINGDIR=/var/slapt-get EXCLUDE=^kernel-.*,^alsa-.*,^glibc.*,.*-[0-9]+dl$,^devs$,^udev$,aaa_elflibs,x86_64 SOURCE=http://gd.tuwien.ac.at/opsys/linux/slackware/slackware-12.0/
slapt-get --update slapt-get --upgradeAfter installation we upgrade the latest changes, especially the security fixes. To keep our server also in future up to date we install a mail notification. We use the Perl script from Joran Kvalvaag:
wget http://www.nerdworks.org/download/scripts/update-notifier/update-notifier cp /etc/cron.daily/rc.slackware-update-notifierWe set our Email address into the script. If new slackware packages are available we get a notification
Apache and MySQL
The following instruction; we can use in the host system (dom0) or inside our XEN guests! We use the slackware MySQL Server package. Follow the installation instructions:vim /etc/rc.d/rc.mysqld #modify #SKIP="--skip-networking"
su - mysql #as root mysql_install_db exit chmod +x /etc/rc.d/rc.mysqld /etc/rc.d/rc.mysqld start # look if mysql is running ps -aef | grep mysql /usr/bin/mysqladmin -u root password 'password'After MySQL configuration make the init script executable and start them
chmod +x /etc/rc.d/rc.mysqld /etc/rc.d/rc.mysqld startThe Apache web server we install from source. Download the latest version from the [Apache website]. We use httpd-2.2.4 with some modules:
cd httpd-2.2.4 ./configure --enable-proxy --enable-ssl --enable-rewrite --enable-dav --enable-info make make install slapt-get --remove httpd-2.2.4-i486-6 ln -s /usr/local/apache2/bin/apachectl /usr/sbin/apachectl chmod +x /etc/rc.d/rc.httpd /etc/rc.d/rc.httpd startIn the future i will use these servers for a Apache Web Cluster so i have to add some relevant things as modperl or embperl:
cpan cpan[1]> install ModPerl::PerlRun cpan[1]> install EmbperlAdd the Perl Modules to the apache configuration file and restart the web server:
vim /usr/local/apache2/conf/httpd.conf # Example: # LoadModule foo_module modules/mod_foo.so # LoadModule perl_module modules/mod_perl.so LoadModule embperl_module /usr/lib/perl5/site_perl/5.8.8/i486-linux-thread-multi/auto/Embperl/Embperl.so
XEN Installation on Slackware
First we download the latest Sources from Xen [Xen website] and then we build the Xen Kernel:wget http://bits.xensource.com/oss-xen/release/3.1.0/src.tgz/xen-3.1.0-src.tgz tar xzf xen-3.1.0-src.tgz cd xen-3.1.0-src make world make installWe need some extra options in our Xen Kernel so we re- configure the kernel and rebuild them in the following way:
make linux-2.6-xen-config CONFIGMODE=menuconfig #or xconfig make linux-2.6-xen-build make linux-2.6-xen-instalIn our case we need to rebuild the Kernel because we must add our SATA device drivers.
Before we can use the Xen Kernel we have to replace our bootloader because XEN is not working with LILO so we need to install GRUB from our slackware dvd:
cd /mnt/[DVD]/extra/grub installpkg grub-0.97-i486-3.tgzWe install now on both disks a MBR with the following:
Grub>device (hd0) /dev/sda Grub>root (hd0,0) Grub>setup (hd0) Grub>device (hd0) /dev/sdb Grub>root (hd0,0) Grub>setup (hd0)Under "/boot/grub" we but the "menu.1st" file:
# Boot automatically after 5 secs. timeout 5 # # By default, boot the first entry. default 0 # # Fallback to the second entry. fallback 1 # # For booting with disc 0 kernel title Xen 3.1 on Slackware 12 root (hd0,0) kernel /boot/xen.gz noreboot module /boot/vmlinuz-2.6-xen root=/dev/md0 ro console=tty0 title Slackware 12 root (hd0,0) kernel /boot/vmlinuz ro root=/dev/md0Now we can reboot our System and boot the Xen Kernel based Slackware. During the boot sequence the following message are seen:
*************************************************************** ** WARNING: Currently emulating unsupported memory accesses ** ** in /lib/tls glibc libraries. The emulation is ** ** slow. To ensure full performance you should ** ** install a 'xen-friendly' (nosegneg) version of ** ** the library, or disable tls support by executing ** ** the following as root: ** ** mv /lib/tls /lib/tls.disabled ** ** Offending process: init (pid=1) ** ***************************************************************Slackware does not have a "/lib/tls". The problem is we need a Xen compatible glibc otherwise we loose huge performance . In detail the Native POSIX Thread Library, Definition: [NPTL wikipedia website] need to be reconfigured. We should rebuild our glibc with the following "-mno-tls-direct-seg-refs" option. For that we copy as the glibc sources from the dvd and add the CFLAGS option in the "glibc.SlackBuild" file:
# old # CFLAGS="-g $OPTIMIZ" \ # new CFLAGS="-g $OPTIMIZ -mno-tls-direct-seg-refs" \After that we execute the build file and install the created slackware package.
./glibc.SlackBuild installpkg glibc-2.5-i486-4.tgz rebootWith our new glibc we rebuild the sysvinit and the mdadm package. We copy the sources from the packages and call again the build script and install the new created packages.
UPDATE:Kernel headers
If the kernel headers version does not match your slackware glibc source package - install the kernel headers under a temp. destination like /tmp/kernel-headersinstallpkg -root /tmp/kernel-headers kernel-headers-2.6.27.7_smp-x86-1.tgzand add the following to the CFLAGS in the glibc.SlackBuild file
# --with-headers=/usr/src/linux-${KERNEL_HEADERS}/include \
--with-headers=/tmp/kernel-headers/usr/include \
UPDATE:Upgrade to slackware 12.2
Blacklist the coreutils if you use the 2.6.18 xen kernel. New version dont work together with a too old kernels.You can download xen slackware 12.0 glibc package here: [xenbuild.tar.gz]
download unpack and:
cd xenbuild/pkg installpkg glibc-2.5-i486-4.tgzTo configure xen daemon xend use "/etc/xen/xend-config.sxp". A good description you can find here [website](German). For the network configuration we decide to use bridging. Here we need to replace the "network-bridge" with the "network-bridge-bonding" you can find them in the Xen users Archives [link]. Copy them under "/etc/xen/scripts/" and replace in "/etc/xen/xend-config.sxp":
#old #(network-script network-bridge) #new (network-script network-bridge-bonding)Now we start the daemon with the command "xend". The next step is to prepare our Xen for automatic startup after reboot. First we add the xen domains starter script to our "/etc/rc.d/":
cp /[XEN-SOURCE-DIR]/dist/install/etc/init.d/xendomains /etc/rc.d/rc.xendomains chmod +x /etc/rc.d/rc.xendomainsNow we modify our "/etc/rc.d/rc.local":
# start xend
if [ `uname -r | grep -c 'xen'` -eq "1" ]; then
echo start xen
xend start
echo start autostarter for xendomains
/etc/rc.d/rc.xendomains start
else
echo xen not started
fi
UPDATE: Xen Network via Proxy ARP (routing)
Because i used device bonding i get problems with xen-bridged-mode (read more at the xen forum). So i decided to change to routing mode via Proxy ARP. But here i run into problems because whenever i moved one xen guest from one xen server to the other, this xen guest was inaccessible from outside (network behinde the router). The reason was that the router in my network make an ARP Cache update every 300min. And i have no access to this router!The solution is:
Shutdown the moved xen guest on his new xen server. But remember the IP address(es) which the xen guest use. And follow this procedure.
IP=X.X.X.X (the IP Address form the xen guest) IP_ROUTER=X.X.X.X ifconfig bond0:0 $IP netmask 255.255.255.0 arping -U -Ibond0:0 -s $IP $IP_ROUTER #output: #Unicast reply from X.X.X.X [00:.....] 1.820ms #. #. #. ifconfig bond0:0 downBoot the xen guest and the host will be accessible from outside!
Setup Xen guests
Under our lvm mount "/mnt/xen" we crete two directoriesmkdir /mnt/xen/images mkdir /mnt/xen/mount
Build a Debian guest
Build image files for our guest:dd if=/dev/zero of=/mnt/xen/images/guest-debian-4.img bs=1024k count=5000 dd if=/dev/zero of=/mnt/xen/images/guest-debian-4-swap.img bs=1024k count=500 mkreiserfs -f /mnt/xen/images/guest-debian-4.img mkswap /mnt/xen/images/guest-debian-4-swap.imgMount the image file in our filesystem:
mount -o loop /mnt/xen/images/guest-debian-4.img /mnt/xen/mountInstall debian with debootstrap. To build a debootstrap for Slackware we download the debian package debootstrap_0.3.3.2_all.deb from a debian mirror and prepare them for usage under slackware:
ar x debootstrap_0.3.3.2_all.deb mv data.tar.gz debootstrap_0.3.3.2_all.tgz installpkg debootstrap-0.2.45-i486-1.tgzNow use debootstrap with the following to install debian:
debootstrap --verbose --arch i386 etch /mnt/xen/mount/debian-4/ http://ftp.de.debian.org/debian
Duplicate to the other servers
For that is used two Knoppix DVDs. I booted both servers with the knoppix dvds. Put both servers in a private class C network.ifconfig eth0 192.168.0.1 netmask 255.255.255.0 # installed one ifconfig eth0 192.168.0.2 netmask 255.255.255.0 # new oneWe use dd to mirror the data. On the empty server machine we start a sshd deamon. On the already installed server we use the following command:
dd if=/dev/sda | root@192.168.0.2 "dd of=/dev/sda"After a successfull data copy we can boot our new server! To put the second sata disk into the RAID we do the following:
sfdisk -d /dev/sda |sfdisk /dev/sdb mdadm /dev/md0 -a /dev/sdb1 mdadm /dev/md1 -a /dev/sdb2 mdadm /dev/md2 -a /dev/sdb3 mdadm /dev/md3 -a /dev/sdb5 mdadm /dev/md4 -a /dev/sdb6