pervasive logo
Linux Server Setup Crib Sheet

Linux Server Setup Crib Sheet

Table of Contents

Next: , Up: (dir)   [Contents]

Setup Crib Sheet

This short manual is a reminder on how-to setup a Linux server. That you know next time what to do! This manual uses slackware as a reference linux distribution. Slackware follows the KISS principles. In the Slackware philosophy the user isn’t considered stupid. But basically any other linux distribution can be used, differences might be the package manager or the os init daemon. This document includes some useful nodes on Xen, DRBD and other life saving features from open source world.

(version 0.1, D. Hochreiter).


Next: , Previous: , Up: Top   [Contents]

1 Chapter - Introduction

Based on a living example. This manual describes a setup process of a two server-configuration. So this manual also includes fault tolerance and redundancy considerations. Though the hints and tips can be used for a single-setup too.

1.1 Overview

A picture of the architecture


                        ................................................            NAS
                        .                                  Linux + XEN .         __________ 
                        .                                              .        [_|||||||_°]
                        .                                        <-----------   [_|||||||_°]
                        .                   DRBD                     10GbE      [_|||||||_°]
                        .                ^        ^                    .
                        .               /  (10GbE) \                   .
                        .              /            \                  .
                        .          LVM               LVM               .
                        .          ^                   ^               .
                        .         /                     \              .
                        .        /                       \             .
                        .      RAID1                    RAID1          .
                        .      |====|                  |====|          .
                        .      |    |                  |    |          .
                        .      |    |                  |    |          .
                        .      |____|                  |____|          .
                        .  |                |                          .
                        .  |  Server A      |         Server B         .
                        ...|................|...........................
                           |                |
                           v                v
                          .------------.   .----------------.
                          | App Server |   | Windows Server |
                          '------------'   '----------------'

Next: , Previous: , Up: Top   [Contents]

2 Chapter - Installation

Enter installation media and start server box. Some points to consider during installation:

2.1 Disks, Partitions, RAID and LVM

Note: prepare hd drives, setup partitions, e.g. use "FD Linux raid autodetect" partition type in case you plan to use RAID

#cfdisk /dev/sd[a|b]

Hint: A RAID installation with similar disks can use the sfdisk to clone the partition table.

sfdisk -d /dev/sdX | sfdisk /dev/sdY

Hint: If you plan to use grub2 as boot loader on MBR including LVM and/or RAID support keep enough free space on the first sectors (e.g.: 0 - 2048 or more)

Note: RAID initial setup

mdadm --create /dev/mdX --metadata=0.90 --level=1 --raid-devices=2 /dev/sdX /dev/sdY

Hint: Increase RAID sync speed

echo 50000 >/proc/sys/dev/raid/speed_limit_min
echo 400000 >/proc/sys/dev/raid/speed_limit_max

Tip: mdadm mail alerts, edit: /etc/mdadm.conf

MAILADDR admin@domain..
MAILFROM hostnameA - mdadm

Tip: test configuration

mdadm --monitor --scan --test --oneshot /dev/md[[:digit:]]*

Tip: remove and/or add hdd disk

mdadm --manage /dev/mdX --fail /dev/sdY
mdadm --manage /dev/mdX --remove /dev/sdY
mdadm --manage /dev/mdX --add /dev/sdY

Edit: /etc/rc.d/rc.local add:

mdadm --monitor --scan --daemonize --test --syslog /dev/md[[:digit:]]*

Note: LVM initial setup: E.g.: create a LVM volume group called vg0, create a logical volume called vol0 on volume group vg0. Create a file system on it.

pvcreate /dev/mdX
vgcreate vg0 /dev/mdX
lvcreate -L 30G -n vol0 vg0
mkfs.ext4 /dev/vg0/vol0

Hint: LVM commands fyi: vgdisplay, pvscan, vgextend

Hint: LVM install howtos can be found here: http://gd.tuwien.ac.at/opsys/linux/freesoftware.com/slackware64-14.1/README_LVM.TXT

2.2 Boot loader

Both common boot loaders can be used

2.2.1 LILO

Note: Some OS like Slackware needs a initrd (initial ramdisk) to boot from LVM

chroot /mnt
/usr/share/mkinitrd/mkinitrd_command_generator.sh

Edit: lilo.conf add to default boot entry

initrd = /boot/initrd.gz
root = /dev/vg0/vol0

run lilo again and reboot

2.2.2 GRUB

Slackware 14.1 suffers on the grub bug #41582 "Double free in grub-probe when using LVM" (http://savannah.gnu.org/bugs/?41582)

It got fixed: here

crucial if you wanna operate a slack with a boot partition using (mdadm) raid1 + lvm + grub as boot loader.

Checkout of the source (git clone git://git.savannah.gnu.org/grub.git, Thu Oct 15 17:50:23 CEST 2015)

stored the tarball: grub-2.02_beta2.tar.xz and the build script: grub.tar.bz2

Hint: Enable boot from RAID in grub config

Edit: /etc/default/grub add

GRUB_CMDLINE_LINUX_DEFAULT="domdadm"

Hint: Grub install howtos can be found here:

http://docs.slackware.com/howtos:slackware_admin:set_up_grub_as_boot_loader_on_uefi_based_hardware

http://docs.slackware.com/howtos:slackware_admin:grub_on_first_install

grub-install /dev/sdX
grub-install /dev/sdY
grub-mkconfig -o /boot/grub/grub.cfg

2.3 Install OS / Packages

Follow the ordinary slack installation... Slackware-HOWTO

Hint: In case of reboot during installation process (with no boot loader installed) change to already installed environment by:

mdadm --assemble /dev/mdX /dev/sdX1 /dev/sdY1   #(or --auto-detect)
vgchange -a y vg0
mount /dev/vg0/vol0 /mnt
mount -t proc none /mnt/proc
mount -o bind /dev /mnt/dev
mount -t sysfs none /mnt/sys
chroot /mnt

Hint: Install package manager for SlackBuilds.org: sbopkg and sbotools

2.4 Upgrade OS / Packages

First have a look here: http://docs.slackware.com/howtos:slackware_admin:systemupgrade, ftp://ftp.slackware.com/pub/slackware/slackware-current/UPGRADE.TXT, http://docs.slackware.com/slackware:slackpkg

2.4.1 Upgrade a running server on LVM

Hint: First clone the volume during run-time:

lvcreate -L3000M -s -n vol0-snap /dev/vg0/vol0
lvcreate -L 30G -n vol1 vg0
dd if=/dev/vg0/vol0-snap of=/dev/vg0/vol1 bs=100M
lvremove /dev/vg0/vol0-snap
fsck.ext4 /dev/vg0/vol1

#generate a NEW! UUID 
uuidgen
1b191059-f930-48bd-bb2e-446664138344
tune2fs /dev/mapper/vg0-vol1 -U 1b191059-f930-48bd-bb2e-446664138344
#Dont forget to use that one inside of grub

blkid /dev/vg0/vol1 #check UUID

Hint: Second change to the new volume und start upgrade process:

mkdir /mnt/upgrade 
mount /dev/vg0/vol1 /mnt/upgrade

mount -t proc none /mnt/upgrade/proc 
mount -o bind /dev /mnt/upgrade/dev
mount -t sysfs none /mnt/upgrade/sys

chroot /mnt/upgrade

#ALERT!! update volume path in /etc/fstab

#start upgrade procedure, e.g.: http://docs.slackware.com/howtos:slackware_admin:systemupgrade

#change to version
vim /etc/slackpkg/mirrors 
slackpkg update gpg
slackpkg update 
slackpkg upgrade slackpkg

slackpkg upgrade glibc-solibs
slackpkg install-new
slackpkg upgrade-all
slackpkg clean-system 


exit # leave chroot, eventually copy rc.*, network and ssh config's
cp /etc/rc.d/rc.local /mnt/upgrade/etc/rc.d/rc.local
cp /etc/ssh/sshd_config /mnt/upgrade/etc/ssh/

Hint: Third update grub config, reboot to new env. and apply new long term kernel

grub-mkconfig -o /boot/grub/grub.cfg
# you should got now a new entry with your upgraded version!
# reboot and select upgraded version
# if grub not working(!), can occur if some old vmlinuz images are in the boot folder
# add a custom config to /etc/grub.d/40_custom: ALERT use correct UUID generated with uuidgen

menuentry 'Slackware-14.2 GNU/Linux, with Linux 4.1.10' --class slackware_14_2 --class gnu-linux 'Upgrade Slackware 14.2' {
                load_video
                insmod gzio
                insmod part_msdos
                insmod part_msdos
                insmod diskfilter
                insmod mdraid09
                insmod lvm
                insmod ext2
                set root='lvmid/l1g7c6-cxBU-9zJ9-xEXX-lgdV-32yk-OE1BHh/fdbcc9-oyal-sPI5-K748-D9Zn-H4uV-UJFIPH'
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root --hint='lvmid/l1g7c6-cxBU-9zJ9-xEXX-lgdV-32yk-OE1BHh/fdbcc9-oyal-sPI5-K748-D9Zn-H4uV-UJFIPH'  1b191059-f930-48bd-bb2e-446664138344
                else
                  search --no-floppy --fs-uuid --set=root 1b191059-f930-48bd-bb2e-446664138344
                fi
                echo    'Loading Linux 4.1.10 ...'
                linux   /boot/vmlinuz-dominik-4.1.10 root=/dev/mapper/vg0-vol1 ro  domdadm bootdegraded=true
                echo    'Loading initial ramdisk ...'
                initrd  /boot/initrd-4.1.10.gz
}

# get latest long term kernel
# follow instruction in section "Kernel"
# careful with initrd:
mkinitrd -c -k 4.4.14 -f ext4 -r /dev/vg0/vol1 -m usb-storage:xhci-hcd:xhci-pci:ohci-pci:ehci-pci:uhci-hcd:ehci-hcd:hid:usbhid:i2c-hid:hid_generic:hid-cherry:hid-logitech:hid-logitech-dj:hid-logitech-hidpp:hid-lenovo:hid-microsoft:hid_multitouch:ext4 -L -R -u -o /boot/initrd-4.4.14.gz

# add new kernel to grub!

menuentry 'Slackware-14.2 GNU/Linux, with Linux 4.4.14' --class slackware_14_2 --class gnu-linux 'Upgrade Slackware 14.2' {
                load_video
                insmod gzio
                insmod part_msdos
                insmod part_msdos
                insmod diskfilter
                insmod mdraid09
                insmod lvm
                insmod ext2
                set root='lvmid/l1g7c6-cxBU-9zJ9-xEXX-lgdV-32yk-OE1BHh/fdbcc9-oyal-sPI5-K748-D9Zn-H4uV-UJFIPH'
                #if [ x$feature_platform_search_hint = xy ]; then
                #  search --no-floppy --fs-uuid --set=root --hint='lvmid/l1g7c6-cxBU-9zJ9-xEXX-lgdV-32yk-OE1BHh/fdbcc9-oyal-sPI5-K748-D9Zn-H4uV-UJFIPH'  1b191059-f930-48bd-bb2e-446664138344
                #else
                  search --no-floppy --fs-uuid --set=root 1b191059-f930-48bd-bb2e-446664138344
                #fi
                echo    'Loading Linux 4.4.14 ...'
                linux   /boot/vmlinuz-4.4.14 root=/dev/mapper/vg0-vol1 ro  domdadm bootdegraded=true
                echo    'Loading initial ramdisk ...'
                initrd  /boot/initrd-4.4.14.gz
}


#boot to new env (/dev/vg0/vol1)
cp /mnt/slackware-14.1/etc/grub.d/40_custom /etc/grub.d

#re-install grub that it uses "grub.cfg" from "/dev/vg0/vol1"
grub-install /dev/sdX
grub-install /dev/sdY
grub-mkconfig -o /boot/grub/grub.cfg

#reboot

2.5 Kernel

Get the latest longterm kernel from kernel.org

Hint: A more detailed description can be found here:kernelbuilding

Hint: add lvm support to kernel config


cd /usr/src/

tar xvf linux-*.tar.xz 
cd linux-*

zcat /proc/config.gz > /usr/src/linux/.config
make oldconfig
make menuconfig

make -j48 bzImage modules            # compile the kernel and the modules -j<cores> 
make -j48 modules_install

cp arch/x86_64/boot/bzImage /boot/vmlinuz-custom-4.1.10
cp System.map /boot/System.map-custom-4.1.10
cp .config /boot/config-custom-4.1.10
cd /boot/
rm System.map
ln -s System.map-custom-4.1.10 System.map

#update init ramdisk
/usr/share/mkinitrd/mkinitrd_command_generator.sh -k 4.1.10 (add -L for LVM, add -R for RAID and -u for udev support to the initrd!!)

#if bootloader grub 
/etc/default/grub: set e.g. #GRUB_DEFAULT="1>4"
grub-mkconfig -o /boot/grub/grub.cfg


#add missing firmware, e.g.: Intel Skylake Graphics, download from https://01.org/linuxgraphics/intel-linux-graphics-firmwares 
#and install to /lib/firmware

Hint: Kernel crash dump, some useful links: https://www.kernel.org/doc/Documentation/kdump/kdump.txt https://slackbuilds.org/repository/14.1/system/kexec-tools/?search=kexec https://help.ubuntu.com/lts/serverguide/kernel-crash-dump.html


Next: , Previous: , Up: Top   [Contents]

3 Chapter - XEN

3.1 Install dom0

  1. First install dependencies
    sboinstall yajl
    sboinstall acpica
    #sboinstall mbootpack (mbootpack only required if "lilo" is used )
    
  2. get it from: http://slackbuilds.org/repository/14.1/system/xen/

    Follow the install advices here:

    http://slackbuilds.org/slackbuilds/14.1/system/xen/README

    http://slackbuilds.org/slackbuilds/14.1/system/xen/dom0/README.dom0

3.2 Init scripts

Edit: /etc/rc.d/rc.local add:

if [ -d /proc/xen ]; then
  if [ -x /etc/rc.d/rc.xencommons ]; then
    echo "Starting XEN commons:  /etc/rc.d/rc.xencommons"
    /etc/rc.d/rc.xencommons start
  fi
  if [ -x /etc/rc.d/rc.xendomains ]; then
    echo "Starting XEN domains:  /etc/rc.d/rc.xendomains"
    /etc/rc.d/rc.xendomains start
  fi
fi

Hint: IMPORTANT: Follow instructions at: http://wiki.xenproject.org/wiki/Xen_Project_Best_Practices, "Xen Project dom0 dedicated memory and preventing dom0 memory ballooning!" http://wiki.xenproject.org/wiki/Network_Configuration_Examples_(Xen_4.1%2B), "Disable Netfilter on Bridges (All Distributions)!"

Edit: /etc/xen/xl.conf set: autoballoon="off"

Edit: /etc/rc.d/rc.local set (if not set, can be a reason for entire system crashes): xl sched-credit -d 0 -w 512 echo "0" > /proc/sys/net/bridge/bridge-nf-call-ip6tables echo "0" > /proc/sys/net/bridge/bridge-nf-call-iptables echo "0" > /proc/sys/net/bridge/bridge-nf-call-arptables

Note: If no multilib installation

ln -s /usr/include/gnu/stubs-64.h /usr/include/gnu/stubs-32.h

Setup Xen Kernel

cd /data/software/slackware/slackbuilds/xen/dom0_custom

Edit: kernel-xen.sh adapt:

  1. change to new kernel version (cp config-3.10.17-xen.x86_64 config-4.1.10-xen.x86_64)
  2. adapt the mkinitrd settings
  3. add -L, -R and -u (LVM, RAID & udev support to the initrd!!): mkinitrd -c -k $KERNEL-xen -m $ROOTMOD -f $ROOTFS -r $ROOTDEV -L -R -u -o /boot/initrd-$KERNEL-xen.gz
export ROOTDEV=/dev/vg0/vol0

# The needed modules for $ROOTMOD you can find by /usr/share/mkinitrd/mkinitrd_command_generator.sh ( the values after "-m" )
export ROOTMOD=usb-storage:ehci-hcd:ehci-pci:usbhid:hid_generic:mbcache:jbd2:ext4

#in case of grub
export BOOTLOADER=grub

./kernel-xen.sh

#create initrd afterwards
mkinitrd -c -k 4.4.1-xen -m usbhid:hid_generic:ext4 -f ext4 -r /dev/vg0/vol0 -L -R -u -o /boot/initrd-4.4.1-xen.gz

#in case update grub
/etc/default/grub: set e.g. #GRUB_DEFAULT="2"
grub-mkconfig -o /boot/grub/grub.cfg

./xen.SlackBuild
installpkg xen-4.3.1-x86_64-1_SBo.tgz

Hint: Shutting down xen domUs without saving them: /etc/default/xendomains set: XENDOMAINS_SAVE= XENDOMAINS_RESTORE=false

3.3 boot loader

3.3.1 grub

Xen boot options more

Hint: /etc/default/grub add:

GRUB_DEFAULT="2"
GRUB_CMDLINE_XEN_DEFAULT="dom0_mem=4G,max:8G dom0_max_vcpus=4-8"

#in case of an old AMD opteron cpu, add:
GRUB_CMDLINE_XEN="allow_unsafe=true"

and run

grub-mkconfig -o /boot/grub/grub.cfg

3.3.2 lilo

Update lilo!

Edit: /etc/lilo.conf:

image = /boot/vmlinuz-xen
  root = /dev/vg0/vol0
  label = XenLinux
  append="dom0_mem=512M -- nomodeset"
  read-only
# If you need X remove "nomodeset" from lilo.conf

3.4 XEN domU

Create a domU test image 25G

dd if=/dev/zero of=test.img bs=1024k count=25600

Graphical Console Support: http://xenbits.xen.org/docs/4.2-testing/man/xl.cfg.5.html, Configuring a VNC based Graphical Console for a Xen Paravirtualized domainU Guest. HVM Guests http://xenbits.xen.org/docs/4.2-testing/man/xl.cfg.5.html#emulated_vga_graphics_device

slackpkg install xf86-video-fbdev-0.4.4-x86_64-1 #if missing on domU

Tip: Generate a random MAC Address

perl -e 'for ($i=0;$i<4;$i++){@m[$i]=int(rand(256));} printf "00:16:%X:%X:%X:%X\n",@m;'

Tip: A Debian Guest howto: http://wiki.xenproject.org/wiki/Debian_Guest_Installation_Using_Debian_Installer

3.4.1 domU win with vga passthrough

grub default config!!

http://wiki.xenproject.org/wiki/Xen_4.2:_xl_and_pci_pass-through xl pci-assignable-list

http://wiki.xen.org/wiki/Xen_PCI_Passthrough http://wiki.xenproject.org/wiki/Xen_VGA_Passthrough#The_effect_of_gfx_passthru.3D_option http://wiki.xenproject.org/wiki/XenVGAPassthroughTestedAdapters#Nvidia_display_adapters

Keyboard inside Passthrough-Environment: http://wiki.xenproject.org/wiki/XenUSBPassthrough http://www.virtuatopia.com/index.php/Adding_USB_Devices_to_a_Xen_HVM_domainU_Guest

note nvidia: http://www.overclock.net/t/1205216/guide-create-a-gaming-virtual-machine/760#post_22351657 DevCap’ (FLReset- is present)

FLReset- means that it doesn’t support it, FLReset+ that it does. No Radeons support FLR that I’m aware of, and on Xen Wiki it says very few devices actually support it. nVidia Quadro supports it for sure, GeForces no idea. I think it was one of the reasons why some users were modding GeForces into Quadros or GRIDs.

Tip HVM Guests: Windows PV Drivers http://www.xenproject.org/developers/teams/windows-pv-drivers.html


Previous: , Up: Top   [Contents]

4 Configuration

4.1 Network

4.1.1 Setup Network bridges for XEN!

first adapt the device mapping inside: /etc/udev/rules.d/70-persistent-net.rules Which name (eth?) has each physical device, separate e.g.: 1Gb from 10Gb devices

Edit: /etc/rc.d/rc.inet1.conf add all the net dev’s you wanna use inside xen guests:

IFNAME[0]="xenbr0"
BRNICS[0]="eth0"
IPADDR[0]="xx.xx.xx.xx"
NETMASK[0]="255.255.255.0"
USE_DHCP[0]=""
DHCP_HOSTNAME[0]=""

IFNAME[1]="xenbr1"
BRNICS[1]="eth1"
IPADDR[1]=""
NETMASK[1]=""
USE_DHCP[1]=""
DHCP_HOSTNAME[1]=""

IFNAME[2]="xenbr2"
BRNICS[2]="eth2"
IPADDR[2]=""
NETMASK[2]=""
USE_DHCP[2]=""
DHCP_HOSTNAME[2]=""

IFNAME[3]="xenbr3"
BRNICS[3]="eth3"
IPADDR[3]=""
NETMASK[3]=""
USE_DHCP[3]=""
DHCP_HOSTNAME[3]=""

to be sure all the bridges that are not configured inside rc.inet1.conf are up and ready

Edit: /etc/rc.d/rc.local:

..
if [ -d /proc/xen ]; then

ifconfig xenbr1 up
ifconfig xenbr2 up
ifconfig xenbr3 up

..

4.1.2 Firmware

In case of missing firmware e.g. for the 10Gb Devices install them (e.g. Broadcom) git clone https://kernel.googlesource.com/pub/scm/linux/kernel/git/firmware/linux-firmwaregit install missing file e.g.: /data/software/kernel/linux-firmware# cp bnx2x/bnx2x-e2-7.10.51.0.fw /lib/firmware/bnx2x/

4.1.3 Setup private 10Gb Network, NAS & DRBD

Edit: /etc/rc.d/rc.inet1.conf add:

#DRBD net
IFNAME[4]="eth4"
IPADDR[4]="xx.xx.xx.[1|2]"
NETMASK[4]="255.255.255.0"
USE_DHCP[4]=""
DHCP_HOSTNAME[4]=""

#NAS net
IFNAME[5]="eth5"
IPADDR[5]=""xx.xx.xx.[1|2]""
NETMASK[5]="255.255.255.0"
USE_DHCP[5]=""
DHCP_HOSTNAME[5]=""

Tip: increase jumbo frames size (MTU) ifconfig eth5 mtu 9000 add MTU[5]=9000 to /etc/rc.d/rc.inet1.conf more: http://dak1n1.com/blog/7-performance-tuning-intel-10gbe/, https://www.thomas-krenn.com/de/wiki/10GBit_Performance_Tuning

4.1.4 Bonding

Configure two 10GbE devices to one bond device, have a fail-over! more

Add a init file: /etc/rc.d/rc.bond:

#!/bin/sh

case "$1" in
  'start')
            echo "start bond0"
            modprobe bonding mode=balance-alb miimon=100
            ifconfig bond0 xx.xx.xx.[1|2] netmask 255.255.255.0 up
            ip link set eth4 master bond0
            ip link set eth5 master bond0
          ;;
  'stop')
            ifconfig bond0 down
            rmmod bonding
          ;;
          *)
            echo "Usage: $0 start|stop"
          ;;
esac

Edit: /etc/rc.d/rc.M add:

(add before  #Initialize the networking hardware)
# If script rc.bond is executeable then start it
if [ -x /etc/rc.d/rc.bond ]; then
  . /etc/rc.d/rc.bond start
fi

4.1.5 Hostnames

Info: Edit following files to meet your hosts configuration: /etc/HOSTNAME, /etc/hosts, /etc/resolv.conf

Info: DRBD:

  1. hostnameA: xx.xx.xx.1
  2. hostnameB: xx.xx.xx.2

Info: NAS:

  1. hostnameA: xx.xx.xx.1
  2. hostnameB: xx.xx.xx.2

4.1.6 Firewall

Since we are now on-line protect dom0, only connections from local network + ssh from external

Create: /etc/rc.d/rc.firewall and edit:

#Flushing all rules
iptables -F
iptables -X

# Set the default policy of the INPUT chain to DROP
#iptables -P INPUT DROP
#iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -i xenbr0 -sXX.XX.XX.0/XX -j ACCEPT

#all IPv4 addresses on the local machine
iptables -A INPUT -i xenbr0 -s0.0.0.0 -j ACCEPT

# Accept incomming connections from xenbr0 on port XX
iptables -A INPUT -i xenbr0 -p tcp --dport XX -j ACCEPT
# allow ntpdate
iptables -A INPUT -i xenbr0 -p udp --dport 123 -j ACCEPT

#Allowing Established Sessions
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

#logging to syslog
iptables -N LOGGING
iptables -A INPUT -i xenbr0 -j LOGGING
iptables -A LOGGING -i xenbr0 -m limit --limit 2/min -j LOG --log-prefix "IPTables-Dropped: " --log-level 5
iptables -A LOGGING -i xenbr0 -j DROP

#drop rest
iptables -A INPUT -i xenbr0 -j DROP
chmod +x /etc/rc.d/rc.firewall

Edit: /etc/rc.d/rc.local, add:

/etc/rc.d/rc.firewall

Hint: log firewall messages to own log file /var/log/iptables.log, append to /etc/syslog.conf

kern.notice /var/log/iptables.log

Hint: create /etc/logrotate.d/iptables:

/var/log/iptables.log {
    rotate 
    size 500k
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2>/dev/null` 2>/dev/null || true
    endscript
}

Hint: Real-time Iptables Monitor - perlmonks.org: http://www.perlmonks.org/?node_id=513732 (observe xen bridges)

4.1.7 IPv6

Hint: Disable

echo "blacklist ipv6" > /etc/modprobe.d/blacklist-ipv6.conf

of in grub config add "ipv6.disable=1"

4.1.8 traffic limiting via tc

http://tdistler.com/2011/06/10/netem-wan-emulation-how-to-setup-a-netem-box http://serverfault.com/questions/125880/bridge-traffic-limiting-via-tc

4.2 DRBD

Install software with sbopkg. The following configuration shows a master-/slave example based on LVM volumes involving two machines.

Note: Prepare a lvm volume:

lvcreate -L 30G -n vol-drbd0 vg0

Note: Add /etc/drbd.d/r0.res on both hosts (hostnameA, hostnameB) :

resource r0 {
        protocol C;

        startup {
                wfc-timeout 120; ## 2 min
                degr-wfc-timeout 120; ## 2 minutes.
        }

        disk {
                resync-rate 236M;
                on-io-error detach;
        }

        net {
                # xen live migration needs it
                allow-two-primaries;

                after-sb-0pri discard-zero-changes;
                after-sb-1pri discard-secondary;

                timeout 60;
                connect-int 10;
                ping-int 10;
                max-buffers 4096;
                max-epoch-size 4096;
        }

        syncer {
        }

        on hostnameA {
                address xx.xx.xx.1:7789;
                device /dev/drbd0;
                disk /dev/vg0/vol-drbd0;
                meta-disk internal;
                #meta-disk /dev/vg0/vol-drbd-meta[0];

        }

        on hostnameB {
                address xx.xx.xx.2:7789;
                device /dev/drbd0;
                disk /dev/vg0/vol-drbd0;
                meta-disk internal;
                #meta-disk /dev/vg0/vol-drbd-meta[0];
        }
}

Edit: /etc/rc.d/rc.local, add (befor xen is started):

/etc/rc.d/rc.drbd start 

Hint: Start a drbd resource manually

drbdadm create-md r0
drbdadm up r0

#manual mount 
#on hostnameA
drbdadm primary --force r0

#on hostnameB
drbdadm secondary r0

4.2.1 Testing / Configuration

Tip: drbd resync-rate calculation

(1) detect real network speed:
	TCP:
	on hostnameA: nc -l -p 1234 -s xx.xx.xx.1 | dd of=/dev/null
        on hostnameB: dd if=/dev/zero bs=4096 count=1048576 | nc -s xx.xx.xx.2 xx.xx.xx.1 1234
	UDP:
	on hostnameA: nc -l -u -p 1234 -s xx.xx.xx.1 | dd of=/dev/null
	on hostnameB: dd if=/dev/zero bs=4096 count=1048576 | nc -u -s xx.xx.xx.2 xx.xx.xx.1 1234

    result TCP: ~336 MB/s

(2) according to drbd (https://drbd.linbit.com/users-guide/s-configure-sync-rate.html)
    "good rule of thumb for this value is to use about 30% of the available replication bandwidth"
    (336x0.3 = 101) 
    in our case we use 70%, there only two devices inside of our network:
    336x0.7 = 236
	
   resync-rate 236M; 

Tip: Test drbd device speed:

mkfs.ext4 /dev/drbd0
mount /dev/drbd0 /mnt/hd
dd if=/dev/zero bs=4096 count=10485760 of=/mnt/hd/test.raw

4.2.2 Together with XEN

Create a xen domU image or get it from somewhere in the web and copy it to the drbd resource device. A domU config help: http://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html, http://xenbits.xen.org/docs/unstable/, http://xenbits.xen.org/docs/4.5-testing/misc/xl-disk-configuration.txt

dd if=template-slackware-14.1_x86_64_kernel-4.1.2_ext3_25G.img bs=100M of=/dev/drbd0

or via ssh (exec from remote machine)

dd if=/dev/vg01/xyz bs=50M | ssh -p70 root@XENHOST "dd of=/dev/drbdX bs=50M"

Tip: check fs:

drbdadm primary r0
fsck.ext3 /dev/drbd0
#drbdadm secondary r0

Tip: configure and run xen domU guest

vim /etc/xen/template-slackware-14.1_x86_64_kernel-4.1.2_ext3_25G.conf
xl create -c /etc/xen/template-slackware-14.1_x86_64_kernel-4.1.2_ext3_25G.conf

Tip: Try also a live migration from hostnameA to hostnameB, enter on hostnameA:

xl migrate 1 hostnameB

Tip: Backup at run time:

1) Mount a running Xen domU during run time: On "drbd primary" : mount -r /dev/drbd0 /mnt/hd/mount 

or

2) Creating a LVM snapshot druing run time (works on drbd primary and secondary):
lvcreate -L500M -s -n vol-drbd0-snap /dev/vg0/vol-drbd0
mount -t ext3 /dev/vg0/vol-drbd0-snap /mnt/hd (trick is to override the file system type flag)
umount /mnt/hd
lvremove /dev/vg0/vol-drbd0-snap

Tip: Get lvm vol fs "type" like old DOS partitions do:

file --dereference --special-files /dev/mapper/vg0-vol--drbd0--snap

Tip: Recycling from DRBD partition and remove fs type "drbd" (mount: unknown filesystem type ’drbd’)

lvcreate -L500M -s -n vol-drbd0-snap /dev/vg0/vol-drbd0
dd if=/dev/vg0/vol-drbd0-snap of=./image.img bs=200M
lvremove /dev/vg0/vol-drbd0-snap

root@:~# wipefs -n -a ./image.img
4 bytes were erased at offset 0x67ffff03c (drbd): 83 74 02 6c
2 bytes were erased at offset 0x00000438 (ext3): 53 ef

root@:~# wipefs -o 0x67ffff03c ./image.img
4 bytes were erased at offset 0x67ffff03c (drbd): 83 74 02 6c

root@:~# mount -o loop ./image.img /mnt/hd

Tip: Alternative shared storage configuration with NBD: http://nbd.sourceforge.net/, http://wiki.xenproject.org/wiki/Migration, http://slackbuilds.org/repository/14.1/network/nbd/?search=nbd

4.3 Monitoring

4.3.1 HP Tools (in case)

Tip: Get a debian pkg converter, install pkg "alien" from http://slackbuilds.org/repository/14.1/system/alien/

Get HP System Health Application and Command line Utilities from http://downloads.linux.hp.com/SDR/project/mcp/

wget http://downloads.linux.hp.com/SDR/repo/mcp/debian/pool/non-free/hp-health_10.0.0.1.3-4._amd64.deb
alien --to-tgz hp-health_10.0.0.1.3-4._amd64.deb
installpkg hp-health-10.0.0.1.3.tgz

Edit: /etc/rc.d/rc.local add:

/etc/init.d/hp-health start

Tip: hplog -t

4.3.2 Setup monitorix

Install via sbopokg http://slackbuilds.org/repository/14.1/system/monitorix/?search=monitorix, depends on rrdtool http://slackbuilds.org/repository/14.1/libraries/rrdtool/

Edit: /etc/monitorix/monitorix.conf:

hostname = hostnameA 

<httpd_builtin>
        enabled = y
        host =
        port = 10001
        user = nobody
        group = nobody
        log_file = /var/log/monitorix-httpd
        hosts_deny =
        hosts_allow =
        <auth>
                enabled = y
                msg = Monitorix: Restricted access
                htpasswd = /var/lib/monitorix/htpasswd
        </auth>
</httpd_builtin>

#enable monitor setting: man monitorix.conf has some nice hints
#since we are on a hp ProLiant we enable:
hptemp          = y


#monitor disks:
disk            = y

<disk>
        <list>
                0 = /dev/sda, /dev/sdb
        </list>
        <alerts>
                realloc_enabled = n
                realloc_timeintvl = 0
                realloc_threshold = 1
                realloc_script = /path/to/script.sh
                pendsect_enabled = n
                pendsect_timeintvl = 0
                pendsect_threshold = 1
                pendsect_script = /path/to/script.sh
        </alerts>
</disk>


<net>
        list = eth0, eth1, eth2, eth3, eth4, eth5
        <desc>
                eth0 = FastEthernet LAN, 0, 10000000
                eth1 = FastEthernet LAN, 0, 10000000
                eth2 = FastEthernet LAN, 0, 10000000
                eth3 = FastEthernet LAN, 0, 10000000
                eth4 = FastEthernet LAN, 0, 10000000
                eth5 = FastEthernet LAN, 0, 10000000
        </desc>
        gateway = eth0
</net>

install missing perl modules:

cpan HTTP::Server::Simple::CGI
cpan Config::General

add user to: /var/lib/monitorix/htpasswd

htpasswd -d -c /var/lib/monitorix/htpasswd root
chmod +x /etc/rc.d/rc.monitorix

Edit: /etc/rc.d/rc.local add:

/etc/rc.d/rc.monitorix start

4.4 Miscellaneous

Get ride of password prompt, exchange ssh keys between hostnameA & hostnameB

Host A: ssh-keygen -t rsa -b 4096
Host A: scp id_rsa.pub root@hostnameB:/root/.ssh/
Host B: cat id_rsa.pub > authorized_keys 
Host B: rm id_rsa.pub

4.4.1 File System

Increasing the Size of Ext3 or Ext4

resize2fs /dev/xyz

Upgrade ext3 to ext4

tune2fs -O extents,uninit_bg,dir_index /dev/drbd0
fsck -fCVD /dev/drbd0

4.4.2 Limiting time and memory

Useful Info: http://coldattic.info/shvedsky/pro/blogs/a-foo-walks-into-a-bar/posts/40 A script to measure and limit CPU time and memory consumption of black-box processes in Linux: https://github.com/pshved/timeout