Using nfsroot to boot diskless clients on RHEL

Here is a brief outline on the steps needed to set up a Red Hat Enterprise Linux 6 server to boot diskless clients using nfs.

To do this, you need to set up dhcp, tftp (to pxe boot) and a nfs server to serve the rootfs.

To keep it simple, I did everything on the same server, but you can easily have multiple servers for each service. The same steps should also work on Fedora (14+) with minor changes.

Requirements :

Server : Fresh RHEL 6 installation (registered to RHN)

IP = 192.168.50.254/24 (static) on eth0

# vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=52:54:00:73:1C:CB
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.50.254
NETMASK=255.255.255.0

Client : A box on the same network (with support for PXE)

MAC = 52:54:00:ca:c6:71

On the Server :

Step 1 – Create the nfs rootfs first :

# mkdir /netboot/
# rsync -av --progress --exclude=/proc --exclude=/sys --exclude=/netboot / /netboot/

(i.e. copy everything from root to the /netboot folder)

# mkdir /netboot/{proc,sys}
# vi /netboot/etc/fstab

192.168.50.254:/netboot/  /                      nfs     defaults       1 1
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0

# vi /netboot/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE="eth0"
HWADDR="52:54:00:ca:c6:71"
ONBOOT="no"
BOOTPROTO="dhcp"

Step 2 – Install and enable services :

# yum install dhcp syslinux tftp-server -y
# yum groupinstall "NFS file server" -y

# chkconfig dhcpd on
# chkconfig tftp on
# chkconfig nfs on
# service xinetd start

For simplicity, disable the firewall :

# iptables -F
# service iptables save

Step 3 – configure dhcp server :

# vi /etc/dhcp/dhcpd.conf

# DHCP Server Configuration file.
#   see /usr/share/doc/dhcp*/dhcpd.conf.sample
#   see 'man 5 dhcpd.conf'
#
ddns-update-style interim;
ignore client-updates;
allow booting;
allow bootp;
subnet 192.168.50.0 netmask 255.255.255.0 {
option routers      192.168.50.254;
option subnet-mask   255.255.255.0;
option nis-domain   "domain.org";
option domain-name  "domain.org";
option domain-name-servers  192.168.50.254;
option time-offset    -18000; # Eastern Standard Time
default-lease-time 21600;
max-lease-time 43200;

host netboot6 {
next-server 192.168.50.254;
hardware ethernet 52:54:00:ca:c6:71;
fixed-address 192.168.50.100;
filename "pxelinux.0";
}
}

# service dhcpd start

Tip : To restrict dhcpd to a particular interface edit the following file :

# cat /etc/sysconfig/dhcpd
# Command line options here
DHCPDARGS="eth1"

Step 4 – enable nfs and export the rootfs :

# vi /etc/exports
 /netboot/            *(rw,async,no_root_squash)

# service rpcbind start
# service nfs start

Step 5 – generate the initramfs file (here is where the magic happens):

# yum install dracut-network -y
# dracut -f /boot/netboot6.img `uname -r` root=dhcp root-path=nfs:192.168.50.254:/netboot/

Step 6 – configure pxe :

# mkdir /var/lib/tftpboot/pxelinux.cfg/
# cp /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot/
# cp /boot/netboot6.img /var/lib/tftpboot/
# cp /boot/vmlinuz-2.6.32-71.el6.x86_64 /var/lib/tftpboot

# vi /var/lib/tftpboot/pxelinux.cfg/default

 default netboot6
 timeout 30
 prompt 1

 label netboot6
 kernel /vmlinuz-2.6.32-71.el6.x86_64
 append initrd=/netboot6.img rw root=nfs:192.168.50.254:/netboot/ selinux=0 enforcing=0

On the Client :

Configure the client to boot over the network (i.e. enable pxe boot) and start the system.

The following screen shots show a successful boot :

List information about block devices using lsblk

lsblk let’s you view information about block devices on a GNU/Linux system.

The output is tree-like and very elegant, with information that would generally require running 3-4 commands. It combines data from mount, df , dmsetup, lvm and raid commands.

lsblk get it’s data from the /sys filesystem.

Here is lsblk output on my system :

$ lsblk
NAME                       MAJ:MIN RM   SIZE RO MOUNTPOINT
sda                          8:0    0 298.1G  0
├─sda1                       8:1    0   500M  0 /boot
└─sda2                       8:2    0 297.6G  0
  ├─vg_rags-lv_swap (dm-0) 253:0    0   5.8G  0 [SWAP]
  ├─vg_rags-lv_root (dm-1) 253:1    0    50G  0 /
  └─vg_rags-lv_home (dm-2) 253:2    0 241.8G  0
    └─home (dm-3)          253:3    0 241.8G  0 /home
sr0                         11:0    1  1024M  0

lsblk is part of the util-liunx package :

$ rpm -qf `which lsblk`
util-linux-2.19.1-1.4.fc15.x86_64

Running fdisk -l gives similar data. But, fdisk requires root privileges, and It does not understand dm or lvm partitions.

Using WebDAV to access Box.net files on a Fedora system

Dropbox is by far the leader in the consumer “cloud storage” space; It is simple to use, feature rich and just works! But, there is only so much you can do with the provided 2 Gb free storage.

Enter: Box.net  — which has offered (for a limited period) 50 Gb free, lifetime storage, for all Ipone and Ipad users. Check here for the details.

However, I soon found out that Box.net does not provide syncing features that Dropbox is so good at, and there are no clients for Linux.

Lucky, for Linux users, a WebDAV interface exists.

I used fedora 15, but the instructions are generic enough that they should work on any release or any Linux distribution for that matter.

1. Create a folder to mount the box.net share :

$ mkdir ~/box

2. Install davfs2 :

$ sudo yum install davfs2
Note : Nautilus does support webdav, but I’ve found it to be buggy.

3. Run the following to add your user to the davfs2 group :

$ sudo /sbin/usermod -a -G davfs2 "username"

4. Disable locking, as this causes problems with box.net :

$ mkdir ~/.davfs2
$ vi ~/.davfs2/davfs2.conf

and add the following :

use_locks 0

5. (Optional) If you do not want to be prompted for the box.net username/password :

$ vi ~/.davfs2/secrets

and add the following :

http://www.box.net/dav  user@email.com   password

6. Add the following entry to /etc/fstab :

$ sudo vi /etc/fstab

http://www.box.net/dav /home/"username"/box  davfs rw,user,noauto 0 0
Note: Don’t forget to replace “username” with your actual username.

7. Finally, to mount the folder run :

$ mount  ~/box

and the files from your box.net account should be visible.

Use “rsync”, “unison”, or just plain “cp” to sync files directly to box.net. I find this very convenient for backups and file sharing.

How to debug ntp issues?

Ntp has been the de-facto protocol used by computers to synchronize their clocks over a network, and maintain very accurate time, with as much as 10 millisecond precision. The ntp daemon or ntpd is the reference implementation, that can be found running on almost all Linux (and Unix) systems. This may change in the future though, as Chrony is going to replace ntpd, and will be the default ntp client in Fedora 16. Nevertheless, many systems use ntpd, and I don’t see it going away any time soon.

In this post, we will take a brief look at how the ntp daemon works and look at ways to debug some common issues.

When the ntp service first starts, a clock selection process begins, with the daemon polling the servers configured in ntp.conf, at 64 second intervals. Depending on the configuration, this process can take 5 to 10 minutes. To check the status, run the following :

# ntpq
ntpq> peers
     remote           refid           st t when poll reach   delay   offset  jitter
=======================================================================================
*time.ferea.org       8.16.24.15       2 u  972 1024  377   28.066   -0.181   4.126
+dg1.rieta.net        15.15.26.3       3 u  467 1024  377  141.664  -23.531   0.140
 mighty.poclabs.      .STEP.          16 u    - 1024    0    0.000    0.000   0.000
 LOCAL(0)             .LOCL.          10 l   32   64  377    0.000    0.000   0.001

During the clock selection process the refid column should read .INIT.  and the st (stratum) set to 16.

The * indicates that this particular association is the chosen ntp source.
The  + indicates that this ntp peer is a candidate (a peer is a ntp server on the same stratum).
An empty space indicates that the server is unreachable and therefore rejected (stratum 16).

If the current local time is greater than 1000 seconds, ntpd will not set the clock. The time can then be manually set using the “date” command or using “ntpdate” :

# ntpdate time.ferea.org

If no ntp servers get selected, run the following :

ntpq> as

ind assID status  conf reach auth condition  last_event cnt
===========================================================
  1 29581  9624   yes   yes  none  sys.peer   reachable  1
  2 29582  9014   yes   yes  none  candidat   reachable  1
  4 29583  8000   yes   yes  none    reject
  5 29584  9024   yes   yes  none    reject   reachable  2

The associations shown above correspond to the entries shown in the peer command. Most of the fields are self-explanatory,  except the status column. Use the table here to decipher the status codes.

Use the “assID” for the following command  :

ntpq> rv 29583

assID=62236 status=9014 reach, conf, 1 event, event_reach,
srcadr=192.168.23.1, srcport=123, dstadr=192.168.247.11, dstport=123,
leap=00, stratum=3, precision=-6, rootdelay=218.750,
rootdispersion=1381.516, refid=24.1.4.14, reach=377, unreach=0,
hmode=3, pmode=4, hpoll=10, ppoll=10, flash=400 peer_dist, keyid=0,
ttl=0, offset=-29.750, delay=0.316, dispersion=30.400, jitter=1.136,
reftime=d1e4505b.d456f5b0  Thu, Aug  4 2011  0:55:23.829,
org=d1e4c793.e477ba4b  Thu, Aug  4 2011  9:24:03.892,
rec=d1e4c793.ec1fc3ac  Thu, Aug  4 2011  9:24:03.922,
xmt=d1e4c793.ec0b133c  Thu, Aug  4 2011  9:24:03.922,
filtdelay=     0.32    0.40    0.33    0.45    0.42    0.42    0.33    0.38,
filtoffset=  -29.75  -30.89  -29.97  -30.11  -30.15  -29.20  -30.25  -30.36,
filtdisp=     15.63   31.00   46.38   61.75   77.14   92.52  107.91  123.28

The flash codes in the rv command output give the reason for the ntp source to get rejected :

flash=400 peer_dist

This flash code corresponds to “distance threshold exceeded”. Check all the flash codes here.

Also, check the following variables :

rootdispersion=1381.516
dispersion=30.400
jitter=1.136

Dispersion is an estimate of error, and a large value indicates that the ntp server is not a reliable source, and can indicate conditions such as severe packet loss and network congestion.

Another useful aid is to run ntpdate with the -d switch :

# ntpdate -d time.rhl.com

17 Oct 00:20:51 ntpdate[26388]: ntpdate 4.2.2p1@1.1570-o Thu Nov 26 11:34:35 UTC 2009 (1)
Looking for host time.rhl.com and service ntp
host found : time.rhl.com
transmit(66.125.13.54)
receive(66.125.13.54)
transmit(66.125.13.54)
receive(66.125.13.54)
transmit(66.125.13.54)
receive(66.125.13.54)
transmit(66.125.13.54)
receive(66.125.13.54)
transmit(66.125.13.54)
server 66.125.13.54, port 123
stratum 1, precision -16, leap 00, trust 000
refid [CDMA], delay 0.32297, dispersion 0.00040
transmitted 4, in filter 4
reference time:    d245a5fe.2fdfe09b  Mon, Oct 17 2011  0:20:38.187
originate timestamp: d245a60c.e2117d1e  Mon, Oct 17 2011  0:20:52.883
transmit timestamp:  d245a60c.b9c9b413  Mon, Oct 17 2011  0:20:52.725
filter delay:  0.32361  0.32382  0.32297  0.32619
         0.00000  0.00000  0.00000  0.00000
filter offset: 0.003892 0.004005 0.003607 0.004972
         0.000000 0.000000 0.000000 0.000000
delay 0.32297, dispersion 0.00040
offset 0.003607
17 Oct 00:20:53 ntpdate[26388]: adjust time server 66.187.233.4 offset 0.003607 sec

Most, if not all ntp issues can be resolved with the information gathered from the above commands.

Do you have any tips on debugging ntp problems?