Openfiler – Creating a HA Cluster

Openfiler – Creating a HA Cluster

Introduction

Openfiler can be configured as a High Availability Active / Passive cluster and in this article the steps to implement this solution are explained. The infrastructure used for this article was using two virtual machines running on a Fedora Core 8 64-bit server with VMware Server 1.0.8 running.

Important Notes

After seeing that this article gets a few hits I have re-ran through the installation routine and found that there are some issues. The first issue is that the drbd.conf file is written incorrectly and the IP Addresses:Ports in the resource vg0drbd were a duplicate of the cluster_metadata IP Addresses:Ports.  This has now been rectified in the article and I apologise for any trouble caused.

Pre-requisitites

In order to implement an Openfile HA cluster the following pre-requisites must be met :-

  • 2 x Servers with the same hardware configurations and drive sizes
  • 2 x Ethernet cards in each server

Node build configuration

Both nodes used in the article were built the same and the configuration settings used are provided in the following sub-sections. Hardware Configuration

  • 2 x 5Gb Hard drives – /dev/sda and /dev/sdb
  • 2 x Ethernet Cards
  • 256Mb of RAM

Disk Layout

  • Partition Name = /boot – Filesystem = efs3 – Partition Size = 100Mb – Disk = /dev/sda1
  • Partition Name = /swap – Filesystem = swap – Partition Size = 1024Mb – Disk = /dev/sda2
  • Partition Name = / – Filesystem = efs3 – Partition Size = 3992Mb – Disk = /dev/sda3
  • Partition Name = /meta – Filesystem = efs3 – Partition Size = 512Mb – Disk = /dev/sdb1
  • Partition Name = /data – Filesystem = lvm – Partition Size = 4608Mb – Disk = /dev/sdb2

Network Configuration – Node1

  • Network Card = Eth0 – LAN – Production LAN – IP Address = 192.168.0.50
  • Network Card = Eth1 – LAN – Replication LAN – IP Address = 192.168.211.50

Network Configuration – Node2

  • Network Card = Eth0 – LAN – Production LAN – IP Address = 192.168.0.60
  • Network Card = Eth1 – LAN – Replication LAN – IP Address = 192.168.211.60

Adding the host file entries

In order to communicate correctly and not rely upon DNS it is necessary to configure the hostnames for the cluster nodes in each servers host file.  To add the required entries in to the servers host files perform the following steps :-

  1. Logon to the Openfiler server using SSH
  2. Navigate to /etc and edit the file named hosts
  3. Add a new line to the bottom of the hosts file as shown below

{Other Nodes Replication Address} {Node Name} E.G. on Node1 add a line in as shown below substituting the 2nd Nodes replication IP address and name 192.168.211.60  Node2

Generating the ssh shared keys

To allow the two Openfilers to communicate it is necessary to configure ssh access using shared keys.  To generate and configure this perform the following steps on each server :-

  1. Logon to the Openfiler server using SSH
  2. At the command prompt execute the command ssh-keygen -t dsa
  3. Accept the Default Location to save the file
  4. When prompted by pressing Enter to set no passphrase
  5. Confirm no passphrase by pressing Enter again
  6. Once created copy the file /root/.ssh/id_dsa.pub to the /root/.sshfolder on other node and name the files as authorized_keys2

Configuring DRBD (Distributed Replicated Block Device)

The next step in implementing an Openfiler HA cluster is to configure the DRBD to provide a distributed filesystem over the network.  In order to configure DRBD perform the following steps :-

  1. If the file exists move /etc/drbd.conf file to /etc/drbd.conf.old
  2. Create a new /etc/drbd.conf file and add the text shown below these steps, substituting the {Node 1} and {Node 2} markers for the two node names in the cluster, the {Replication IP} markers for the replication IP Addresses, and the {Partition Number} for the partition number to be assigned, E.G. /dev/sdb1.
  3. Save the new /etc/drbd.conf file and copy it to the second node
global {
    # minor-count 64;
    # dialog-refresh 5; # 5 seconds
    # disable-ip-verification;
    usage-count ask;
 }
 
 common {
    syncer { rate 100M; }
 }
 
 resource cluster_metadata {
    protocol C;
    handlers {
    pri-on-incon-degr "echo O > /proc/sysrq-trigger ; halt -f";
    pri-lost-after-sb "echo O > /proc/sysrq-trigger ; halt -f";
    local-io-error "echo O > /proc/sysrq-trigger ; halt -f";
    # outdate-peer "/usr/sbin/drbd-peer-outdater";
  }
  
  startup {
    # wfc-timeout 0;
    degr-wfc-timeout 120; # 2 minutes.
  }
 
  disk {
    on-io-error detach;
  }
 
  net {
    after-sb-0pri disconnect;
    after-sb-1pri disconnect;
    after-sb-2pri disconnect;
    rr-conflict disconnect;
  }
 
  syncer {
    # rate 10M;
    # after "r2";
    al-extents 257;
  }
 
  on {Node 1} {
    device /dev/drbd0;
    disk {partition number};
    address {Replication IP}:7788;
    meta-disk internal;
  }
 
  on {Node 2} {
    device /dev/drbd0;
    disk{partition number};
    address {Replication IP}:7788;
    meta-disk internal;
  }
 }
 
 resource vg0drbd {
    protocol C;
    startup {
    wfc-timeout 0; ## Infinite!
    degr-wfc-timeout 120; ## 2 minutes.
  }
 
  disk {
    on-io-error detach;
  }
 
  net {
    # timeout 60;
    # connect-int 10;
    # ping-int 10;
    # max-buffers 2048;
    # max-epoch-size 2048;
  }
 
  syncer {
    after "cluster_metadata";
  }
 
  on {Node 1} {
    device /dev/drbd1;
    disk {partition number};
    address {Replication IP}:7789; 
    meta-disk internal;
  }
 
  on {Node 2} {
    device /dev/drbd1;
    disk {partition number};
    address {Replication IP}:7789;
    meta-disk internal;
  }
 }

Create the partitions for Drbd

The next step is to create the partitions for the Cluster Metadata and the Data partition you wish to be part of the cluster.  To create the required partitions perform the following steps on the Primary Node if the Cluster Disk is a shared disk or on both Nodes if they are locally attached :-

  1. At the command prompt execute fdisk {disk to be used}, E.G. /dev/hdb
  2. Within fidsk execute n to create a new partition
  3. When prompted select the type of Partition to be created, E.G. p for a primary partition
  4. When prompted select the Partition number to be created, E.G. 1 for the first on a second disk
  5. When prompted accept the First Cyclinder value and then set the set the Last Cyclinder to +512M
  6. The partition is created and you are returned to the main Command menu of fdisk
  7. Within fidsk execute n to create a new partition
  8. When prompted select the type of Partition to be created, E.G. p for a primary partition
  9. When prompted select the Partition number to be created, E.G. 2 for the second on a second disk
  10. When prompted accept the First Cyclinder value and then set the set the Last Cyclinder to the of partition desired in Mb
  11. The partition is created and you are returned to the main Command menu of fdisk
  12. Within fdisk execute w to save the partition changes to the disk and exit fdisk

Initialise the metadata

Once the configuration file has been created on both nodes perform the following steps on both nodes to initialise the metadata :-

  1. At the command prompt execute drbdadm create-md cluster_metadata
  2. At the command prompt execute drbdadm create-md vg0drbd

Start the drbd service

Once the drbd metadata has been initialised on both nodes of the cluster start the drbd service on both by perform the step below :-

  1. Confirm that the partitions to be used are not mounted
  2. In the command prompt on both nodes execute the commandservice drbd start

Configure the primary drbd node

Once the drbd service has started on both nodes of the cluster perform the steps below on the first node to configure it as the Primary node :-

  1. At the command prompt on the first node execute the command drbdsetup /dev/drbd0 primary -o
  2. At the command prompt on the first node execute the command drbdsetup /dev/drbd1 primary -o

Configure the drbd service to start at boot

To enable the drbd service to start at boot time perform the steps below on both nodes of the cluster :-

  1. At the command prompt execute the command chkconfig –level 2345 drbd on

Create the metadata partition filesystem

On the primary node of the cluster perform the step below to create the filesystem for cluster metadata partition :-

  1. At the command prompt on the first node execute the command mkfs.ext3 /dev/drbd0

Configure the LVM partition

To configure the second partition as an LVM partition perform the steps shown below :-

  1. Edit the file /etc/lvm/lvm.conf on both nodes
  2. Find the line below

filter = [ “a/.*/” ] and change it to filter = [ “r|{Second drive allocated in to the cluster” ] E.G. [ “r|/dev/sdb2|” ]

  1. Create the LVM partition on the first node only as it will be replicated to the second by executing the command pvcreate /dev/drbd1

Configure heartbeat

The heartbeat service controls failover between the two nodes and is ran over the second network card we set up using the Restricted IP address.  To configure the heartbeat service perform the steps below on both nodes :-

  1. Create the file /etc/ha.d/authkeys and add the text below
  2. Edit the file /etc/ha.d/ha.cf and add the text below
  3. Restrict access to the authkeys file by executing the command chmod 600 /etc/ha.d/authkeys

auth 2 2 crcdebugfile /var/log/ha-debug logfile /var/log/ha-log logfacility local0 bcast eth1 keepalive 5 warntime 10 deadtime 120 initdead 120 udpport 694 auto_failback off node filer01 node filer02

Configure the heartbeat service to start at boot

To enable the heartbeat service to start at boot perform the following step on both nodes :-

  1. At the command prompt execute the command chkconfig –level 2345 heartbeat on

Openfiler configuration changes

To ensure that the HA Services are able to run as expected and are available during failover perform the following steps on the nodes as directed :-

  1. At the command prompt on the primary node of the cluster execute the following commands :-mkdir /cluster_metadata mount /dev/drbd0 /cluster_metadata mv /opt/openfiler/ /opt/openfiler.local mkdir /cluster_metadata/opt cp -a /opt/openfiler.local /cluster_metadata/opt/openfiler ln -s /cluster_metadata/opt/openfiler /opt/openfiler rm /cluster_metadata/opt/openfiler/sbin/openfiler ln -s /usr/sbin/httpd /cluster_metadata/opt/openfiler/sbin/openfiler rm /cluster_metadata/opt/openfiler/etc/rsync.xml ln -s /opt/openfiler.local/etc/rsync.xml /cluster_metadata/opt/openfiler/etc/ mkdir -p /cluster_metadata/etc/httpd/conf.d cp /etc/httpd/conf.d/* /cluster_metadata/etc/httpd/conf.d
  2. Next edit the file /opt/openfiler.local/etc/rsync.xml and populate with the folowing text :-<?xml version=”1.0″ ?> <rsync> <remote hostname=”{Replication IP of secondary node}”/> <item path=”/etc/ha.d/haresources”/> <item path=”/etc/ha.d/ha.cf”/> <item path=”/etc/ldap.conf”/> <item path=”/etc/openldap/ldap.conf”/> <item path=”/etc/ldap.secret”/> <item path=”/etc/nsswitch.conf”/> <item path=”/etc/krb5.conf”/> </rsync>
  3. At the command prompt on the secondary node of the cluster execute the following commands :-mkdir /cluster_metadata mv /opt/openfiler/ /opt/openfiler.local ln -s /cluster_metadata/opt/openfiler /opt/openfiler
  4. Next edit the file /opt/openfiler.local/etc/rsync.xml and populate with the folowing text :-<?xml version=”1.0″ ?> <rsync> <remote hostname=”{Replication IP of secondary node}”/> <item path=”/etc/ha.d/haresources”/> <item path=”/etc/ha.d/ha.cf”/> <item path=”/etc/ldap.conf”/> <item path=”/etc/openldap/ldap.conf”/> <item path=”/etc/ldap.secret”/> <item path=”/etc/nsswitch.conf”/> <item path=”/etc/krb5.conf”/> </rsync>

Heartbeat cluster configuration

To configure the heartbeat services haresources file it is necessary to modify the Openfiler cluster.xml file.  To modify the Openfiler cluster.xml file perform the following steps :-

  1. On the primary node of the cluster edit the file /cluster_metadata/etc/cluster.xml and populate it with the following text :-<?xml version=”1.0″ ?> <cluster> <clustering state=”on” /> <nodename value=”{Primary Node Name” /> <resource value=”MailTo::it@company.com::ClusterFailover”/> <resource value=”IPaddr::{Cluster IP Address Required}/24″ /> <resource value=”drbddisk::”> <resource value=”LVM::vg0drbd”> <resource value=”Filesystem::/dev/drbd0::/cluster_metadata::ext3::defaults,noatime”> <resource value=”MakeMounts”/> </cluster>

Configuring Samba and NFS services support

In order for the Samba service to be part of the HA cluster perform the following steps :-

  1. At the command prompt on the primary node of the cluster execute the following commands :-mkdir /cluster_metadata/etc mv /etc/samba/ /cluster_metadata/etc/ ln -s /cluster_metadata/etc/samba/ /etc/samba mkdir -p /cluster_metadata/var/spool mv /var/spool/samba/ /cluster_metadata/var/spool/ ln -s /cluster_metadata/var/spool/samba/ /var/spool/samba mkdir -p /cluster_metadata/var/lib mv /var/lib/nfs/ /cluster_metadata/var/lib/ ln -s /cluster_metadata/var/lib/nfs/ /var/lib/nfs mv /etc/exports /cluster_metadata/etc/ ln -s /cluster_metadata/etc/exports /etc/exports
  2. At the command prompt on the secondary node of the cluster execute the following commands :-rm -rf /etc/samba/ ln -s /cluster_metadata/etc/samba/ /etc/samba rm -rf /var/spool/samba/ ln -s /cluster_metadata/var/spool/samba/ /var/spool/samba rm -rf /var/lib/nfs/ ln -s /cluster_metadata/var/lib/nfs/ /var/lib/nfs rm -rf /etc/exports ln -s /cluster_metadata/etc/exports /etc/exports

Configuring iSCSI services support

In order for the iSCSI service to be part of the HA cluster perform the following steps :-

  1. At the command prompt on the primary node of the cluster execute the following commands :-mv /etc/ietd.conf /cluster_metadata/etc/ ln -s /cluster_metadata/etc/ietd.conf /etc/ietd.conf mv /etc/initiators.allow /cluster_metadata/etc/ ln -s /cluster_metadata/etc/initiators.allow /etc/initiators.allow mv /etc/initiators.deny /cluster_metadata/etc/ ln -s /cluster_metadata/etc/initiators.deny /etc/initiators.deny
  2. At the command prompt on the secondary node of the cluster execute the following commands :-rm /etc/ietd.conf ln -s /cluster_metadata/etc/ietd.conf /etc/ietd.conf rm /etc/initiators.allow ln -s /cluster_metadata/etc/initiators.allow /etc/initiators.allow rm /etc/initiators.deny ln -s /cluster_metadata/etc/initiators.deny /etc/initiators.deny

Configuring a volume group

The next stage is to create a volume group which must be done before the heartbeat service is started.  To create the volume group perform the following steps :-

  1. At the command prompt on the primary node of the cluster execute the following commands :-vgcreate vg0drbd /dev/drbd1

Configuring and starting the heartbeat service for the first time

In order to get Openfiler to write the /etc/ha.d/haresources file based on the cluster.xml config file perform the following steps :-

  1. At the command prompt on the primary node of the cluster execute the following commands :-

rm /opt/openfiler/etc/httpd/modules ln -s /usr/lib64/httpd/modules /opt/openfiler/etc/httpd/modules

  1. Restart the Openfiler service on the primary node
  2. Log onto the primary node web interface
  3. Click on Services and enable SMB /CIFS and iSCSI
  4. At the command prompt on the primary node execute lvcreate -L 400M -n filer vg0drbd
  5. Copy the file /etc/ha.d/haresources from the primary node to the secondary node
  6. To enable the heartbeat configuration on the servers reboot the primary node first and then the secondary node

Testing the configuration

Once the two nodes have rebooted test the configuration performing the following steps :-

  1. Go to the cluster administration web page in a browser using the url https://{Cluster IP Address}:446
  2. When prompted logon to the Openfiler cluster
  3. Check under the System tab that the Hostname is showing as the Primary Node Name.
  4. Close the web administration page and shutdown the primary node
  5. Once shutdown re-open the cluster administration web page and logon
  6. Check under the System tab that the Hostname is showing as the Secondary Node Name.
  7. Close the web administration page and shutdown the secondary node
  8. Boot up the primary node and then the secondary node

Finally to clean up the configuration remove the temporary volume created earlier by performing the following step on the primary node :-

  1. At the command prompt on the primary node execute the command lvremove filer

Conclusion

The Openfiler cluster is now configured and you can now create new volumes for the provision of Samba shares or iSCSI disks. N.B. When creating new resources or enabling new services you must copy the /etc/ha.d/haresources file from the active node of the cluster to the passive node

Useful links

http://www.howtoforge.com/installing-and-configuring-openfiler-with-drbd-and-heartbeat