Implementing High Available Applications on RHEL 7.

Julio Villarreal Pelegrino, RHCA

Senior Cloud Architect

Cloud Practice - Red Hat, Inc.


What is High Availability (HA)?

In IT, High Availability refers to a system or component that is continuously operational for a desirably long length of time.

3 cores principles to HA:

  • Elimination of single point of failures.
  • Reliable crossover.
  • Detection of failures as they occur.

How High Availability is implemented on RHEL 7?


What is Clustering?

A cluster is a set of computers working together on a single task. Which task is performed, and how that task is performed, differs from cluster to cluster.

There are four (4) different kinds of clusters:

High-availability clusters

Known as an HA cluster or failover cluster, their function is to keep running services as available as they can be. You could find them in two main configurations:

  • Active-Active (where a service runs on multiple nodes).
  • Active-Passive (where a service only runs on one node at a time).
Load-balancing clusters

All nodes run the same software and perform the same task at the same time and the requests from the clients are distributed between all the nodes.

Compute clusters

Also know as high-performance computing (HPC) cluster. In these clusters tasks are divided into smaller chunks, which then get computed on different nodes.

Storage clusters

All nodes provide a single cluster file system that will be used by clients to read and write data simultaneously.

Benefits of Highly Available applications.

In two words: "Application resiliency".

  • Apply patches.
  • Planned outages.
  • Unplanned outages due to failures (server, software, network, storage).

Avoid this!

Nuclear App Fallout!

How HA is implemented on RHEL 7.

Red Hat Enterprise Linux High Availability Add-On.

The High Availability Add-On consists of the following major components:

Cluster infrastructure

Provides fundamental functions for nodes to work together as a cluster: configuration-file management, membership management, lock management, and fencing.

High availability Service Management

Provides failover of services from one cluster node to another.

Cluster administration tools

Configuration and management tools for setting up, configuring, and management.

To provide the above services multiple software components are required on the cluster nodes.

How HA is implemented on RHEL 7.


The cluster infrastructure software is provided by Pacemaker and performs the next set of functions:

  • Cluster management
  • Lock management
  • Fencing
  • Cluster configuration management

Cluster software:


It’s responsible for all cluster-related activities, such as monitoring cluster membership, managing the services and resources, and fencing cluster members. The RPM contains three (3) important components:

  • Cluster Information Base (CIB).
  • Cluster Resource Management Daemon (CRMd).

This is the framework used by Pacemaker for handling communication between the cluster nodes.


Provides a command-line interface to create, configure, and control every aspect of a Pacemaker/corosync cluster.

Requirements and Support I.

Here are some requirements and limits for Pacemaker.

Number of Nodes:

  • Up to 16 nodes per cluster.
  • Minimum number of nodes: 3.
    • 2 nodes cluster could be configured but is not recommended.

Cluster location:

  • Single site: A cluster setup where all cluster members are in the same physical location, connected by a local area network. (Supported).
  • Multisite: Two clusters, one active and one for disaster recovery. Failover for multisite clusters must be managed manually. (Supported).
  • Strech (or) Geo Clusters: Clusters stretched out over multiple physical locations. (Required architecture review to be supported).


Fencing is the process of cutting a node off from shared storage. This can be done by power cycling a node or disabling communication to the storage level.

Warning Fencing is required for all nodes in the cluster, either via power fencing, storage fencing, or a combination of both.
Note If the cluster will use integrated fencing devices like ILO or DRAC, the systems acting as cluster nodes must power off immediately when a shutdown signal is received, instead of initiating a clean shutdown.

Requirements and Support II.


Virtual Machines supported as nodes and resources.

Note VM as a resource means that virtualization host is participating in a cluster and the virtual machine is a resource that can move between cluster nodes.



  • Multicast and IGMP (Internet Group Management Protocol).
  • Gratuitous ARP used for floating IP Address.


  • 5405/UDP - corosync
  • 2224/TCP - pcsd
  • 3121/TCP - pacemaker
  • 21064/TCP - dlm

RHN Channels


  • rhel-7-server-rpms
  • rhel-ha-for-rhel-7-server-rpms

Demo: Building a 3 node Apache cluster I.

Preparing the systems:

All this actions will happen on the all the cluster nodes.

Configure Firewall

Let’s start by configuring FirewallD to allow traffic.

firewall-cmd --permanent --add-service=high-availability
firewall-cmd --reload

Install required software

yum install pcs fence-agents-all

The pcs package requires corosync and pacemaker, so all your software be installed by doing this.

Enable pcsd.

pcsd provides cluster configuration sync and the web front end. Needs to be enable in all the servers.

systemctl enable pcsd; systemctl start pcsd

Set the hacluster user password.

After the software install, a user hacluster will be created. This user will be used for all cluster communication (pcsd).

Note You should use the same password across all cluster nodes for this user. If you echo your password like is show below, clear your history afterwards :) .
echo password | passwd hacluster --stdin

Configuring DNS.

You should be able to resolve all the nodes in the cluster by name. On this example we are going to use host files to define our 3 nodes. This is whati added to my hosts files (/etc/hosts).    node1   node1.example.local    node2   node2.example.local    node3   node3.example.local

Demo: Building a 3 node Apache cluster II.

Preparing the systems:

Authenticate pcsd.

pcsd requires that the cluster nodes authenticate, we are going to use the hacluster user and password. This actions only needs to happen on one of the nodes.

[[email protected]] pcs cluster auth node1.example.local node2.example.local node3.example.local

Creating the cluster.

Lets create the cluster:

pcs cluster setup --name demo-cluster --start node1.example.local node2.example.local node3.example.local

An important step will be to enable the cluster services on all nodes. By default if a node is rebooted will not join the cluster until started manually. To avoid this to happen do:

[[email protected]] pcs cluster enable --all

Check the cluster status:

[[email protected]] pcs cluster status

Configuring Fencing.

This is a critical step, you must have fencing on the cluster! In our case we are using KVM for the demo and we will use fence_xvm.

[[email protected]] pcs stonith create fence_node1_vm fence_xvm port="node1" pcmk_host_list="node1.example.local"

[[email protected]] pcs stonith create fence_node2_vm fence_xvm port="node2" pcmk_host_list="node2.example.local"

[[email protected]] pcs stonith create fence_node3_vm fence_xvm port="node3" pcmk_host_list="node3.example.local"

Open the port for the fencing agent fencing:

[[email protected]] for i in `seq 1 3`; do ssh [email protected]$i.example.local firewall-cmd --add-port=1229/tcp --permanent; done

[[email protected]] for i in `seq 1 3`; do ssh [email protected]$i.example.local firewall-cmd reload; done

Check fencing status.

[[email protected]] pcs stonith show

Demo: Building a 3 node Apache cluster III.


Clustered services consist of one or more resources. A resource can be:

  • IP address
  • file system
  • Service (example: httpd)

Also usually the resources are member of resource groups.

Creating the resources for our demo-cluster.

First lets create the resource group for our Apache cluster, we are going to name it personal-web and will have a floating IP.

[[email protected]] pcs resource create floatingip IPaddr2 ip= cidr_netmask=24 --group personal-web
  • Install Apache (httpd).
[[email protected]] yum install httpd -y
  • Create the web-1 resource using Apache and put it on the personal-web group.
[[email protected]] pcs resource create web-1 apache --group personal-web
  • Let’s check that the created resources are present in the cluster configuration and check the cluster status.
[[email protected]] pcs resource show

[[email protected]] pcs status
  • Create a file in /var/www/html/ with this content:
echo "Website responding from $HOSTNAME" > /var/www/html/index.html

Demo: Building a 3 node Apache cluster IV.

Manipulating the cluster.

Here are some commands that will help you to manage the cluster.

  • Start and Stop the cluster , to do it in all nodes use the --all switch.
[[email protected]] pcs cluster start

[[email protected]] pcs cluster stop
  • Stop the cluster service in an specific remote node:
[[email protected]] pcs cluster stop
  • Disable cluster on reboot on a node:
[[email protected]] pcs cluster disable
  • How to add a node to the cluster:
[[email protected]] pcs cluster node add

On the new node, you need to Authenticate the rest of the cluster. Also you will need to add a fence device for it too.

[[email protected]] pcs cluster auth
  • How to remove a node to the cluster:
[[email protected]] pcs cluster node remove

[[email protected]] pcs stonith remove fence_newnode
  • Set a node in standby. (This bans the node from hosting resources).
[[email protected]] pcs cluster node standby
  • Set the cluster in standby:
[[email protected]] pcs cluster standby --all
  • Unset standby
[[email protected]] pcs cluster unstandby --all
  • Displaying quorum status.
[[email protected]] corosync-quorumtool


Contact Info: