Hadoop Automatic Failover

How Hadoop automatically initiate NameNode Failover?

Want to learn how Hadoop initiates the failover from active NameNode to Standby NameNode?

In the Hadoop HDFS NameNode High Availability tutorial, we had seen how the High availability feature introduced in Hadoop 2 solves the NameNode SPOF (Single Point Of Failure ) problem. That tutorial describes the manual failover, where when the active node fails, then the standby node takes over the accountability of the active NameNode. But in manual failover, the system will not automatically trigger the failover from active NameNode to the standby NameNode.

In this tutorial, we will learn Hadoop Automatic Failover. We will first have a look at what failover is? And various types of failover. The key objective of this tutorial is to explain how HDFS achieves high availability through automatic failover. The tutorial explains the components, that is, ZooKeeper quorum and the ZKFailoverController deployed on Hadoop HDFS for achieving automatic failover.

Let us now first have a look at what is Failover.

What is Failover And Types of Failover

Failover submits to the procedure of transferring control to a redundant or standby system upon the failure of the previously active system.

There are two types of failover, that is, Graceful Failover and Automatic Failover.

1. Graceful Failover

Graceful Failover is initiated by the Administrator manually. In Graceful Failover, even when the active NameNode fails, the system will not automatically initiate the failover from active NameNode to standby Namenode. The Administrator initiates Graceful Failover, for example, in the case of routine maintenance.

2. Automatic Failover

In Automatic Failover, the system automatically initiates the failover from active NameNode to the Standby NameNode.

Introduction to Hadoop NameNode Automatic Failover

Hadoop Automatic Failover confirms Hadoop HDFS NameNode High Availability. It automatically initiates the failover from active Namenode to the standby NameNode. The default implementation uses ZooKeeper to attain Automatic Failover.

The two new components are deployed to Hadoop HDFS for implementing Automatic Failover. These two components are-

1. ZooKeeper quorum

2. ZKFailoverController process(ZKFC)

Let us now discuss each of these components in detail.

Hadoop - Automatic Failover Architecture

1. ZooKeeper quorum

Apache ZooKeeper quorum is a highly available service for maintaining little amounts of coordination data. It notifies the clients about the changes in that data. It monitors clients for the failures.

The HDFS implementation of automatic failover relies on ZooKeeper for the following things:

a. Failure detection

Each NameNode machine in the Hadoop cluster maintains a persistent session in the ZooKeeper. If any of the machines crashes, then the ZooKeeper session maintained will get terminated—zooKeeper than reveal to all the other NameNodes to begin the failover process.

 b. Active NameNode election

To fully select the active NameNode, ZooKeeper offers a simple mechanism. In the case of active NameNode failure, other standby NameNode may take the special exclusive lock in the ZooKeeper, stating that it should become the next active NameNode.

2. ZKFailoverController(ZKFC)

The ZKFC is the ZooKeeper client, which is also accountable for managing and monitoring the NameNode state. ZKFC runs on all machines on the Hadoop cluster, which is running NameNode.

It is responsible for:

• Health monitoring – ZKFC is responsible for health monitoring. It pings its local NameNode with health-check commands periodically. As long as the NameNode responds with a healthy status timely, it considers the NameNode as healthy. In the case, if the NameNode got crashed, froze, or entered an unhealthy state, then it marks the NameNode as unhealthy.

• ZooKeeper session management – It is also responsible for the session management with ZooKeeper. The ZKFC maintains a session open in the ZooKeeper when the native Namenode is healthy. Also, if the native NameNode is the active NameNode, then with the session, it also holds a special lock “znode”. This lock uses ZooKeeper support for the ”ephemeral” nodes. Hence, if the session gets terminated, the lock node will be deleted automatically.

• ZooKeeper-based election – When the native Namenode is healthy and ZKFC founds that no other NameNode acquires the lock “znode”, then it will try by itself to acquire the lock. If it gets successful in acquiring the lock, then ZKFC has won the election, and now it is accountable for running the failover to make it’s native NameNode active. The failover process run by the ZKFC is the same as the failover process run by the manual failover described in the NameNode High Availability tutorial.

Configuring automatic failover

As soon as the cluster is running, it is not possible to transfer from a manual failover setup to an automatic failover setup. Thus, before configuring automatic failover, the Hadoop cluster should be shut down.

For configuring automatic failover, we need to add two new parameters.

1. In hdfs-site.xml file, add:

  1. <property>
  2. <name>dfs.ha.automatic-failover.enabled</name>
  3. <value>true</value>
  4. </property>

This parameter specifies that the cluster is now set for automatic failover.

2. In core-site.xml file, add:

  1. <property>
  2. <name>ha.zookeeper.quorum</name> <value>zk1.example.com:2181,zk2.example.com:2181,zk3.example.com:2181</value>
  3. </property>

This parameter records the host-port pairs which are running the ZooKeeper service.

After adding the configuration keys, we have to reset required state in ZooKeeper using below command from one of the NameNode hosts:

  1. [hdfs]$ $HADOOP_HOME/bin/zkfc -formatZK

It will generate a lock “znode” in the ZooKeeper inside which the automatic failover system stores its data.

Now, begin the cluster with start-dfs.sh that will automatically start a ZKFC daemon on any machine that runs a NameNode.

ZKFCs now automatically selects one of the NameNodes to become active.

Summary

In this tutorial, we have studied the automatic failover that automatically prompts the NameNode failure. The two new components that are ZooKeeper quorum and ZKFC Automatic failover are deployed on HDFS for implementing automatic failure. The tutorial also describes the steps for configuring Automatic failure.