Skip to content

How to get active NameNode

Jump to bottom
stanislawbartkowski edited this page Sep 23, 2019 · 6 revisions

Problem

If HA (high availability) is enabled, one NameNode is active and the second is passive. But which one? While dealing with WebHDFS, the request should be sent to the active NameNode, otherwise, the request will be rejected. Sounds strange, but it looks that there is not Ambari REST API answering the question.

Knox Gateway

The obvious solution is to use Knox GateWay. If properly configured, the gateway will direct the request to the currently active NameNode.
https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.0.1/configuring-proxy-knox/content/setting_up_knox_services_for_ha.html

Query JMX

https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/data-storage/content/using_jmx_for_accessing_hdfs_metrics.html
The query to extract if NameNode is active or passive.The {hostname} should be the hostname of the node where NameNode, active or passive, is installed.

curl http://{hostname}:50070/jmx?qry=Hadoop:service=NameNode,name=NameNodeStatus Example assuming that hurds1.fyre.ibm.com and a1.fyre.ibm.com nodes contain NameNodes.
curl http://hurds1.fyre.ibm.com:50070/jmx?qry=Hadoop:service=NameNode,name=NameNodeStatus

{
  "beans" : [ {
    "name" : "Hadoop:service=NameNode,name=NameNodeStatus",
    "modelerType" : "org.apache.hadoop.hdfs.server.namenode.NameNode",
    "NNRole" : "NameNode",
    "HostAndPort" : "hurds1.fyre.ibm.com:8020",
    "SecurityEnabled" : false,
    "LastHATransitionTime" : 0,
    "BytesWithFutureGenerationStamps" : 0,
    "SlowPeersReport" : null,
    "SlowDisksReport" : null,
    "State" : "standby"
  } ]
}

curl http://a1.fyre.ibm.com:50070/jmx?qry=Hadoop:service=NameNode,name=NameNodeStatus

{
  "beans" : [ {
    "name" : "Hadoop:service=NameNode,name=NameNodeStatus",
    "modelerType" : "org.apache.hadoop.hdfs.server.namenode.NameNode",
    "NNRole" : "NameNode",
    "HostAndPort" : "a1.fyre.ibm.com:8020",
    "SecurityEnabled" : false,
    "LastHATransitionTime" : 1568796935217,
    "BytesWithFutureGenerationStamps" : 0,
    "SlowPeersReport" : null,
    "SlowDisksReport" : null,
    "State" : "active"
  } ]
}

If the cluster is Kerberized, the valid Kerberos ticket should be obtained and curl --negotiate parameter added to the commandline.

kinit {obtain Kerberos ticket}
curl -i --negotiate -u : http://mdp1:50070/jmx?qry=Hadoop:service=NameNode,name=NameNodeStatus

{
  "beans" : [ {
    "name" : "Hadoop:service=NameNode,name=NameNodeStatus",
    "modelerType" : "org.apache.hadoop.hdfs.server.namenode.NameNode",
    "NNRole" : "NameNode",
    "HostAndPort" : "mdp1.sb.com:8020",
    "SecurityEnabled" : true,
    "LastHATransitionTime" : 0,
    "BytesWithFutureGenerationStamps" : 0,
    "SlowPeersReport" : null,
    "SlowDisksReport" : null,
    "State" : "active"
  } ]
}

Use hdfs command.

https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#haadmin
This command can be executed only localy, on the node belonging to the cluster.
Discover node IDs assigned to the NameNodes.

vi /etc/hadoop/conf/hdfs-site.xml

 <property>
      <name>dfs.namenode.http-address.mycluster.nn1</name>
      <value>a1.fyre.ibm.com:50070</value>
    </property>

    <property>
      <name>dfs.namenode.http-address.mycluster.nn2</name>
      <value>hurds1.fyre.ibm.com:50070</value>
    </property>

NameNode a1.fyre.ibm.com is labelled as nn1 and hurds1.fyre.ibm.com as nn2.

hdfs haadmin -getServiceState nn1

active

hdfs haadmin -getServiceState nn2

standby

A simple bash script to discover an active NameNode

This script is pretty straightforward and does not require match comment.
Customization.

Variable Description
NN1NODE a hostname where nn1 NameNode is deployed
NN2NODE a hostname where nn2 NameNode is deployed
KEYTAB keytab used to deal with Kerberized cluster. If cluster is not Kerberized, comment out the variable
PRINCIPAL principal in KEYTAB
USEJXM Set this variable if the active name node is extracted from JMX. Comment out if hdfs haadmin command is going to be used 
NN1NODE=a1.fyre.ibm.com
NN2NODE=hurds1.fyre.ibm.com

#NN1NODE=mdp1.sb.com
#NN2NODE=mdp1.sb.com

NN1=nn1
NN2=nn2

PORT=50070

# comment out if cluster not Kerberized
#KEYTAB=/home/sbartkowski/bin/keytabs/sb.keytab
#KEYTAB=/home/sb/sb.keytab
#PRINCIPAL=sb@CENTOS.COM.REALM

KERBEROSPARAM=

# comment out if use hdfs haadmin
# remove comment if jmx is to be used 

#USEJMX=X

checkjmx() {
    local -r host=$1
    curl $KERBEROSPARAM -# http://$host:$PORT/jmx?qry=Hadoop:service=NameNode,name=NameNodeStatus 2>/dev/null | 
        sed -n 's/\"State" *: *"\(active\|standby\)\"/\1/p'
}

checkhdfs() {
  local -r nn=$1
  hdfs haadmin -getServiceState $nn
}

checkstate() {
    [ -n "$USEJMX" ] && checkjmx $1
    [ -z "$USEJMX" ] && checkhdfs $2
}

getactive() {
    if [ `checkstate $NN1NODE $NN1`X == 'activeX' ]; then echo $NN1NODE; 
      elif [ `checkstate $NN2NODE $NN2`X == 'activeX' ]; then echo $NN2NODE; 
        else echo "No active NameNode, verify that cluster is up and running."
    fi   
}

checkkerberos() {
    if [ -n "$KEYTAB" ]; then 
      kinit -kt $KEYTAB $PRINCIPAL
      if [ $? -ne 0 ]; then 
        echo "kinit - cannot obtain Kerberos ticket."
        exit
      fi
      KERBEROSPARAM=" -i --negotiate -u : "
    fi
}

checkkerberos
getactive
[ -n "$KEYTAB" ] && kdestroy -A
Clone this wiki locally