Configuring Zabbix Monitoring For All Hadoop Services (Zookeeper,Spark, namenode, datanode , job history server , hdfs journal node, hive and HBase)

Below info document Zabbix monitoring configuration for all Hadoop services like Zookeeper,Spark, namenode, datanode , job history server , hdfs journal node, hive and HBase with respect file changes.

For Zookeeper:
vi /etc/default/zookeeper
export SERVER_JVMFLAGS="-Dcom.sun.management.jmxremote.port=1703 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.password.file=/opt/shekhar/jmx/zookeeper.password -Dcom.sun.management.jmxremote.access.file=/opt/shekhar/jmx/jmx.access"
---------------------------------
Spark:
vi /etc/spark/conf/metrics.properties
*.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink
*.source.jvm.class=org.apache.spark.metrics.source.JvmSource
---------------------------------
For namenode, datanode and job history server and hdfs journal node:
vi /etc/hadoop/conf/hadoop-env.sh

# Use YARN for all hadoop commands
export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce

# Enable NameNode JMX connections on port 9980
HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote.port=9980 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.password.file=/opt/shekhar/jmx/hdfs.password -Dcom.sun.management.jmxremote.access.file=/opt/shekhar/jmx/jmx.access"

# Enable DateNode JMX connections on port 9981
HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote.port=9981 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.password.file=/opt/shekhar/jmx/hdfs.password -Dcom.sun.management.jmxremote.access.file=/opt/shekhar/jmx/jmx.access"

# Enable JournalNode JMX connections on port 1704
HADOOP_JOURNALNODE_OPTS="${HADOOP_JOURNALNODE_OPTS} -Dcom.sun.management.jmxremote.port=1704 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.password.file=/opt/shekhar/jmx/hdfs.password -Dcom.sun.management.jmxremote.access.file=/opt/shekhar/jmx/jmx.access"

# Enable Job History Server JMX connections on port 9991
HADOOP_JOB_HISTORYSERVER_OPTS="${HADOOP_JOB_HISTORYSERVER_OPTS} -Dcom.sun.management.jmxremote.port=9991 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.password.file=/opt/shekhar/jmx/mapred.password -Dcom.sun.management.jmxremote.access.file=/opt/shekhar/jmx/jmx.access"

----------------------------
For resourcemanager, nodemanager:
vi /etc/hadoop/conf/yarn-env.sh
YARN_RESOURCEMANAGER_OPTS="-Dcom.sun.management.jmxremote.port=9983 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.password.file=/opt/shekhar/jmx/yarn.password -Dcom.sun.management.jmxremote.access.file=/opt/shekhar/jmx/jmx.access"

YARN_NODEMANAGER_OPTS="-Dcom.sun.management.jmxremote.port=9984 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.password.file=/opt/shekhar/jmx/yarn.password -Dcom.sun.management.jmxremote.access.file=/opt/shekhar/jmx/jmx.access"
---------------------------------
For Hive

vi /etc/hive/conf/hive-env.sh
JMX_OPTS="-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.password.file=/opt/shekhar/jmx/hive.password -Dcom.sun.management.jmxremote.access.file=/opt/shekhar/jmx/jmx.access"
if [ "$SERVICE" = "hiveserver2" ]; then
  export HADOOP_OPTS="$HADOOP_OPTS $JMX_OPTS -Dcom.sun.management.jmxremote.port=1705 -Dproc_hiveserver"
elif [ "$SERVICE" = "metastore" ]; then
  export HADOOP_OPTS="$HADOOP_OPTS $JMX_OPTS -Dcom.sun.management.jmxremote.port=1706 -Dproc_metastore"
fi
----------------------------------
For HBase:

vi /etc/hbase/conf/hbase-env.sh
HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.password.file=/opt/shekhar/jmx/hbase.password -Dcom.sun.management.jmxremote.access.file=/opt/shekhar/jmx/jmx.access"
export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10101"
export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10102"
export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10103"
#export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10104"
export HBASE_REST_OPTS="$HBASE_REST_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10105"