Alter Kafka topic partition

Steps to change Kafka Partitions

1. Create topic demo_topic13 with partitions 6

/opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 6 --topic demo_topic13

2. Describe topic before alter topic
[root@node1 ~]# /opt/kafka/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic
demo_topic13
Topic:demo_topic13      PartitionCount:6        ReplicationFactor:2     Configs:
        Topic: demo_topic13     Partition: 0    Leader: 3297    Replicas: 3297,2812     Isr: 3297,2812
        Topic: demo_topic13     Partition: 1    Leader: 2812    Replicas: 2812,3297     Isr: 2812,3297
        Topic: demo_topic13     Partition: 2    Leader: 3297    Replicas: 3297,2812     Isr: 3297,2812
        Topic: demo_topic13     Partition: 3    Leader: 2812    Replicas: 2812,3297     Isr: 2812,3297
        Topic: demo_topic13     Partition: 4    Leader: 3297    Replicas: 3297,2812     Isr: 3297,2812
        Topic: demo_topic13     Partition: 5    Leader: 2812    Replicas: 2812,3297     Isr: 2812,3297

3. alter partition with 7 (note: you con't decrease  partitions)
[root@node1 ~]# /opt/kafka/bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic demo_topic13 --partitions 7
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!

4. describe topic and check

[root@node1 ~]# /opt/kafka/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic demo_topic13
Topic:demo_topic13      PartitionCount:7        ReplicationFactor:2     Configs:
        Topic: demo_topic13     Partition: 0    Leader: 3297    Replicas: 3297,2812     Isr: 3297,2812
        Topic: demo_topic13     Partition: 1    Leader: 2812    Replicas: 2812,3297     Isr: 2812,3297
        Topic: demo_topic13     Partition: 2    Leader: 3297    Replicas: 3297,2812     Isr: 3297,2812
        Topic: demo_topic13     Partition: 3    Leader: 2812    Replicas: 2812,3297     Isr: 2812,3297
        Topic: demo_topic13     Partition: 4    Leader: 3297    Replicas: 3297,2812     Isr: 3297,2812
        Topic: demo_topic13     Partition: 5    Leader: 2812    Replicas: 2812,3297     Isr: 2812,3297
        Topic: demo_topic13     Partition: 6    Leader: 3297    Replicas: 3297,2812     Isr: 3297,2812

Hadoop Namenode and ZKFC tuning

Separate port(8022) for internal communication Changing handler count for better Name node performance:

Set below parameter in hdfs-site.xml

<property> <name>dfs.namenode.servicerpc-address.mas.NomeNodeHost1-com</name> <value>NomeNodeHost1.com:8022</value> </property> <property> <name>dfs.namenode.servicerpc-address.mas.NomeNodeHost1-com</name> <value>NomeNodeHost2.com:8022</value> </property>
For 15 data nodes: ln(number of DNs)*20= 2.708*20 = 54
<property> <name>dfs.namenode.handler.count</name> <value>55</value> </property> <property> <name>dfs.namenode.service.handler.count</name> <value>55</value> <!-- For internal service communications for 8022 -> </property> Steps to refresh cluster with above changes:
1. stop hadoop-hdfs-zkfc in both systems
service hadoop-hdfs-zkfc stop
2. remove hadoop-ha in zookeeper
$zookeeper-client
>rmr /hadoop-ha
3. create hadoop-ha using below command
sudo -u hdfs hdfs zkfc -formatZK
4. start both zkfc services
service hadoop-hdfs-zkfc start
5. start name node