拷贝kafka文件

$ cd /home
$ tar zcf kafka.tar.gz kafka --exclude=kafka/logs/*

然后scp到目标主机

确认broker id

选择不在集群内的broker id

$ cd /home/zookeeper/bin/
$ ./zkCli.sh
[zk: localhost:2181(CONNECTED) 0] ls /kafka/brokers/ids
[1, 2, 3, 4, 5]

修改配置

主要修改的就是IP地址和broker id

$ vi config/server.properties 
broker.id=6
listeners=PLAINTEXT://192.168.121.92:9092
advertised.listeners=PLAINTEXT://192.168.121.92:9092
log.dirs=/disk1/kafka/kafka-logs

格盘挂盘

挂载到对应的log目录

/disk1/kafka/kafka-logs/

启动服务

$ yum install -y java
$ ./bin/kafka-server-start.sh ./config/server.properties &

检查服务启动是否正常

$ ./bin/kafka-broker-api-versions.sh --bootstrap-server 192.168.121.30:9092

今天踩的坑是服务器防火墙规则导致服务没正常启动加入集群

重新分区

定义需要重新分区的topic,有两种方式

第一种是直接生成迁移计划

3是需要进行迁移的主机的broker id

$ ./bin/kafka-topics.sh --describe --zookeeper 192.168.121.4:2181/kafka | grep -B 1 "Replicas: 3" | egrep "^Topic" | grep -v '__consumer_offsets' | awk '{printf "{"}{printf $1}{print "\",\"partition\":0,\"replicas\":[6]},"}' | sed 's@Topic:@"topic":"@g' > topics-to-move.json

然后收尾加上以下配置构成json

{"partitions":[
和
],"version":4}

然后进行重新分区

$ ./bin/kafka-reassign-partitions.sh --zookeeper 192.168.121.4:2181/kafka --reassignment-json-file topics-to-move.json --execute

今天踩得坑是没有加url的/kafka

获取分区结果

$ ./bin/kafka-reassign-partitions.sh --zookeeper 192.168.121.4:2181/kafka --reassignment-json-file topics-to-move.json --verify

也可以获取剩余没迁移的topic数

$ /bin/kafka-topics.sh --describe --zookeeper 192.168.121.4:2181/kafka | grep "Leader: 3" | wc -l

第二种是获取需要迁移的topic，然后由kafka生成迁移计划

$ ./bin/kafka-topics.sh --describe --zookeeper 192.168.121.4:2181/kafka | grep -B 1 "Leader: 3"  | egrep "^Topic" | grep -v '__consumer_offsets' |  awk '{printf "{"}{printf $1}{printf "\"},"}' | sed 's@Topic:@"topic":"@g' > topics-to-move.json

然后收尾加上以下配置构成json

{"topics": [
和
],"version":1 }

生成迁移计划

$ ./bin/kafka-reassign-partitions.sh --zookeeper 192.168.121.4:2181/kafka --topics-to-move-json-file topics-to-move.json --broker-list "6" --generate

进行重新分区

$ ./bin/kafka-reassign-partitions.sh --zookeeper 192.168.121.4:2181/kafka --reassignment-json-file topics-to-move.json --execute

迁移过程中遇到的坑

迁移过程中没有迁移__consumer_offsets这个topic，这个topic有三个Replicas

$ ./bin/kafka-topics.sh --describe --zookeeper 192.168.121.4:2181/kafka | grep "__consumer_offsets"  
Topic:__consumer_offsets    PartitionCount:50   ReplicationFactor:3 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
    Topic: __consumer_offsets   Partition: 0    Leader: 1   Replicas: 1,7,2 Isr: 1,2,7
    Topic: __consumer_offsets   Partition: 1    Leader: 2   Replicas: 2,1,5 Isr: 2,1,5
    Topic: __consumer_offsets   Partition: 2    Leader: 5   Replicas: 5,2,6 Isr: 5,6,2
    Topic: __consumer_offsets   Partition: 3    Leader: 6   Replicas: 6,5,7 Isr: 5,6,7
    Topic: __consumer_offsets   Partition: 4    Leader: 7   Replicas: 7,6,1 Isr: 1,6,7
    Topic: __consumer_offsets   Partition: 5    Leader: 1   Replicas: 1,2,5 Isr: 1,5,2
    Topic: __consumer_offsets   Partition: 6    Leader: 2   Replicas: 2,5,6 Isr: 2,6,5
    Topic: __consumer_offsets   Partition: 7    Leader: 5   Replicas: 5,6,7 Isr: 5,6,7

之前晓雷迁移broker0的时候没有迁移，昨天我迁移下掉broker3的时候也还有一个副本，但是今天下broker4的时候，Replicas位于0,3,4的就全部都失效了

这边启动3和4，然后迁移__consumer_offsets这个topic

用于生成迁移计划的json

{"topics":
[{"topic": "__consumer_offsets"}],
"version":1
}

进行执行

$ ./bin/kafka-reassign-partitions.sh --zookeeper 192.168.121.4:2181/kafka --topics-to-move-json-file topics-to-move.json --broker-list "7" --generate
Partitions reassignment failed due to replication factor: 3 larger than available brokers: 1
org.apache.kafka.common.errors.InvalidReplicationFactorException: replication factor: 3 larger than available brokers: 1

这个报错意思是，Replicas有3个，至少需要提供3个broker

$ ./bin/kafka-reassign-partitions.sh --zookeeper 192.168.121.4:2181/kafka --topics-to-move-json-file topics-to-move.json --broker-list "1,2,5,6,7" --generate