Tag Archives: Kafka

Kafka prompts no brokers found when trying to rebalance

Kafka prompts when executing the following command:

bin/kafka-console-consumer.sh --zookeeper localhost102:2181 --topic test

WARN [console-consumer-87796_ localhost002-1592779486563-9b43649b], no brokers found when trying to rebalance.( kafka.consumer.ZookeeperConsumerConnector )

The reason is that the Kafka process is not started or there is no Kafka cluster information on zookeeper

[root@localhost002 ~]# jps
3667 DataNode
3365 ResourceManager
21446 QuorumPeerMain
23386 Jps
3230 NodeManager

Solve the problem after starting Kafka

Centos7 installing Kafka

Download kafka
website: http://kafka.apache.org/
enter the url: http://kafka.apache.org/downloads
System environment
1, the operating system: 64 CentOS7
2, JDK version: 1.8.0 comes with _271 version
3, they are: they are – 3.4.6
4, three server deployment success: 192.168.192.150; 192.168.192.152; 192.168.192.155;
Pre-setting:

[root@localhost /]# timedatectl set-timezone Asia/Shanghai
root@localhost /]# timedatectl
[root@localhost /]# timedatectl
[root@localhost /]# timedatectl
[root@localhost /]# timedatectl

[root@master /]# yum-y install NTP
p1
[>daoplaceholder1 /]# ntpdate ntp1.aliyun.com
hostname ectl set-hostname master
stname ectl set-hostname node1
h>ame ectl set-hostname node2
hostname ectl set-hostname node2
hostname ectl set-hostname node2
hostname ectl set-hostname node2

192.168.192.150 master
2.168.192.152 node1
1>68.192.155 node2
192.168.192.155

127.0.0.1 localhost
: 1 localhost
1>68.93.150 master
192.168>152 node1
192.168.93.15>de2
192.168.93.155 node2
192.168.93.155 node2

192.168.93.155 nod> br> If you change the hosts in Windows 10, you can copy this file to the desktop and add the above contents. Then replace the hosts file under C:\Windows\System32\ Drivers \etc with
.

props. Put (consumerConfig. Bootstrap_servers_config,”node1:9092,node2:9092″);
tar-zxvf /usr/local/soft/kafka_2.11-2.2.0.tgz -c /usr/local/ kafka
tar-zxvf /usr/local/soft/kafka_2.11-2.2.0.tgz -c /usr/local/
2, configuration, kafka
[root @ localhost] # vi/usr/local/kafka_2. 11-2.2.0/config/server properties
Kafka’s configuration information is configured in server.properties
Find the following two lines of code and comment them separately
#broker.id=0
#zookeeper.connect=localhost:2181
Modify the directory where the logs are stored
The log. The dirs =/kafka_2. 11-2.2.0/kafka – logs
Add the following three configurations at the bottom of the file:
Broker. Id = 1
zookeeper. Connect = 192.168.192.150:2181192168 192.152:2181192168 192.155:2181

Listeners = PlainText :// Master :9092 Listeners :// Master :9092

[root@localhost local]# CD kafka2.11-2.2.0 /
root@localhost kafka2.11-2.2.0]# mkdir kafka-logs
[root@localhost local]# CD kafka2.11-2.2.0 /
[root@localhost kafka2.11-2.2.0]# mkdir kafka-logs

Note: If it is a stand-alone version, the default is fine, we do not need to change anything. Now we are configuring the cluster, so we need to configure some parameters
1), Broker. id: Each machine cannot be the same
2), ZooKeeper. Connect: Since I have 3 ZooKeeper servers, I set ZooKeeper. Connect to 3 servers and must add all of them
3), Listeners: Listeners must be set when configuring the cluster, otherwise the leader error will not be found for future operations
WARN [Producer clientId = console – Producer] Error while fetching the metadata with the correlation id 40: {test = LEADER_NOT_AVAILABLE} workClient (org.apache.kafka.clients.Net)
4) Notice that the two servers, ZooKeeper. Connect, are configured the same as here, but the Broker. ID and Listeners cannot be configured the same
5 copies, kafka to two other servers
[root @ localhost config] # SCP – r/usr/local/kafka_2. 11-2.2.0 [email protected]:/usr/local/
[root @ localhost config] # SCP – r/usr/local/kafka_2. 11-2.2.0 [email protected]:/usr/local /
You will be asked to enter the password of the target machine. Just follow the instructions and modify the Broker. id and Listeners of the two servers as follows:
6, 192.168.192.152
[root @ localhost] # vi/usr/local/kafka_2. 11-2.2.0/config/server properties
Then a change on the broker. Id = 2 and listeners = PLAINTEXT:// 192.168.192.152:9092, the zookeeper. Connect don’t need to change are the same
Broker.
id = 2 zookeeper. Connect = 192.168.192.150:2181192168 192.152:2181192168 192.155:2181
listeners = PLAINTEXT:// node1:9092
7, 192.168.192.155
[root @ localhost] # vi/usr/local/kafka_2. 11-2.2.0/config/server properties
Then a change on the broker. Id = 3 and listeners = PLAINTEXT:// 192.168.192.155:9092, the zookeeper. Connect don’t need to change are the same
Broker. Id = 3
zookeeper. Connect = 192.168.192.150:2181192168 192.152:2181192168 192.155:2181
listeners = PLAINTEXT:// 2:9092
If the firewall has been turned off, it can be ignored.
All three machines must be turned on. Kafka communication is conducted through port 9092 by default, which is the Listed Listeners we have provided above
[root@localhost config]# firewall-cmd –zone=public –add-port=9092/tcp –permanent
[root@localhost config]# firewall-cmd –reload

[root@localhost /]# /usr/local/ ZooKeeper -3.4.6/bin/ zkserver.sh start
[root@localhost /]# /usr/local/ ZooKeeper -3.4.6/bin/ zkserver.sh start
[root@localhost /]# /usr/local/ ZooKeeper -3.4.6/bin/ zkserver.sh start
10, start, kafka,
three is to launch the
[root @ localhost /] #/usr/local/kafka_2. 11-2.2.0/bin/kafka – server – start. Sh – daemon/usr/local/kafka_2. 11-2.2.0/config/server properties

[root@localhost /]# JPS
87 JPS
9>kafka
9224 Kafka
9224 Kafka
[root@localhost /]# CD /usr/local/kafka_2.11-2.2.0/
root@localhost kafka_2.11-2.2.0]# bin/ kafka-Topics. Sh –create — ZooKeeper 192.168.192.150:2181 –replication-factor 1 –partitions 1 –topic test
Created topic test.
If successful, it outputs: Created Topic “test”.
13, see the topic
while 192.168.192.150 created on the topic, but the other two machine also can see that the 192.168.192.152 client
[root@localhost /]# CD /usr/local/kafka_2.11-2.2.0/
[root@localhost kafka_2.11-2.2.0]# bin/ kafka-Topics — List — ZooKeeper 192.168.192.152:2181
st
Note: The IP here can be 192.168.192.150, 192.168.192.152, 192.168.192.155. Themes can be seen on any of the three servers


root@localhost kafka_2.11-2.2.0]# bin/kafka-console-producer.sh –broker — list 192.168.192.150:9092 –topic test
>> a
> b
> c
# bin/kafka-console-consumer.sh –bootstrap-server 192.168.192.152:9092 — Topic test –from — Beginning
a

c>

Kafka connection abnormal org.apache.kafka . common.errors.TimeoutException : Failed to update metadata after 60000 ms.

Introduction: there are many problems about hosts on the Internet. After testing, I found out that it is not. After testing by myself, I introduced three solutions to most problems:

1. Firewall port not opened or closed

firewall generally local test will be closed, the line is generally open, it is recommended to add port

add port: 

# --permanent 为永久添加，不指定此参数重启后失效
firewall-cmd --zone=public --add-port=9092/tcp --permanent

delete port: 

firewall-cmd --zone=public --remove-port=80/tcp --permanent

view all ports: 

 firewall-cmd --zone=public --list-ports

view specified port: 

firewall-cmd --zone= public --query-port=9092/tcp

view port status: 

netstat -tunlp

overload: 

firewall-cmd --reload

firewall command: 

#开启防火墙 
service firewalld start
#启用防火墙 
service firewalld enable (永久生效)
#关闭防火墙 
service firewalld stop （重启后失效）
#禁用防火墙 
service firewalld disable (永久生效)
#重启防火墙 
service firewalld restart
#查看状态 
service firewalld status

2. Kafka service entry address not specified

edit the server.properties in the config directory of kafka, and add the address of the service entry to the external service: 

注意：此ip为kafka所在主机的ip地址
listeners=PLAINTEXT://192.168.5.228:9092

3. The kafka version in the project is inconsistent with the kafka version installed on the server

modify maven pom. XML file, specify the corresponding kafka coordinates,
I installed kafka_2.12-2.1.0, the corresponding coordinates are: 

<dependency>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka_2.11</artifactId>
      <version>0.10.0.0</version>
</dependency>