Problem phenomenon
At present, there are three Hadoop clusters with 01, 02 and 03 nodes, among which 01 is the master node and 02 and 03 are the slave nodes. After dynamically adding 04 slave nodes, the number of nodes displayed in Hadoop web interface does not increase. Further observation shows that the contents of slave nodes in the node list are sometimes 02 and 04 nodes, and sometimes 02 and 03 nodes, which is very strange.
Analyze the reasons
Find the current/version file in the dataDir directory, and the datanodeuuid values in the 03 and 04 servers are exactly the same. It can be inferred that the master node treats 03 and 04 as the same server. The reason is that the 03 server has been started before, and the datanodeuuid has been recorded in the generated version. When the virtual machine is cloned, all the information is the same as the 03 server.
Solution
Considering that if it is not a clone but a direct installation, the dataDir and TMPDIR directories of the 04 server are blank, so stop the datanode and nodemanager services of the 04 server, delete all the files in the dataDir and TMPDIR directories, restart the datanode and nodemanager services, and observe again that the contents of the Hadoop cluster display three datanodes 02, 03 and 04, which are similar to the The expectation is consistent and the problem is solved.
matters needing attention
When adding new nodes by copying or cloning virtual machines, the following operations need to be done:
1. Modify the host name and IP address, restart the new node, and copy the/etc/hosts file to other nodes in the cluster (if using the local DNS server, this step can be omitted, just add the modified node to the domain name resolution record in the DNS server);
2. The new node needs to re run SSH keygen – t RSA to generate the public key and add it to authorized_ And copy it to other nodes in the cluster.
reference material:
1. Hadoop datanode starts normally, but there is no node in live nodes
https://blog.csdn.net/wk51920/article/details/51729460
2. Hadoop 2.7 dynamically adds and deletes nodes
in this paper
https://blog.csdn.net/Mark_ LQ/article/details/53393081