At present, there are three Hadoop clusters with 01, 02 and 03 nodes, among which 01 is the master node and 02 and 03 are the slave nodes. After dynamically adding 04 slave nodes, the number of nodes displayed in Hadoop web interface does not increase. Further observation shows that the contents of slave nodes in the node list are sometimes 02 and 04 nodes, and sometimes 02 and 03 nodes, which is very strange.
Analyze the reasons
Find the current/version file in the dataDir directory, and the datanodeuuid values in the 03 and 04 servers are exactly the same. It can be inferred that the master node treats 03 and 04 as the same server. The reason is that the 03 server has been started before, and the datanodeuuid has been recorded in the generated version. When the virtual machine is cloned, all the information is the same as the 03 server.
Considering that if it is not a clone but a direct installation, the dataDir and TMPDIR directories of the 04 server are blank, so stop the datanode and nodemanager services of the 04 server, delete all the files in the dataDir and TMPDIR directories, restart the datanode and nodemanager services, and observe again that the contents of the Hadoop cluster display three datanodes 02, 03 and 04, which are similar to the The expectation is consistent and the problem is solved.
matters needing attention
When adding new nodes by copying or cloning virtual machines, the following operations need to be done:
1. Modify the host name and IP address, restart the new node, and copy the/etc/hosts file to other nodes in the cluster (if using the local DNS server, this step can be omitted, just add the modified node to the domain name resolution record in the DNS server);
2. The new node needs to re run SSH keygen – t RSA to generate the public key and add it to authorized_ And copy it to other nodes in the cluster.
1. Hadoop datanode starts normally, but there is no node in live nodes
2. Hadoop 2.7 dynamically adds and deletes nodes
in this paper
- Linux Mint installs Hadoop environment
- dfs.namenode.name . dir and dfs.datanode.data .dir dfs.name.dir And dfs.data.dir What do you mean
- Hadoop datanode using JPS to view the solution that can’t be started
- Hadoop — HDFS data writing process
- Introduction of Hadoop HDFS and the use of basic client commands
- Common problems of Hadoop startup error reporting
- What to do if you repeatedly format a cluster
- hdfs 192.168.2.19:9000 failed on connection exception: java.net.ConnectException:Connection refused
- Ambari shield big data components
- Centos7 view and close firewall
- (2) unrecoverable error: corrupted cluster config file
- Starting zookeeper… Already running as process 1949
- Kafka prompts no brokers found when trying to rebalance
- Error: attempting to operate on HDFS namenode as root
- Failed to connect to ourself
- MySQL – ERROR 1146 (42S02): Table ‘mysql.user’ doesn’t exist
- Ambari failed to monitor yarn ResourceManager, restart invalid
- Why namenode can’t be started and its solution
- Error syncing pod, skipping: failed to “StartContainer” for “POD” with Image
- Ranger yarn plug-in installation