Tag Archives: Big data

[Solved] Hadoop Start NameNode Error: ERROR: Cannot set priority of namenode process 2639

Project scene:

HadoopStart NameNodeReport Error: ERROR: Could not set priority of namenode process
=

Description of the question:

Hadoop Start NameNode errror: ERROR: Cannot set priority of namenode process 2639

[ atguigu@localhost333 ↓ logs]$ tail -100 [UNK]hadoop-localhost -namenode-hadoop333.log

😉

@Override public void run() { bytes = mmInStream.read(buffer); mHandler.obtainMessage(READ_DATA, bytes, -1, buffer).sendToTarget(); }

Reasons analysis:

Date of First Visit: hadoop-atguigu-namenode-hadoop103.log

order as follows:
[localhost @hadoop333 logs]$ tail -100 ˚hadoop-atguigu-namenode-hadoop103.log log log log

2021-09-17 16:41:25,656 INFO org.apache.localhost.util.ExitUtil: 
Exiting with status 1: org.apache.localhost.hdfs.server.common.InconsistentFSStateException: 
Directory /opt/ha/hadoop-3.1.3/data/name is in an inconsistent state: 
storage directory does not exist or is not accessible.
2021-09-17 16:41:25,663 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: 
SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop333/192.168.1.103

Solution:

Try to restart centos7, and then re delete the data and logs in Hadoop and the files in TMP in the system root directory to re initialize Hadoop. The problem is solved.

rm -rf /opt/ha/hadoop-3.1.3/data /opt/ha/hadoop-3.1.3/logs

sudo rm -rf /tmp/*

hdfs namenode -format

After switching the tidb database, an error could not commit JDBC transaction appears from time to time

1. Exception description

org.springframework.transaction.TransactionSystemException: Could not commit JDBC transaction; nested exception is java.sql.SQLException: Write conflict, txnStartTS=426985930345676879,

After switching the tidb database, the above error reports occur from time to time;

2. Problem analysis

It is found that there is a write conflict by searching the online data according to the error content. The optimistic lock is used by default under version 3.0 of tidb database. This problem will occur in the case of multiple concurrent modifications;

3. Solution

(1) Add & amp; database connection; sessionVariables=tidb_ txn_ Change mode = pessimistic to pessimistic lock

(2) Break big things apart

4. Subsequent optimization selection

Mysql database error (communications link failure)

Reason: mysql5 defaults the wait_timeout of its connection to 8 hours
modify the configuration file. The steps are as follows:

【1】 Log in to MySQL and CMD
MySQL – H host address – U user name – P user password (for example, MySQL – hlocalhost – uroot – p123456)

【2】 View wait_ Timeout, CMD command:
show global variables like ‘wait_ timeout’;

[3] if wait_ Timeout is too small to be modified. CMD command:
set global wait_ timeout=604800;
set global interactive_ timeout=604800;
after modification, you can test.

How to Solve null/hadoopbinary/wintils.exe Error

Null/Hadoop binary/wintils.exe reports an error
because Windows lacks Hadoop common package, you need to download and unzip hadoop-common-2.2.0-bin-master package

Set the environment variable in the environment variable,
1. Create Hadoop in the user variable_Home variable name, variable value is the location of common package
2. Add% Hadoop in the system variable path_HOME%\bin;
click OK to save and restart the computer

How to Solve Spark Writes Hudi Error

Error message

java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$POSIX.stat(Ljava/lang/String;)Lorg/apache/hadoop/io/nativeio/NativeIO$POSIX$Stat;
        at org.apache.hadoop.io.nativeio.NativeIO$POSIX.stat(Native Method)
        at org.apache.hadoop.io.nativeio.NativeIO$POSIX.getStat(NativeIO.java:456)
        at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfoByNativeIO(RawLocalFileSystem.java:821)
        at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:735)
        at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:703)
        at org.apache.hadoop.fs.LocatedFileStatus.<init>(LocatedFileStatus.java:52)
        at org.apache.hadoop.fs.FileSystem$4.next(FileSystem.java:2091)
        at org.apache.hadoop.fs.FileSystem$4.next(FileSystem.java:2071)
        at org.apache.hadoop.fs.FileSystem$5.hasNext(FileSystem.java:2190)
        at org.apache.hudi.table.marker.DirectWriteMarkers.lambda$createdAndMergedDataPaths$69cdea3b$1(DirectWriteMarkers.java:138)
        at org.apache.hudi.client.common.HoodieSparkEngineContext.lambda$flatMap$7d470b86$1(HoodieSparkEngineContext.java:78)
        at org.apache.spark.api.java.JavaRDDLike.$anonfun$flatMap$1(JavaRDDLike.scala:125)
        at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
        at scala.collection.Iterator.foreach(Iterator.scala:941)
        at scala.collection.Iterator.foreach$(Iterator.scala:941)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
        at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
        at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
        at scala.collection.TraversableOnce.to(TraversableOnce.scala:315)
        at scala.collection.TraversableOnce.to$(TraversableOnce.scala:313)
        at scala.collection.AbstractIterator.to(Iterator.scala:1429)
        at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:307)
        at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:307)
        at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1429)
        at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:294)
        at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:288)
        at scala.collection.AbstractIterator.toArray(Iterator.scala:1429)
        at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1030)
        at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2236)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:131)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Error reporting reason

Because the Hadoop version loaded at runtime is version 3.2, while the Hadoop version in the cluster is version 3.0.0-cdh6.3.2
this code is different for two different versions

Solution:

Download the spark source code, compile spark by yourself, and change the built-in Hadoop to version 3.0.0-cdh6.3.2

ERROR server.datanode.DataNode: BlockSender.sendChunks() exception [How to Solve]

View log error messages:

2021-09-13 14:56:08,737 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: BlockSender.sendChunks() exception: 
java.io.IOException: Connection reset by the other party
	at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
	at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
	at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
	at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
	at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223)
	at org.apache.hadoop.hdfs.server.datanode.FileIoProvider.transferToSocketFully(FileIoProvider.java:280)
	at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:619)
	at org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:803)
	at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:750)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:606)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:152)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:104)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:289)
	at java.lang.Thread.run(Thread.java:748

Add configuration of yarn-site XML

<!-- The minimum memory resources (in MB) requested per container. -->
<property>
	<name>yarn.scheduler.minimum-allocation-mb</name>
	<value>512</value>
</property>
<! -- The maximum memory resource (in MB) requested per container. -->
<property>
	<name>yarn.scheduler.maximum-allocation-mb</name>
	<value>512</value>
</property>
<property>
	<name>yarn.scheduler.minimum-allocation-mb</name>
	<value>512</value>
</property>
<! -- The maximum memory resource (in MB) requested per container. -->
<property>
	<name>yarn.scheduler.maximum-allocation-mb</name>
	<value>2048</value>
</property>
<! -- The ratio between container virtual memory and physical memory-->
<property>
	<name>yarn.nodemanager.vmem-pmem-ratio</name>
	<value>4</value>
</property>

These configurations are all about memory resources

Previously, running with small files can be successful   If you change to a large file, you will report the above error   It should have something to do with memory

Pro test effective

ERROR: JAVA_HOME is not set and could not be found.

ERROR: JAVA_ HOME is not set and could not be found.

Background

    attempting to operate on HDFS namenode as root, but there is no HDFS_ NAMENODE_ USER defined. Aborting

Solution

    in $Hadoop_ Add the location of the JDK in the home/etc/Hadoop directory. Note that it is distributed to other machines in the cluster

    export JAVA_HOME=/opt/module/jdk1.8

The uibot database connection is unrecognized error: The object definition error

When defining the database, the connection object is too long, so the object is separated from the definition first, and then database.createdb();

In order to simplify and enhance readability when separating objects, some typesetting and possible errors will be made, as follows:

//The following two cases are possible
dim a={"a":"123","b":"fg"}
dim a={
    "a":"123",
    "b":"fg"}
//One of the following cases is not allowed
dim a={
    "a":"123",
    "b":"fg"
}
//Separating the closing curly brace from the last element does not work.

HDFS and local file transfer and error reporting

  Exception during HDFS and local file transfer in Maven project:
  org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Ljava/lang/String; JJJI)Ljava/io/FileDescriptor;
  org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(II[BI[BIILjava/lang/String; JZ)V

  Solution: add hadoop.dll and winutils files in the windows/system32 directory of the local C disk (in the bin directory of Hadoop) and
in the system environment variable   Configure Hadoop_ Home path and% Hadoop_ HOME%\bin

[Solved] Error running query: MetaException(message:Got exception: java.net.ConnectException Call From XXXX

Problem screenshot

Problem description

Error: Error running query: MetaException(message:Got exception: 
java.net.ConnectException Call From hadoop102/192.168.121.102 to hadoop102:9000 
failed on connection exception: 
java.net.ConnectException: Connection denied;
For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused) (state=,code=0)

Cause:

Hadoop cluster is not started

Solution:

Start Hadoop cluster: detailed steps

Result screenshot

HBase hangs up immediately after startup. The port reports an error of 500 and hmaster aborted

[error 1]:

java.lang.RuntimeException: HMaster Aborted
	at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:261)
	at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:149)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
	at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2971)
2021-08-26 12:25:35,269 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x37b80a4f6560008

[attempt 1]: delete the HBase node under ZK
but not solve my problem
[attempt 2]: reinstall HBase
but not solve my problem
[attempt 3]: turn off HDFS security mode

hadoop dfsadmin -safemode leave

Still can’t solve my problem
[try 4]: check zookeeper. You can punch in spark normally or add new nodes. No problem.

Turn up and report an error
[error 2]:

master.HMaster: Failed to become active master
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
	at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:108)
	at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2044)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1409)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:2961)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1160)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:880)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:507)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1034)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1003)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:931)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1926)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2854)

Here’s the point: an error is reported that my master failed to become an hmaster. The reason is: operation category read is not supported in state standby. That is, the read operation fails because it is in the standby state. Check the NN1 state at this time

#  hdfs haadmin -getServiceState nn1

Sure enough, standby

[solution 1]:
manual activation

hdfs haadmin -transitionToActive --forcemanual nn1

Kill all HBase processes, restart, JPS view, access port

Bravo!!!

It’s been changed for a few hours. It’s finally good. The reason is that I forgot the correct startup sequence
zookeeper —— & gt; hadoop——-> HBase
every time I start Hadoop first, I waste a long time. I hope I can help you.

CDH opens Kerberos and reports an error: ticket expired

I did it with reference to the open Kerberos wizard in the official cloudera documentation. Address: https://docs.cloudera.com/cdp-private-cloud-base/7.1.5/security-kerberos-authentication/topics/cm-security-kerberos-enabling-step4-kerberos-wizard.html

During the startup process, the last step of starting the cluster starts to report errors. The services reporting errors include Kafka, HBase, jobhistory, etc. the relevant logs are as follows:

Kafka, HBase and other logs:

Jobhistory log:

It can be seen from the log that the ticket is expired, so the relevant configurations of krb5.conf and kdc.conf files are checked. The configuration seems to be OK, as follows:

So, I took a CM automatically generated KeyTab (/ var/run/cloudera SCM agent/process/15 HDFS datacode/HDFS. KeyTab) on the server and tried Kinit to see if it really expired, but the results are as follows:

This error seems to indicate that the KeyTab format of this version is not supported. Therefore, it is considered that the Kerberos version is incompatible. Just before, I saw that some people said that the Kerberos version is sometimes incompatible. Link: https://community.cloudera.com/t5/Support-Questions/Kerberos-ticket-expired-kinit-keytab-successfully-java/td-p/80522

The respondents here give such tips:

Therefore, after the version was changed to 1.15.50, the problem was solved. This log hint is still very lame. Kerberos style has always been like this.