Article catalog

1. Problem 2. Solution

1. Problems

When running Mr task in win10 environment, the error is as follows:

D:\Java\jdk1.8.0_201\bin\java.exe "-javaagent:D:\Program Files\JetBrains\IntelliJ IDEA 2020.1\lib\idea_rt.jar=64181:D:\Program Files\JetBrains\IntelliJ IDEA 2020.1\bin" -Dfile.encoding=UTF-8 -classpath D:\Java\jdk1.8.0_201\jre\lib\charsets.jar;D:\Java\jdk1.8.0_201\jre\lib\deploy.jar;D:\Java\jdk1.8.0_201\jre\lib\ext\access-bridge-64.jar;D:\Java\jdk1.8.0_201\jre\lib\ext\cldrdata.jar;D:\Java\jdk1.8.0_201\jre\lib\ext\dnsns.jar;D:\Java\jdk1.8.0_201\jre\lib\ext\jaccess.jar;D:\Java\jdk1.8.0_201\jre\lib\ext\jfxrt.jar;D:\Java\jdk1.8.0_201\jre\lib\ext\localedata.jar;D:\Java\jdk1.8.0_201\jre\lib\ext\nashorn.jar;D:\Java\jdk1.8.0_201\jre\lib\ext\sunec.jar;D:\Java\jdk1.8.0_201\jre\lib\ext\sunjce_provider.jar;D:\Java\jdk1.8.0_201\jre\lib\ext\sunmscapi.jar;D:\Java\jdk1.8.0_201\jre\lib\ext\sunpkcs11.jar;D:\Java\jdk1.8.0_201\jre\lib\ext\zipfs.jar;D:\Java\jdk1.8.0_201\jre\lib\javaws.jar;D:\Java\jdk1.8.0_201\jre\lib\jce.jar;D:\Java\jdk1.8.0_201\jre\lib\jfr.jar;D:\Java\jdk1.8.0_201\jre\lib\jfxswt.jar;D:\Java\jdk1.8.0_201\jre\lib\jsse.jar;D:\Java\jdk1.8.0_201\jre\lib\management-agent.jar;D:\Java\jdk1.8.0_201\jre\lib\plugin.jar;D:\Java\jdk1.8.0_201\jre\lib\resources.jar;D:\Java\jdk1.8.0_201\jre\lib\rt.jar;D:\code_java\BigDataLearningDemo\MapReduceDemo\target\classes;D:\MavenRepos\junit\junit\4.13\junit-4.13.jar;D:\MavenRepos\org\hamcrest\hamcrest-core\1.3\hamcrest-core-1.3.jar;D:\MavenRepos\org\slf4j\slf4j-log4j12\1.7.30\slf4j-log4j12-1.7.30.jar;D:\MavenRepos\org\slf4j\slf4j-api\1.7.30\slf4j-api-1.7.30.jar;D:\MavenRepos\log4j\log4j\1.2.17\log4j-1.2.17.jar;D:\MavenRepos\org\apache\hadoop\hadoop-client\3.1.1\hadoop-client-3.1.1.jar;D:\MavenRepos\org\apache\hadoop\hadoop-common\3.1.1\hadoop-common-3.1.1.jar;D:\MavenRepos\com\google\guava\guava\11.0.2\guava-11.0.2.jar;D:\MavenRepos\commons-cli\commons-cli\1.2\commons-cli-1.2.jar;D:\MavenRepos\org\apache\commons\commons-math3\3.1.1\commons-math3-3.1.1.jar;D:\MavenRepos\org\apache\httpcomponents\httpclient\4.5.2\httpclient-4.5.2.jar;D:\MavenRepos\org\apache\httpcomponents\httpcore\4.4.4\httpcore-4.4.4.jar;D:\MavenRepos\commons-codec\commons-codec\1.11\commons-codec-1.11.jar;D:\MavenRepos\commons-io\commons-io\2.5\commons-io-2.5.jar;D:\MavenRepos\commons-net\commons-net\3.6\commons-net-3.6.jar;D:\MavenRepos\commons-collections\commons-collections\3.2.2\commons-collections-3.2.2.jar;D:\MavenRepos\org\eclipse\jetty\jetty-servlet\9.3.19.v20170502\jetty-servlet-9.3.19.v20170502.jar;D:\MavenRepos\org\eclipse\jetty\jetty-security\9.3.19.v20170502\jetty-security-9.3.19.v20170502.jar;D:\MavenRepos\org\eclipse\jetty\jetty-webapp\9.3.19.v20170502\jetty-webapp-9.3.19.v20170502.jar;D:\MavenRepos\org\eclipse\jetty\jetty-xml\9.3.19.v20170502\jetty-xml-9.3.19.v20170502.jar;D:\MavenRepos\javax\servlet\jsp\jsp-api\2.1\jsp-api-2.1.jar;D:\MavenRepos\com\sun\jersey\jersey-servlet\1.19\jersey-servlet-1.19.jar;D:\MavenRepos\commons-logging\commons-logging\1.1.3\commons-logging-1.1.3.jar;D:\MavenRepos\commons-lang\commons-lang\2.6\commons-lang-2.6.jar;D:\MavenRepos\commons-beanutils\commons-beanutils\1.9.3\commons-beanutils-1.9.3.jar;D:\MavenRepos\org\apache\commons\commons-configuration2\2.1.1\commons-configuration2-2.1.1.jar;D:\MavenRepos\org\apache\commons\commons-lang3\3.4\commons-lang3-3.4.jar;D:\MavenRepos\org\apache\avro\avro\1.7.7\avro-1.7.7.jar;D:\MavenRepos\org\codehaus\jackson\jackson-core-asl\1.9.13\jackson-core-asl-1.9.13.jar;D:\MavenRepos\org\codehaus\jackson\jackson-mapper-asl\1.9.13\jackson-mapper-asl-1.9.13.jar;D:\MavenRepos\com\thoughtworks\paranamer\paranamer\2.3\paranamer-2.3.jar;D:\MavenRepos\org\xerial\snappy\snappy-java\1.0.5\snappy-java-1.0.5.jar;D:\MavenRepos\com\google\re2j\re2j\1.1\re2j-1.1.jar;D:\MavenRepos\com\google\protobuf\protobuf-java\2.5.0\protobuf-java-2.5.0.jar;D:\MavenRepos\com\google\code\gson\gson\2.2.4\gson-2.2.4.jar;D:\MavenRepos\org\apache\hadoop\hadoop-auth\3.1.1\hadoop-auth-3.1.1.jar;D:\MavenRepos\com\nimbusds\nimbus-jose-jwt\4.41.1\nimbus-jose-jwt-4.41.1.jar;D:\MavenRepos\com\github\stephenc\jcip\jcip-annotations\1.0-1\jcip-annotations-1.0-1.jar;D:\MavenRepos\net\minidev\json-smart\2.3\json-smart-2.3.jar;D:\MavenRepos\net\minidev\accessors-smart\1.2\accessors-smart-1.2.jar;D:\MavenRepos\org\ow2\asm\asm\5.0.4\asm-5.0.4.jar;D:\MavenRepos\org\apache\curator\curator-framework\2.12.0\curator-framework-2.12.0.jar;D:\MavenRepos\org\apache\curator\curator-client\2.12.0\curator-client-2.12.0.jar;D:\MavenRepos\org\apache\curator\curator-recipes\2.12.0\curator-recipes-2.12.0.jar;D:\MavenRepos\com\google\code\findbugs\jsr305\3.0.0\jsr305-3.0.0.jar;D:\MavenRepos\org\apache\htrace\htrace-core4\4.1.0-incubating\htrace-core4-4.1.0-incubating.jar;D:\MavenRepos\org\apache\commons\commons-compress\1.4.1\commons-compress-1.4.1.jar;D:\MavenRepos\org\tukaani\xz\1.0\xz-1.0.jar;D:\MavenRepos\org\apache\kerby\kerb-simplekdc\1.0.1\kerb-simplekdc-1.0.1.jar;D:\MavenRepos\org\apache\kerby\kerb-client\1.0.1\kerb-client-1.0.1.jar;D:\MavenRepos\org\apache\kerby\kerby-config\1.0.1\kerby-config-1.0.1.jar;D:\MavenRepos\org\apache\kerby\kerb-core\1.0.1\kerb-core-1.0.1.jar;D:\MavenRepos\org\apache\kerby\kerby-pkix\1.0.1\kerby-pkix-1.0.1.jar;D:\MavenRepos\org\apache\kerby\kerby-asn1\1.0.1\kerby-asn1-1.0.1.jar;D:\MavenRepos\org\apache\kerby\kerby-util\1.0.1\kerby-util-1.0.1.jar;D:\MavenRepos\org\apache\kerby\kerb-common\1.0.1\kerb-common-1.0.1.jar;D:\MavenRepos\org\apache\kerby\kerb-crypto\1.0.1\kerb-crypto-1.0.1.jar;D:\MavenRepos\org\apache\kerby\kerb-util\1.0.1\kerb-util-1.0.1.jar;D:\MavenRepos\org\apache\kerby\token-provider\1.0.1\token-provider-1.0.1.jar;D:\MavenRepos\org\apache\kerby\kerb-admin\1.0.1\kerb-admin-1.0.1.jar;D:\MavenRepos\org\apache\kerby\kerb-server\1.0.1\kerb-server-1.0.1.jar;D:\MavenRepos\org\apache\kerby\kerb-identity\1.0.1\kerb-identity-1.0.1.jar;D:\MavenRepos\org\apache\kerby\kerby-xdr\1.0.1\kerby-xdr-1.0.1.jar;D:\MavenRepos\com\fasterxml\jackson\core\jackson-databind\2.7.8\jackson-databind-2.7.8.jar;D:\MavenRepos\com\fasterxml\jackson\core\jackson-core\2.7.8\jackson-core-2.7.8.jar;D:\MavenRepos\org\codehaus\woodstox\stax2-api\3.1.4\stax2-api-3.1.4.jar;D:\MavenRepos\com\fasterxml\woodstox\woodstox-core\5.0.3\woodstox-core-5.0.3.jar;D:\MavenRepos\org\apache\hadoop\hadoop-hdfs-client\3.1.1\hadoop-hdfs-client-3.1.1.jar;D:\MavenRepos\com\squareup\okhttp\okhttp\2.7.5\okhttp-2.7.5.jar;D:\MavenRepos\com\squareup\okio\okio\1.6.0\okio-1.6.0.jar;D:\MavenRepos\com\fasterxml\jackson\core\jackson-annotations\2.7.8\jackson-annotations-2.7.8.jar;D:\MavenRepos\org\apache\hadoop\hadoop-yarn-api\3.1.1\hadoop-yarn-api-3.1.1.jar;D:\MavenRepos\javax\xml\bind\jaxb-api\2.2.11\jaxb-api-2.2.11.jar;D:\MavenRepos\org\apache\hadoop\hadoop-yarn-client\3.1.1\hadoop-yarn-client-3.1.1.jar;D:\MavenRepos\org\apache\hadoop\hadoop-mapreduce-client-core\3.1.1\hadoop-mapreduce-client-core-3.1.1.jar;D:\MavenRepos\org\apache\hadoop\hadoop-yarn-common\3.1.1\hadoop-yarn-common-3.1.1.jar;D:\MavenRepos\javax\servlet\javax.servlet-api\3.1.0\javax.servlet-api-3.1.0.jar;D:\MavenRepos\org\eclipse\jetty\jetty-util\9.3.19.v20170502\jetty-util-9.3.19.v20170502.jar;D:\MavenRepos\com\sun\jersey\jersey-core\1.19\jersey-core-1.19.jar;D:\MavenRepos\javax\ws\rs\jsr311-api\1.1.1\jsr311-api-1.1.1.jar;D:\MavenRepos\com\sun\jersey\jersey-client\1.19\jersey-client-1.19.jar;D:\MavenRepos\com\fasterxml\jackson\module\jackson-module-jaxb-annotations\2.7.8\jackson-module-jaxb-annotations-2.7.8.jar;D:\MavenRepos\com\fasterxml\jackson\jaxrs\jackson-jaxrs-json-provider\2.7.8\jackson-jaxrs-json-provider-2.7.8.jar;D:\MavenRepos\com\fasterxml\jackson\jaxrs\jackson-jaxrs-base\2.7.8\jackson-jaxrs-base-2.7.8.jar;D:\MavenRepos\org\apache\hadoop\hadoop-mapreduce-client-jobclient\3.1.1\hadoop-mapreduce-client-jobclient-3.1.1.jar;D:\MavenRepos\org\apache\hadoop\hadoop-mapreduce-client-common\3.1.1\hadoop-mapreduce-client-common-3.1.1.jar;D:\MavenRepos\org\apache\hadoop\hadoop-annotations\3.1.1\hadoop-annotations-3.1.1.jar com.learning.nokerberos.mapreduce2.sgg9reduceJoin.TableDriver
2021-08-20 08:53:01,390 WARN [org.apache.hadoop.util.NativeCodeLoader] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021-08-20 08:53:01,852 INFO [org.apache.commons.beanutils.FluentPropertyBeanIntrospector] - Error when creating PropertyDescriptor for public final void org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)! Ignoring this property.
2021-08-20 08:53:01,864 WARN [org.apache.hadoop.metrics2.impl.MetricsConfig] - Cannot locate configuration: tried hadoop-metrics2-jobtracker.properties,hadoop-metrics2.properties
2021-08-20 08:53:01,912 INFO [org.apache.hadoop.metrics2.impl.MetricsSystemImpl] - Scheduled Metric snapshot period at 10 second(s).
2021-08-20 08:53:01,912 INFO [org.apache.hadoop.metrics2.impl.MetricsSystemImpl] - JobTracker metrics system started
2021-08-20 08:53:02,382 WARN [org.apache.hadoop.mapreduce.JobResourceUploader] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2021-08-20 08:53:02,438 WARN [org.apache.hadoop.mapreduce.JobResourceUploader] - No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2021-08-20 08:53:02,457 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Cleaning up the staging area file:/tmp/hadoop/mapred/staging/264741960596537/.staging/job_local1960596537_0001
Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
	at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:640)
	at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:1223)
	at org.apache.hadoop.fs.FileUtil.list(FileUtil.java:1428)
	at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:468)
	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1919)
	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1961)
	at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:678)
	at org.apache.hadoop.fs.Globber.listStatus(Globber.java:77)
	at org.apache.hadoop.fs.Globber.doGlob(Globber.java:235)
	at org.apache.hadoop.fs.Globber.glob(Globber.java:149)
	at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2085)
	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:303)
	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:274)
	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:396)
	at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:313)
	at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:330)
	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:203)
	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)
	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)
	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1588)
	at com.learning.nokerberos.mapreduce2.sgg9reduceJoin.TableDriver.main(TableDriver.java:37)

Process finished with exit code 1

2. Solution

Copy the Hadoop. DLL file in the D: \ hadoop-3.1.1 \ bin directory to the C: \ windows \ system32 directory, and re run the MR task successfully