1) Package guide is error prone. Especially text and combinetextinputformat. 2) The first input parameter in mapper must be longwritable or nullwritable, not intwritable.
the error reported is a type conversion exception.
3） java.lang.Exception : java.io.IOException : Legal partition for 13926435656 (4), indicating that partition
is not matched with the number of reducetask, so adjust the number of reducetask. 4) If the number of partitions is not 1, but reductask is 1, do you want to execute the partitioning process. The answer is: no partitioning.
Because in the source code of maptask, the premise of partition execution is to determine whether the number of reducenums is greater than 1. No more than 1 is definitely not implemented.
5) import the jar package compiled in Windows environment into linux environment to run,
Hadoop jar wc.jar com.atguigu.mapreduce . wordcount.WordCountDriver /User/atguigu/
/user/atguigu/output
reports the following error:
exception in thread “main” java.lang.UnsupportedClassVersionError :
com/atguigu/mapreduce/wordcount/WordCountDriver : Unsupported major.minor Version 52.0
the reason is that jdk1.7 is used in Windows environment and JDK1.8 is used in Linux environment.
Solution: unified JDK version.
6) cache pd.txt In the case of small files, the report cannot be found pd.txt File
reason: most of them are path writing errors. There is also to check pd.txt.txt It’s a matter of time. There are also some computers that write relative paths
that cannot be found pd.txt , which can be changed to absolute path. 7) Report type conversion exception.
It’s usually a writing error when setting the map output and final output in the driver function.
If the key output from map is not sorted, an exception of type conversion will be reported.
8) running in cluster wc.jar An error occurred when unable to get the input file.
Reason: the input file of wordcount case cannot be placed in the root directory of HDFS cluster.
9) the following related exceptions appear
exception in thread “main” java.lang.UnsatisfiedLinkError :
org.apache.hadoop . io.nativeio.NativeIO

W

i

n

d

o

w

s

.

a

c

c

e

s

s

0

(

L

j

a

v

a

/

l

a

n

g

/

S

t

r

i

n

g

;

I

)

Z

a

t

o

r

g

.

a

p

a

c

h

e

.

h

a

d

o

o

p

.

i

o

.

n

a

t

i

v

e

i

o

.

N

a

t

i

v

e

I

O

Windows.access0 (Ljava/lang/String;I)Z at org.apache.hadoop . io.nativeio.NativeIO

Windows.access0 (Ljava/lang/String; I) Zatorg.apache.hadoop . io.nativeio.NativeIOWindows .access0(Native Method)
at org.apache.hadoop . io.nativeio.NativeIO $ Windows.access ( NativeIO.java:609 )
at org.apache.hadoop . fs.FileUtil.canRead ( FileUtil.java:977 )
java.io.IOException : Could not locate executable null\bin\ winutils.exe in the Hadoop binaries.
at org.apache.hadoop . util.Shell.getQualifiedBinPath ( Shell.java:356 )
at org.apache.hadoop . util.Shell.getWinUtilsPath ( Shell.java:371 )
at org.apache.hadoop . util.Shell .( Shell.java:364 ）
solution: copy hadoop.dll File to the windows directory C::?Windows?System32. Some students need to modify the Hadoop source code.
Scheme 2: create the following package name and NativeIO.java Copy to the package name
10) when customizing the output format, note that the close method in recordwirter must close the stream resource. Otherwise, the data in the output file is empty.
@Override
public void close(TaskAttemptContext context) throws IOException,
InterruptedException {
if (atguigufos != null) {
atguigufos.close ();
}
if (otherfos != null) {
otherfos.close ();
} }