1. Problem description
The
sqoop list-tables tool works, but there is a problem with the import tool. Strange!!
[hadoop@master ~]$ sqoop import --username test --password test --connect jdbc:mysql://172.1.1.96:3306/test -m 1 --delete-target-dir --table student_exam_score_20170802 --fields-terminated-by '\t' --hive-import --hive-database bond_edu --hive-table student_exam_score_20170802
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/11/08 11:31:13 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
18/11/08 11:31:13 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/11/08 11:31:13 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/11/08 11:31:13 INFO tool.CodeGenTool: Beginning code generation
18/11/08 11:31:14 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student_exam_score_20170802` AS t LIMIT 1
18/11/08 11:31:14 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student_exam_score_20170802` AS t LIMIT 1
18/11/08 11:31:14 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce
Note: /tmp/sqoop-hadoop/compile/98d200728b6a446d170eba746e2c9a4c/student_exam_score_20170802.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
18/11/08 11:31:18 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/98d200728b6a446d170eba746e2c9a4c/student_exam_score_20170802.jar
18/11/08 11:31:19 INFO tool.ImportTool: Destination directory student_exam_score_20170802 is not present, hence not deleting.
18/11/08 11:31:19 WARN manager.MySQLManager: It looks like you are importing from mysql.
18/11/08 11:31:19 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
18/11/08 11:31:19 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
18/11/08 11:31:19 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
18/11/08 11:31:19 INFO mapreduce.ImportJobBase: Beginning import of student_exam_score_20170802
18/11/08 11:31:20 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
18/11/08 11:31:20 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
18/11/08 11:31:20 INFO client.RMProxy: Connecting to ResourceManager at master/10.0.1.118:18040
18/11/08 11:31:24 INFO db.DBInputFormat: Using read commited transaction isolation
18/11/08 11:31:24 INFO mapreduce.JobSubmitter: number of splits:1
18/11/08 11:31:24 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1541640011445_0018
18/11/08 11:31:25 INFO impl.YarnClientImpl: Submitted application application_1541640011445_0018
18/11/08 11:31:25 INFO mapreduce.Job: The url to track the job: http://master:18088/proxy/application_1541640011445_0018/
18/11/08 11:31:25 INFO mapreduce.Job: Running job: job_1541640011445_0018
18/11/08 11:31:37 INFO mapreduce.Job: Job job_1541640011445_0018 running in uber mode : false
18/11/08 11:31:37 INFO mapreduce.Job: map 0% reduce 0%
18/11/08 11:31:45 INFO mapreduce.Job: Task Id : attempt_1541640011445_0018_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
... 9 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:341)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2189)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2222)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2017)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
... 10 more
Caused by: java.net.SocketException: Network is unreachable (connect failed)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:300)
... 26 more
18/11/08 11:31:51 INFO mapreduce.Job: Task Id : attempt_1541640011445_0018_m_000000_1, Status : FAILED
Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
... 9 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:341)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2189)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2222)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2017)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
... 10 more
Caused by: java.net.SocketException: Network is unreachable (connect failed)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:300)
... 26 more
18/11/08 11:31:57 INFO mapreduce.Job: Task Id : attempt_1541640011445_0018_m_000000_2, Status : FAILED
Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
... 9 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:341)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2189)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2222)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2017)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
... 10 more
Caused by: java.net.SocketException: Network is unreachable (connect failed)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:300)
... 26 more
18/11/08 11:32:05 INFO mapreduce.Job: map 100% reduce 0%
18/11/08 11:32:06 INFO mapreduce.Job: Job job_1541640011445_0018 failed with state FAILED due to: Task failed task_1541640011445_0018_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
18/11/08 11:32:06 INFO mapreduce.Job: Counters: 8
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=20616
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=20616
Total vcore-seconds taken by all map tasks=20616
Total megabyte-seconds taken by all map tasks=21110784
18/11/08 11:32:06 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
18/11/08 11:32:06 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 46.2311 seconds (0 bytes/sec)
18/11/08 11:32:06 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
18/11/08 11:32:06 INFO mapreduce.ImportJobBase: Retrieved 0 records.
18/11/08 11:32:06 ERROR tool.ImportTool: Error during import: Import job failed!
[hadoop@master ~]$ sqoop list-tables --username test --password test --connect jdbc:mysql://172.1.1.96:3306/test
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/11/08 11:35:27 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
18/11/08 11:35:27 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/11/08 11:35:27 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
a_jw_coursejoin_detail_f
a_test
bk_fee_common
bk_student_rect
bk_subjectdet
bonus
dept
emp
exam_ticket
gg
gg_tets
id
incre_table
p_bk_real_lession_cost_detail
registration
salgrade
student_exam_score_20170802
student_exam_score_20170803
student_exam_score_20170906
student_exam_term
t
t_card_package
t_dictionary
t_sms_send
t_time
t_tracelog_cgi
t_tracelog_cgi_20180920
t_tracelog_cgi_20180921
t_tracelog_cgi_tmp
tbl_name
test_1
test_2
test_3
test_4
test_5
test_x
treenodes
tt1
tt2
tt3
tt4
user
z
[hadoop@master ~]$ 2. Problem solving
2.1 mysql local (sqoop local)
this query data:
1. Change the url localhost to IP (just configure /etc/hosts file)
sqoop import --username test --password test --connect jdbc:mysql://127.0.0.1:3306/test?autoReconnect=true --delete-target-dir --table t --fields-terminated-by '\t' -m 1 --hive-import --hive-database test --hive-table t
2. Empower the user (ignore if you have permission)
grant all privileges on test to test@'%' IDENTIFIED BY 'test' WITH GRANT OPTION;
flush PRIVILEGES;Locahost import failed:
[hadoop@master ~]$ sqoop import --username test --password test --connect jdbc:mysql://127.0.0.1:3306/test?autoReconnect=true --delete-target-dir --table t --fields-terminated-by '\t' -m 1 --hive-import --hive-database test --hive-table t
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/11/08 11:53:58 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
18/11/08 11:53:58 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/11/08 11:53:58 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/11/08 11:53:58 INFO tool.CodeGenTool: Beginning code generation
18/11/08 11:53:59 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `t` AS t LIMIT 1
18/11/08 11:53:59 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `t` AS t LIMIT 1
18/11/08 11:53:59 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce
Note: /tmp/sqoop-hadoop/compile/910a58c86a0bd49d993fb3c8e9860e5f/t.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
18/11/08 11:54:01 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/910a58c86a0bd49d993fb3c8e9860e5f/t.jar
18/11/08 11:54:03 INFO tool.ImportTool: Destination directory t is not present, hence not deleting.
18/11/08 11:54:03 WARN manager.MySQLManager: It looks like you are importing from mysql.
18/11/08 11:54:03 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
18/11/08 11:54:03 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
18/11/08 11:54:03 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
18/11/08 11:54:03 INFO mapreduce.ImportJobBase: Beginning import of t
18/11/08 11:54:03 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
18/11/08 11:54:03 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
18/11/08 11:54:03 INFO client.RMProxy: Connecting to ResourceManager at master/10.0.1.118:18040
18/11/08 11:54:07 INFO db.DBInputFormat: Using read commited transaction isolation
18/11/08 11:54:07 INFO mapreduce.JobSubmitter: number of splits:1
18/11/08 11:54:07 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1541640011445_0023
18/11/08 11:54:07 INFO impl.YarnClientImpl: Submitted application application_1541640011445_0023
18/11/08 11:54:07 INFO mapreduce.Job: The url to track the job: http://master:18088/proxy/application_1541640011445_0023/
18/11/08 11:54:07 INFO mapreduce.Job: Running job: job_1541640011445_0023
18/11/08 11:54:19 INFO mapreduce.Job: Job job_1541640011445_0023 running in uber mode : false
18/11/08 11:54:19 INFO mapreduce.Job: map 0% reduce 0%
18/11/08 11:54:31 INFO mapreduce.Job: Task Id : attempt_1541640011445_0023_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
... 9 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.Util.getInstance(Util.java:408)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:918)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:897)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:886)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860)
at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2097)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2022)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
... 10 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:341)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2189)
at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2038)
... 24 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:300)
... 26 more
18/11/08 11:54:41 INFO mapreduce.Job: Task Id : attempt_1541640011445_0023_m_000000_1, Status : FAILED
Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
... 9 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.Util.getInstance(Util.java:408)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:918)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:897)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:886)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860)
at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2097)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2022)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
... 10 more
Caused by: java.sql.SQLException: Access denied for user 'test'@'localhost' (using password: YES)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:964)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:873)
at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1710)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1226)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2191)
at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2038)
... 24 more
18/11/08 11:54:51 INFO mapreduce.Job: Task Id : attempt_1541640011445_0023_m_000000_2, Status : FAILED
Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
... 9 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.Util.getInstance(Util.java:408)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:918)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:897)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:886)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860)
at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2097)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2022)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
... 10 more
Caused by: java.sql.SQLException: Access denied for user 'test'@'localhost' (using password: YES)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:964)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:873)
at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1710)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1226)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2191)
at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2038)
... 24 more
18/11/08 11:55:03 INFO mapreduce.Job: map 100% reduce 0%
18/11/08 11:55:03 INFO mapreduce.Job: Job job_1541640011445_0023 failed with state FAILED due to: Task failed task_1541640011445_0023_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
18/11/08 11:55:03 INFO mapreduce.Job: Counters: 8
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=36276
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=36276
Total vcore-seconds taken by all map tasks=36276
Total megabyte-seconds taken by all map tasks=37146624
18/11/08 11:55:03 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
18/11/08 11:55:03 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 60.1829 seconds (0 bytes/sec)
18/11/08 11:55:03 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
18/11/08 11:55:03 INFO mapreduce.ImportJobBase: Retrieved 0 records.
18/11/08 11:55:03 ERROR tool.ImportTool: Error during import: Import job failed!
[hadoop@master ~]$ IP import successful:
[hadoop@master ~]$ sqoop import --username test --password test --connect jdbc:mysql://10.0.1.118:3306/test?autoReconnect=true --delete-target-dir --table t --fields-terminated-by '\t' -m 1 --hive-import --hive-database test --hive-table t
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/11/08 11:51:29 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
18/11/08 11:51:29 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/11/08 11:51:30 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/11/08 11:51:30 INFO tool.CodeGenTool: Beginning code generation
18/11/08 11:51:30 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `t` AS t LIMIT 1
18/11/08 11:51:30 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `t` AS t LIMIT 1
18/11/08 11:51:30 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce
Note: /tmp/sqoop-hadoop/compile/8dc5c288374d370426b87eca124b15e0/t.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
18/11/08 11:51:33 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/8dc5c288374d370426b87eca124b15e0/t.jar
18/11/08 11:51:34 INFO tool.ImportTool: Destination directory t deleted.
18/11/08 11:51:34 WARN manager.MySQLManager: It looks like you are importing from mysql.
18/11/08 11:51:34 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
18/11/08 11:51:34 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
18/11/08 11:51:34 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
18/11/08 11:51:34 INFO mapreduce.ImportJobBase: Beginning import of t
18/11/08 11:51:34 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
18/11/08 11:51:34 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
18/11/08 11:51:34 INFO client.RMProxy: Connecting to ResourceManager at master/10.0.1.118:18040
18/11/08 11:51:37 INFO db.DBInputFormat: Using read commited transaction isolation
18/11/08 11:51:37 INFO mapreduce.JobSubmitter: number of splits:1
18/11/08 11:51:38 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1541640011445_0022
18/11/08 11:51:38 INFO impl.YarnClientImpl: Submitted application application_1541640011445_0022
18/11/08 11:51:38 INFO mapreduce.Job: The url to track the job: http://master:18088/proxy/application_1541640011445_0022/
18/11/08 11:51:38 INFO mapreduce.Job: Running job: job_1541640011445_0022
18/11/08 11:51:49 INFO mapreduce.Job: Job job_1541640011445_0022 running in uber mode : false
18/11/08 11:51:49 INFO mapreduce.Job: map 0% reduce 0%
18/11/08 11:51:57 INFO mapreduce.Job: map 100% reduce 0%
18/11/08 11:51:57 INFO mapreduce.Job: Job job_1541640011445_0022 completed successfully
18/11/08 11:51:57 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=160855
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=87
HDFS: Number of bytes written=76
HDFS: Number of read operations=4
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Other local map tasks=1
Total time spent by all maps in occupied slots (ms)=5208
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=5208
Total vcore-seconds taken by all map tasks=5208
Total megabyte-seconds taken by all map tasks=5332992
Map-Reduce Framework
Map input records=10
Map output records=10
Input split bytes=87
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=98
CPU time spent (ms)=820
Physical memory (bytes) snapshot=107737088
Virtual memory (bytes) snapshot=2091540480
Total committed heap usage (bytes)=18919424
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=76
18/11/08 11:51:57 INFO mapreduce.ImportJobBase: Transferred 76 bytes in 22.9794 seconds (3.3073 bytes/sec)
18/11/08 11:51:57 INFO mapreduce.ImportJobBase: Retrieved 10 records.
18/11/08 11:51:57 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `t` AS t LIMIT 1
18/11/08 11:51:57 INFO hive.HiveImport: Loading uploaded data into Hive
18/11/08 11:52:08 INFO hive.HiveImport: SLF4J: Class path contains multiple SLF4J bindings.
18/11/08 11:52:08 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/home/hadoop/apache-hive-2.1.1/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
18/11/08 11:52:08 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/home/hadoop/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
18/11/08 11:52:08 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
18/11/08 11:52:08 INFO hive.HiveImport: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
18/11/08 11:52:09 INFO hive.HiveImport:
18/11/08 11:52:09 INFO hive.HiveImport: Logging initialized using configuration in file:/home/hadoop/apache-hive-2.1.1/conf/hive-log4j2.properties Async: true
18/11/08 11:52:18 INFO hive.HiveImport: OK
18/11/08 11:52:18 INFO hive.HiveImport: Time taken: 1.476 seconds
18/11/08 11:52:19 INFO hive.HiveImport: Loading data to table test.t
18/11/08 11:52:19 INFO hive.HiveImport: OK
18/11/08 11:52:19 INFO hive.HiveImport: Time taken: 1.235 seconds
18/11/08 11:52:20 INFO hive.HiveImport: Hive import complete.
18/11/08 11:52:20 INFO hive.HiveImport: Export directory is contains the _SUCCESS file only, removing the directory.
2.2 mysql is not local (remote)
is the hadoop cluster (multiple nodes), many data nodes, must ensure N data node can access mysql database (N depends on the specified sqoop -m parameter N, greater than or equal to the -m) at the back of the parameters, sqoop run at the bottom of the MR, MR run on hadoop cluster, so the hadoop each node must be able to connect the mysql database, mysql available – h * * * * * u * * * * * – p * * * * * test.
my environment:
three virtual machines, two network CARDS each, one private (communication between clusters), one public bridge (communication with outside machines)
The
| node | IP |
| master | 10.0.1.118 |
| slave – 1 | 10.0.1.227 |
| slave – 2 | 10.0.1.226
slave-1 to connect to remote database : slave-2 connects to remote database: import remote 172.1.1.96 data table test.student_exam_score_20170802 to hive bond_edu.student_exam_score_20170802
summary: 1. Sqoop remote import of mysql data into hive must ensure that N data nodes can connect to the remote database properly (N> or equal to the number of sqoop parallelism). 2. Sqoop local import users may need to use IP or modify the host file. 3. Sqoop list-tables does not require data nodes?(Doubtful??) . Read More:
|
