Sqoop reports an error can’t parse input data: ‘\ n’
Problem Description: use sqoop to push the data in hive to Oracle. The program reports an error: can’t parse input data: ‘\ n’
. Null value cannot be recognized when derivative.
The reason for this exception is that when the table in hive is a partitioned table, the added fields do not use cascade, such as:
alter table app.suntest_user add columns(name string comment 'name') cascade
When does not use the cascade keyword, hive does not update the history partition definition, resulting in differences between the columns in the table and the target table when transmitting history data.
For example:
False table suntest_ User created partition M08 in August, added a name column in September, and created partition M09, and added a name column to the table definition in the target database (Oracle). There is no problem exporting M09 partition data through the sqoop script, but an error will be reported when exporting M08 data: can’t parse input data: ‘\ n’
Solution:
You can rebuild the partition in August , or create a table for derivatives. The following is a table creation statement.
----- Create a table corresponding to the partition and use the temporary derivative
create table app.test_suntest_user as
select id,name from app.suntest_user where partition_month='M08'
----Rebuild partition statement
--Rename partition
alter table app.suntest_user partition (partition_month='M08') rename to partition (partition_month='M08bak');
--New Partition
insert overwrite table app.suntest_user partition(partition_month='M08')
select id ,name from app.suntest_user where partition_month='M08bak';
The sqoop statement is as follows:
sqoop export --connect "jdbc:oracle:thin:@**********" \
--username "abc" \
--password "123" \
--input-null-string '\\N' --input-null-non-string '\\N' \
--table "SUNTEST_USER" \
--export-dir /user/hive/warehouse/app/test_suntest_user\
--input-fields-terminated-by '\001' \
Read More:
- Sqoop exports hive data to MySQL Error [How to Solve]
- Sqoop Error: Error during export:Export job failed [How to Solve]
- Doris BrokerLoad Error: No source file in this table [How to Solve]
- [Solved] lightdb oracle_fdw Error: ERROR: error connecting to Oracle: OCIEnvCreate failed to create environment handle
- How to Solve Doris dynamic partition table routineload Error
- [Solved] ora 01033 linux,ORA-01033: ORACLE initialization or shutdown in progres
- [Solved] Hive Error: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
- [Solved] Hive 2.3.9 Error: Error: Unrecognized column type: UNIONTYPE (state=,code=0)
- [Solved] sqoop Error: jSQLException in nextKeyValue Caused by: ORA-24920:column size too large for client
- [Solved] Doris StreamLoad Error: load by MERGE or DELETE need to upgrade table to support batch delete
- [Solved] Hive execute insert overwrite error: could not be cleared up
- Doris BrokerLoad Error: quality not good enough to cancel
- Doris BrokerLoad Error: quality not good enough to cancel
- ORA-08103: object no longer exists [How to Solve]
- Ant Design Vue-Table Error: warning.js?2149:7 [How to Solve]
- DB::Exception: test: Authentication failed: [How to Solve]
- How to Solve Expdp Error ORA-39126
- How to Sovle expdp/impdp Data Pump Error
- Java.sql.SQLException: ORA-02291: integrity constraint violated – par
- Mysql5.7.18.1 Error 1054 (42S22): Unknown Column’password’ In’field List’ When Changing User Password