Tag Archives: Big data

[Solved] Spark job failed during runtime. Please check stacktrace for the root cause.

hive on spark reports an error
executing the hive command is an error

[42000][3] Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed during runtime. Please check stacktrace for the root cause.

[Reason]
View running tasks on yarn, Query error results from error log

Map operator initialization failed: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected column vector type LIST

List type error
List corresponds to array in hive, array corresponds to list in Java

[Solution]
Temporarily change the execution engine to MR

set hive.execution.engine=mr;

There are many bugs in hive on spark, When an unknown error occurs, First try to replace the underlying execution engine with MR, to execute the sql statement.

[Subsequent modification]
1. View the current execution engine of hive:

set hive.execution.engine;

2. Manually set hive’s current execution engine to Spark

set hive.execution.engine=spark;

3. Manually set hive’s current execution engine to MR

set hive.execution.engine=mr;

[Solved] Kafka Restarts error | Cloudera Manager Access Returns 500 | HDFS Startup Error

Hi~ Long time no update

1.Problems that need attention after restarting kafka:
Kafka will have a write file a in the target storage location during execution,this file a will keep a write state for a while,usually one hour Heavy
Generate a new write file b,End the last write file a(The duration of this ending needs to check the configuration of each cluster). then restart
Here comes the problem,The last write file a,will be recreated after restarting,The last write file b,So the current
a will keep writing status,when reading and writing file a, it will report an error,including importing Hive query will also report an error&# xff08; load to hive
The table will not report an error,but it will report an error when selecting),because this file is always in the write state,It is inoperable,It is also called writing
Lock(I believe everyone has heard of).
Solution:Then we need to manually terminate the write status of the write file,First we need to determine the status of the write file,In the command
Execute the command on the line :
hdfs fsck /data/logs/( Write the directory where the file is located,Change according to where your file is located) -openforwrite
The displayed files are all in the write state:

insert picture description here

 After seeing the writing file,execute the command to stop all writing files,here explain,why all stop&# xff0c; Logically, it should be stopped before
A write file, but stopping all of them can also solve the problem, is relatively simple and violent, because manual stopping will automatically generate a
, writing files, so you can stop them all. then now execute the command :
hdfs debug recoverLease -path /logs/common_log/2022-09 -16/FlumeData.1663292498820.tmp(Execute the previous command to display Output write file path) -retries 3
It can be solved by executing each file once,Say more,If this file has been loaded into hive,, you need to go to /user/warehouse/hive/ to find this write status file

insert picture description here

2.CDH's Cloudera Manager launch browser access returns 500error:
① First check the configuration of the /etc/hosts file, only need to leave these two lines with the cluster The intranet IP mapping can be
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

②It is also necessary to check whether the ports related to ,cm are occupied by the firewall.

③ Then restart CM, execute the command
nameNode:systemctl stop cloudera-scm-server
Then execute :systemctl stop cloudera-scm-agent on each node

nameNode:systemctl start cloudera-scm-server
Then execute :systemctl start cloudera-scm-agent on each node
Attention Pay attention to!!! The execution order of these commands cannot be reversed, Otherwise, there may be problems with cluster startup.
Then you can systemctl status cloudera-scm-server, systemctl status cloudera-scm-agent
Check out the operation.

②If cm starts and can access , but starts HDFS error 1 or 2
1.Unable to retrieve non-local non-loopback IP address. Seeing address: cm/127.0.0.1
2.ERROR ScmActive-0:com.cloudera.server.cmf. components.ScmActive: ScmActive was not able to access CM identity  to validate it.2017-04-18 09:40 :29,308 ERROR ScmActive-0

So congratulations ,find a solution.
First find the source database of CM,Some of them were configured at that time,If you don’t know, ask the person who installed them,Almost all of them are in
Don't ask me for the , account password on nameNode ~, then show databases; can See that there is a cm or scm library

insert picture description here

 use this library,then show tables;
You will see a table called HOSTS,View the data of this table-select * from HOSTS ;

insert picture description here

 You will find that there is a different line , that is, there is a difference between NAME and IP_ADDRESS, Then you need to modify it back, to
The name and IP_ADDRESS of the intranet,I believe everyone will modify it!Then restart the CM,It's done!

start-all.sh Execute error: Stopping journal nodes [slave2 slave1 master]…

Stopping journal nodes [slave2 slave1 master]
ERROR: Attempting to operate on hdfs journalnode as root
ERROR: but there is no HDFS_JOURNALNODE_USER defined. Aborting operation.
Stopping ZK Failover Controllers on NN hosts [master slave1 slave2]
ERROR: Attempting to operate on hdfs zkfc as root
ERROR: but there is no HDFS_ZKFC_USER defined. Aborting operation.

 

Error Cause:
Trying to operate on hdfs namenode as root user, but HDFS _ NAMENODE _ user is not defined. Abort the operation.

Solution:
In the environment variables in add.
1. into the environment variables
vi ~/.bash_profile
2. Add the following codes:

#hadoop
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_JOURNALNODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
export HDFS_ZKFC_USER=root

Enabling environment variables
source ~/.bash_profile
​​​​​

[Windows] elasticsearch.exceptions.RequestError: <unprintable RequestError object>

elasticsearch.exceptions.RequestError: <unprintable RequestError object>

There are many ways to solve this problem.
Use the following two commands in Pycharm
$pip install django haystack
$pip install elasticsearch==2.4.1

Note that the server-side elasticsearch should be consistent with the pip install elasticsearch==2.4.1 version

How to Solve elasticsearch and logstash Install Error

Turning on the logstash service appears: Failed to start logstash.service: Unit not found.

[[email protected] ~]# systemctl start logstash
Failed to start logstash.service: Unit not found.

Issue 1:
First problem: Failed to start logstash.service: Unit not found.
Solution idea.
Generate logstash.service file

[[email protected] ~]# sudo /usr/share/logstash/bin/system-install /etc/logstash/startup.options systemd

Check if the service can be opened normally

Issue 2:
The second problem: If you use this access could not find any executable java binary. Please install java in your PATH or set JAVA_HOME.

[[email protected] ~]# sudo /usr/share/logstash/bin/system-install /etc/logstash/startup.options systemd
Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME.

Reason: logstash can’t get to AVA_HOME variable, need to add refresh profile in the configuration file
Solution.

[[email protected] ~]# vi /etc/profile                #Add the specified version of the JDK directory installed on the local machine
export JAVA_HOME=/usr/local/jdk1.8
export CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/bin:$PATH:$HOME/bin

[[email protected] ~]# vi /usr/share/logstash/bin/logstash.lib.sh
Add source /etc/profile in the last line
[[email protected] ~]# vi /usr/share/logstash/bin/logstash
Add source /etc/profile in the last line

Refresh the configuration file, and then see if the service can be opened normally

Issue 3:
Third problem : /usr/share/logstash/vendor/jruby/bin/jruby:line 388: /usr/bin/java: No that file or directory
Unable to install system startup script for Logstash.
Reason: Can’t get the java executable file
Solution:

[[email protected] ~]# ln -s /usr/local/jdk1.8/bin/java /usr/bin/java

Reinstall the service to install.

[[email protected] ~]# rpm -e logstash
Error: package logstash is not installed
[[email protected] ~]# rpm -ivh /mnt/logstash-5.5.1.rpm
Warning: /mnt/logstash-5.5.1.rpm: Head V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
In preparation…                          ################################# [100%]
Package logstash-1:5.5.1-1.noarch is installed

Generate logstash.service file

[[email protected] ~]# sudo /usr/share/logstash/bin/system-install /etc/logstash/startup.options systemd
Using provided startup.options file: /etc/logstash/startup.options

Start successfully!

[[email protected] ~]# systemctl start logstash

[Solved] org.springframework.beans.factory.UnsatisfiedDependencyException

It can be understood that the dependency failed, and the dependency could not be found

Some errors are reported as follows:

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'pageService': Unsatisfied dependency expressed through field 'pageInfoMepper'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pageInfoMepper' defined in file [D:projectIDEAProjectdemoPageoutartifactsWEB-INFclassescomligleimapperPageInfoMepper.class]: Cannot resolve reference to bean 'sqlSessionFactory' while setting bean property 'sqlSessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sqlSessionFactory' defined in file [D:projectIDEAProjectdemoPageoutartifactsWEB-INFclassesapplicationContext.xml]: Invocation of init method failed; nested exception is org.springframework.core.NestedIOException: Failed to parse config resource: class path resource [mybatis-config.xml]; nested exception is org.apache.ibatis.builder.BuilderException: Error parsing SQL Mapper Configuration. Cause: org.apache.ibatis.builder.BuilderException: Error resolving class. Cause: org.apache.ibatis.type.TypeException: Could not resolve type alias 'com.github.pagehelper.PageInterceptor'.  Cause: java.lang.ClassNotFoundException: Cannot find class: com.github.pagehelper.PageInterceptor
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:596)
	at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:90)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:374)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1411)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:592)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:843)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
	at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:400)
	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:291)
	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4643)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5109)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:743)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:719)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:703)
	at org.apache.catalina.startup.HostConfig.manageApp(HostConfig.java:1737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:287)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
	at org.apache.catalina.mbeans.MBeanFactory.createStandardContext(MBeanFactory.java:457)
	at org.apache.catalina.mbeans.MBeanFactory.createStandardContext(MBeanFactory.java:406)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:287)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
	at com.sun.jmx.remote.security.MBeanServerAccessController.invoke(MBeanServerAccessController.java:468)
	at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
	at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
    .......
    .......

Solution:

1. First, see if the annotation on the Service layer class is added or wrong, it should be @Service, whether the annotation refers to the Spring class, do not import it into another package, and see the path of the package.

2. Also, if the Service layer is divided into interfaces and implementation classes, see if the implementation class is annotated (@Service), and see if there is an implementation class.

3. Take a look at the configuration file. xml, is there any package where the automatic scan service is started?

<context:component-scan base-package="com.liglei.service"></context:component-scan>

4. Check if the jar package is downloaded completely and if there is this jar package

5. Just to see if there are any packages in the red box in the figure below that have not been imported, and if so, follow the steps below:

File–> Project Structure–> Artifacts–> Right click demopage –> Put into Output Root–> OK

Make sure the above steps are OK. Restart the project and try again~

How to Solve hadoop3.x.x sh start-dfs.sh Startup Error

hadoop3.x.x sh start-dfs.sh startup error

Error information:

/app/module/hadoop310/libexec/hadoop-functions.sh: line 398: syntax error near unexpected token `<'
/app/module/hadoop310/libexec/hadoop-functions.sh: line 398: `  done < <(for text in "${input[@]}"; do'
/app/module/hadoop310/libexec/hadoop-config.sh: line 70: hadoop_deprecate_envvar: command not found
/app/module/hadoop310/libexec/hadoop-config.sh: line 87: hadoop_bootstrap: command not found
/app/module/hadoop310/libexec/hadoop-config.sh: line 104: hadoop_parse_args: command not found
/app/module/hadoop310/libexec/hadoop-config.sh: line 105: shift: : numeric argument required
/app/module/hadoop310/libexec/hadoop-config.sh: line 110: hadoop_find_confdir: command not found
/app/module/hadoop310/libexec/hadoop-config.sh: line 111: hadoop_exec_hadoopenv: command not found
/app/module/hadoop310/libexec/hadoop-config.sh: line 112: hadoop_import_shellprofiles: command not found
/app/module/hadoop310/libexec/hadoop-config.sh: line 113: hadoop_exec_userfuncs: command not found
/app/module/hadoop310/libexec/hadoop-config.sh: line 119: hadoop_exec_user_hadoopenv: command not found
/app/module/hadoop310/libexec/hadoop-config.sh: line 120: hadoop_verify_confdir: command not found
/app/module/hadoop310/libexec/hadoop-config.sh: line 122: hadoop_deprecate_envvar: command not found
/app/module/hadoop310/libexec/hadoop-config.sh: line 123: hadoop_deprecate_envvar: command not found
/app/module/hadoop310/libexec/hadoop-config.sh: line 124: hadoop_deprecate_envvar: command not found
/app/module/hadoop310/libexec/hadoop-config.sh: line 129: hadoop_os_tricks: command not found
/app/module/hadoop310/libexec/hadoop-config.sh: line 131: hadoop_java_setup: command not found
/app/module/hadoop310/libexec/hadoop-config.sh: line 133: hadoop_basic_init: command not found
/app/module/hadoop310/libexec/yarn-config.sh: line 36: hadoop_deprecate_envvar: command not found
/app/module/hadoop310/libexec/yarn-config.sh: line 38: hadoop_deprecate_envvar: command not found
/app/module/hadoop310/libexec/yarn-config.sh: line 40: hadoop_deprecate_envvar: command not found
/app/module/hadoop310/libexec/yarn-config.sh: line 42: hadoop_deprecate_envvar: command not found
/app/module/hadoop310/libexec/yarn-config.sh: line 44: hadoop_deprecate_envvar: command not found
/app/module/hadoop310/libexec/yarn-config.sh: line 46: hadoop_deprecate_envvar: command not found
/app/module/hadoop310/libexec/yarn-config.sh: line 48: hadoop_deprecate_envvar: command not found
/app/module/hadoop310/libexec/yarn-config.sh: line 50: hadoop_deprecate_envvar: command not found
/app/module/hadoop310/libexec/yarn-config.sh: line 52: hadoop_deprecate_envvar: command not found
/app/module/hadoop310/libexec/yarn-config.sh: line 54: hadoop_deprecate_envvar: command not found
/app/module/hadoop310/libexec/yarn-config.sh: line 62: hadoop_deprecate_envvar: command not found
/app/module/hadoop310/libexec/yarn-config.sh: line 64: hadoop_deprecate_envvar: command not found
/app/module/hadoop310/libexec/hadoop-config.sh: line 140: hadoop_shellprofiles_init: command not found

The specific cause of the error has not been found yet.
Solution: switch to the Hadoop directory and go directly.
cd /app/module/hadoop310
./sbin/start-dfs.sh

[Solved] SpringBoot Integrate ES Error: Elasticsearch health check failed

Recently, when a springboot integrated es project was started, an error was reported after successful startup: Elasticsearch health check failed

There are two method to solve this error:

1. Close the health check of the actor on elasticsearch (I tried this method, and the project cannot be started later, and this method is not recommended):

management:
  health:
    elasticsearch:
      enabled: false

2. Configure according to spring.elasticsearch.rest. uris (the problem is solved after restart):

spring:
  # ES search engine
  data:
    elasticsearch:
      cluster-nodes: 47.103.5.190:9300
      cluster-name: docker-cluster
      repositories:
        enabled: true
  elasticsearch:
    rest:
      uris: ["http://47.103.5.190:9200"]

[Solved] Logstash Error: Logstash – java.lang.IllegalStateException: Logstash stopped processing because of an err

I recently tried to use Elasticsearch and IK in combination with Logstash to link mysql, and tested Logstash with the following error message.

First enter the command: logstash -e ‘input {stdin{}} output {stdout{}}’

D:\myworkspace\es\logstash-6.4.3\bin>logstash -e 'input {stdin{}} output {stdout{}}'

The command is correct, but the result is:

D:\myworkspace\es\logstash-6.4.3\bin>logstash -e 'input {stdin{}} output {stdout{}}'
ERROR: Unknown command '{stdin{}}'

See: 'bin/logstash --help'
[ERROR] 2022-08-23 09:06:42.875 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

 

Solution:

You should try the following command first:

logstash -e “”

The result was successful:

D:\myworkspace\es\logstash-6.4.3\bin>logstash -e ""
Sending Logstash logs to D:/myworkspace/es/logstash-6.4.3/logs which is now configured via log4j2.properties
[2022-08-23T09:16:16,950][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"D:/myworkspace/es/logstash-6.4.3/data/queue"}
[2022-08-23T09:16:16,958][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"D:/myworkspace/es/logstash-6.4.3/data/dead_letter_queue"}
[2022-08-23T09:16:17,054][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-08-23T09:16:17,164][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"0777ac0f-9efb-463d-8e2c-874bc1dc9feb", :path=>"D:/myworkspace/es/logstash-6.4.3/data/uuid"}
[2022-08-23T09:16:17,592][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.4.3"}
[2022-08-23T09:16:20,129][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2022-08-23T09:16:20,231][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x5fba80a0 run>"}
The stdin plugin is now waiting for input:
[2022-08-23T09:16:20,277][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-08-23T09:16:20,611][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2022-08-23T09:16:43,203][WARN ][logstash.runner          ] SIGINT received. Shutting down.
[2022-08-23T09:16:43,338][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x5fba80a0 run>"}
[2022-08-23T09:16:43,340][FATAL][logstash.runner          ] SIGINT received. Terminating immediately.

Decisively replace with the following command:

logstash -e "input { stdin {} }  output {stdout {} }"

Done!

D:\myworkspace\es\logstash-6.4.3\bin>logstash -e "input { stdin {} }  output {stdout {} }"
Sending Logstash logs to D:/myworkspace/es/logstash-6.4.3/logs which is now configured via log4j2.properties
[2022-08-23T09:17:48,125][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-08-23T09:17:48,690][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.4.3"}
[2022-08-23T09:17:50,871][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2022-08-23T09:17:50,964][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x268e4bb5 run>"}
The stdin plugin is now waiting for input:
[2022-08-23T09:17:51,008][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-08-23T09:17:51,209][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

[Solved] ANTRL Compile HiveLexer.g File Error: syntax error: antlr: NoViableAltException(@[])

Project scenario:

The need for programming to obtain data pedigrees is not common for data warehouses, and separate data pedigree work is only necessary when the scale of data reaches a significant level, or the number of reports with complex data production relationships increases to a significant level.
Until scale is reached, manual identification and management is more cost effective.

antlr is the open source syntax parser that automatically generates syntax trees based on input and displays them visually. antlr-Another Tool for Language Recognition, formerly PCCTS, provides a syntax tree for languages including Java, C++, and C# by description to automatically construct a custom language recognizer (recognizer), compiler (parser) and interpreter (translator) framework. ANTLR is available in v2 v3 v4, with most of the Chinese documentation being in v2 and the hive 1.1.0 version mentioning antlr 3.4 in the comments. antlr combines the above by allowing us to define lexical rules for recognizing character streams and syntactic parser rules for interpreting Token streams. ANTLR will then automatically generate the appropriate lexical/syntactic parser based on the syntax file provided by the user. The user can use them to compile the input text and convert it into other forms (e.g. AST-Abstract Syntax Tree).

Use ANTLR version 3.5.2 to parse Hivesql’s source code grammar file for HiveLexer.g, and place the code:

@lexer::header {
package org.apache.hadoop.hive.ql.parse

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.hive.conf.HiveConf
}

@lexer::members {
  private Configuration hiveConf
  
  public void setHiveConf(Configuration hiveConf) {
    this.hiveConf = hiveConf
  }
  
  protected boolean allowQuotedId() {
    if(hiveConf == null){
      return false
    }
    String supportedQIds = HiveConf.getVar(hiveConf, HiveConf.ConfVars.HIVE_QUOTEDID_SUPPORT)
    return !"none".equals(supportedQIds)
  }
}

This paragraph needs to be annotated so that Java compiled with ANTLR does not contain this package.


Problem description

After this paragraph is annotated with/* * /, it is still compiled and an error is found:

[10:58:10] error(100): HiveLexer.g:42:1: syntax error: antlr: NoViableAltException([email protected][])
[10:58:10] error(100): HiveLexer.g:42:2: syntax error: antlr: NoViableAltException([email protected][])
[10:58:10] error(100): HiveLexer.g:42:7: syntax error: antlr: MissingTokenException(inserted [@-1,0:0='<missing ACTION>',<4>,42:6] at :)
[10:58:10] error(100): HiveLexer.g:42:8: syntax error: antlr: NoViableAltException([email protected][])
[10:58:10] error(100): HiveLexer.g:42:9: syntax error: antlr: NoViableAltException([email protected][])
[10:58:10] error(100): HiveLexer.g:42:9: syntax error: antlr: NoViableAltException([email protected][])
[10:58:10] error(100): HiveLexer.g:47:1: syntax error: antlr: MissingTokenException(inserted [@-1,0:0='<missing SEMI>',<82>,47:0] at KW_TRUE)

Cause analysis:

Query data discovery

‘; ‘ is a terminator in comment and so error occurred


Solution:

Delete all the original annotation documents or delete them; Delete it.

Problem solved.

[Solved] Cause: java.sql.SQLException: TDengine ERROR (8000000b): Unable to establish connection

FQDN is not configured during installation and configuration. There is no error when starting spingboot. The following error is reported when inserting data

Tdengine error (800000b): unable to establish connection

### Cause: java.sql.SQLException: TDengine ERROR (8000000b): Unable to establish connection

Solution:

vi /etc/taos/taos.cfg

/var/lib/taos/dnode/dnodeEps.json

The local windows system must install the client tdengine-client

Hosts to be set, as shown in the following figure

[Solved] hadoop-2.7.1 Error: Error Cannot find configuration directory etchadoop

Since the configuration is hadoop-2.7.1, it will be found later during startup

The terminal executes./start yarn sh
starting yarn daemons
Error: Cannot find configuration directory: /etc/hadoop
Error: Cannot find configuration directory: /etc/hadoop

This is the reason why the directory cannot be found. You can find the solution by reading the corresponding shell script~

Solution:

Configures the directory where a Hadoop configuration file is located in hadoop-env.sh (codes as below):

export HADOOP_CONF_DIR=/opt/hadoop-2.7.1/etc/hadoop/

Execute command

source  hadoop-env.sh