The tasks run by spark are often viewed through the web. However, when the spark streaming task is running, the log is often very large, which makes it inconvenient to view the web. Therefore, it is necessary to locate it on the server. Here are two ways to view the driver side and the executor side logs.
1、 View the web log:
The following is the general web interface of four yarn scheduling spark tasks:
Click the first task: application_ 1509845442132_ 3866 enter the interface below. The log recorded in the lower right corner is actually the log of the driver side. The driver side is on the mosaic node.
In addition, we can view the log on the executor node. As shown in the figure above, open the applicationmaster and jump to the general task scheduling interface of spark
After clicking on the executor, you can see four executors and a driver. See the log on the right. Stdout is the output log of println, and stderr is the standard log of spark output.
2、 Server side log view
The task log of sparkstreaming is often very large, so it is not convenient to view it on the web. So we need to go to the server to view it. As you can see from the web, that node is the driver. The driver side log is usually in the following directory: Horn/container logs/
In case you don’t know which directory it is, you can find it directly: find/- name “application_ 1509845442132_ 3866”
The corresponding executor log is also found on the server in this way.
- Spark SQL startup error: error creating transactional connection factory
- Yarn: runtime.ContainerExecutionException : launch container failed
- Introduction of Hadoop HDFS and the use of basic client commands
- Start cell keyerror and report an error
- Some problems about startup
- ImportError: cannot import name ‘SparkSession‘
- Gulp error report, assertionerror [err_ ASSERTION]: Task function must be specified
- org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session 0354
- History Metastore Canary failed to create database
- Python switch / case statement implementation method
- Error: Cannot find module’webpack/bin/config-yargs’ solution
- Syntax error near unexpected token `newline’script cannot be executed
- An error occurred while starting the application
- Condition judgment and error handling in ansible playbook (fail module failed_when changed_when keyword)
- Connection for controluser as defined in your configuration failed.
- Linux dynamically loads kernel modules
- VBoxManage.exe: error: Failed to create the host-only adapter
- Node.js Using port 80 to report errors in Linux
- Could not start Quartz Scheduler after delay； nested exception is org.quartz.SchedulerException:
- It’s time to upgrade your parquet: IOException: totalvaluecount = = 0