The tasks run by spark are often viewed through the web. However, when the spark streaming task is running, the log is often very large, which makes it inconvenient to view the web. Therefore, it is necessary to locate it on the server. Here are two ways to view the driver side and the executor side logs.

1、 View the web log:

The following is the general web interface of four yarn scheduling spark tasks:

Click the first task: application_ 1509845442132_ 3866 enter the interface below. The log recorded in the lower right corner is actually the log of the driver side. The driver side is on the mosaic node.

In addition, we can view the log on the executor node. As shown in the figure above, open the applicationmaster and jump to the general task scheduling interface of spark

After clicking on the executor, you can see four executors and a driver. See the log on the right. Stdout is the output log of println, and stderr is the standard log of spark output.

2、 Server side log view

The task log of sparkstreaming is often very large, so it is not convenient to view it on the web. So we need to go to the server to view it. As you can see from the web, that node is the driver. The driver side log is usually in the following directory: Horn/container logs/

In case you don’t know which directory it is, you can find it directly: find/- name “application_ 1509845442132_ 3866”

The corresponding executor log is also found on the server in this way.