Tag Archives: Spark ERROR

[Solved] Spark Error: org.apache.spark.SparkException: A master URL must be set in your configuration

Error when running the project to connect to Spark:

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
22/10/08 21:02:10 INFO SparkContext: Running Spark version 3.0.0
22/10/08 21:02:10 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: A master URL must be set in your configuration
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:380)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:120)
	at test.wyh.wordcount.TestWordCount$.main(TestWordCount.scala:10)
	at test.wyh.wordcount.TestWordCount.main(TestWordCount.scala)
22/10/08 21:02:10 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.spark.SparkException: A master URL must be set in your configuration
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:380)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:120)
	at test.wyh.wordcount.TestWordCount$.main(TestWordCount.scala:10)
	at test.wyh.wordcount.TestWordCount.main(TestWordCount.scala)

Process finished with exit code 1

Solution:

Configure the following parameters:

-Dspark.master=local[*]

Restart IDEA.

[Solved] Spark Error: ERROR StatusLogger No log4j2 configuration file found

I. introduction

When running Kafka related procedures of spark # project, it was warned that although it did not affect the operation, OCD looked very uncomfortable, so it was cleared immediately.

ERROR StatusLogger No log4j2 configuration file found. 
Using default configuration: logging only errors to the console.

II. Problem-solving

1. Add log4j2.xml

Level can be configured in loggers

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{YYYY-MM-dd HH:mm:ss} [%t] %-5p %c{1}:%L - %msg%n" />
        </Console>

        <RollingFile name="RollingFile" filename="log/test.log"
                     filepattern="${logPath}/%d{YYYYMMddHHmmss}-fargo.log">
            <PatternLayout pattern="%d{YYYY-MM-dd HH:mm:ss} [%t] %-5p %c{1}:%L - %msg%n" />
            <Policies>
                <SizeBasedTriggeringPolicy size="10 MB" />
            </Policies>
            <DefaultRolloverStrategy max="20" />
        </RollingFile>

    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="Console" />
            <AppenderRef ref="RollingFile" />
        </Root>
    </Loggers>
</Configuration>

2. Add location

Add it to the Src/main/Resources folder and execute MVN install

3. Display

Since the level in the file is set to info, you can see many related logs and modify them yourself

Spark ERROR client.TransportResponseHandler: Still have 1 requests outstanding when connection from

The spark task reports an error (though the calculated result is correct).

21/04/29 14:38:38 ERROR client.TransportResponseHandler: Still have 1 requests outstanding when connection from /172.16.4.156:37528 is closed
21/04/29 14:38:39 ERROR cluster.YarnScheduler: Lost executor 3 on cdh-slave1.test.com: Container marked as failed: container_1618397085347_0529_01_000004 on host: cdh-slave1.test.com. Exit status: 137. Diagnostics: [2021-04-29 14:38:39.112]Container killed on request. Exit code is 137
[2021-04-29 14:38:39.117]Container exited with a non-zero exit code 137.
[2021-04-29 14:38:39.119]Killed by external signal

Looking at the code, I found that hive is not used in the spark program, but hive support is set during spark initialization.

The program works fine after deleting this line of code.