Apache spark output messages are prepended by [error] - scala

Question

This word count works as expected :
System.setProperty("hadoop.home.dir", "H:\\winutils");
val sparkConf = new SparkConf().setAppName("GroupBy Test").setMaster("local[1]")
val sc = new SparkContext(sparkConf)
def main(args: Array[String]) {
val text_file = sc.textFile("h:\\data\\small.txt")
val counts = text_file.flatMap(line => line.split(" "))
.map(word => (word, 1))
.reduceByKey(_ + _)
counts.foreach(println);
}
All output messages are prepended by [error]
example :
[error] 16/03/17 12:13:58 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on por
[error] 16/03/17 12:13:58 INFO NettyBlockTransferService: Server created on 55715
[error] 16/03/17 12:13:58 INFO BlockManagerMaster: Trying to register BlockManager
[error] 16/03/17 12:13:58 INFO BlockManagerMasterEndpoint: Registering block manager localhost:55715 with 1140.4 MB RAM, BlockManage
[error] 16/03/17 12:13:58 INFO BlockManagerMaster: Registered BlockManager
I can prevent these error messaged being displayed using :
import org.apache.log4j.Logger
import org.apache.log4j.Level
Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)
But this does not fix the issue.
[error] should not be displayed as these are not error messages but are info :
[error] 16/03/17 12:13:58 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on por
[error] 16/03/17 12:13:58 INFO NettyBlockTransferService: Server created on 55715
[error] 16/03/17 12:13:58 INFO BlockManagerMaster: Trying to register BlockManager
[error] 16/03/17 12:13:58 INFO BlockManagerMasterEndpoint: Registering block manager localhost:55715 with 1140.4 MB RAM, BlockManage
[error] 16/03/17 12:13:58 INFO BlockManagerMaster: Registered BlockManager
Update :
Why are [error] messages being displayed as they are not errors ?

Answer 1

Those are not Spark labels but sbt ones. In the default log4j config file of Spark you can find:
log4j.appender.console.target=System.err
So by default it will print to stderr in the console.
You probably are setting fork to true in your run config somewhere. When doing so everything that is printed to stderr in sbt is prepended with [error].
You should be able to control it with the OutputStrategy.