Logging for Custom Code Written in Spark - scala

Question

We are writing Scala code for Apache Spark and running the process in Yarn Mode (Yarn Client Mode) in cloudera 5.5. Spark version is 1.5
I need to do logging for this code and want to move logs in Specific Directory for Spark ,outside of noise in spark logs
We are using plain log4j. Dont have time for logging trait for now.
I have changed the default log4j file in ${spark_home} like this
# Set everything to be logged to the console
log4j.rootLogger=ERROR, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
# Spark Logs
log4j.appender.aAppender=org.apache.log4j.RollingFileAppender
log4j.appender.aAppender.File=/var/log/aLogger.log
log4j.appender.aAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.aAppender.layout.ConversionPattern=[%p] %d %c %M - %m%n
log4j.appender.aAppender.maxFileSize=50MB
log4j.appender.aAppender.maxBackupIndex=5
log4j.appender.aAppender.encoding=UTF-8
# My custom logging goes to another file
log4j.logger.fsaLogger=ERROR, fsaAppender
This is my Code in scala
This is inside main method
val logger =LogManager.getLogger("aLogger")
val jobName = "AData"
logger.warn("jobName ::" +jobName +" Started at ::" +
+Calendar.getInstance().getTime())
val sc = new SparkContext(conf)
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
val sourceDF = sqlContext.read
.format("com.databricks.spark.avro")
.load(pathToFiles)
finalCSMDF.map { row =>
//To remove Serilizable Exception
var loggers = org.apache.log4j.LogManager.getLogger("aLogger")
loggers.error("Test Log")
row
}
logger.warn("jobName ::" +jobName +" completed at ::" +
+Calendar.getInstance().getTime())
My problem here is that when ever i run this in yarn mode , i find the lines jobName ::" +jobName started" etc in /var/log/aLogger.log but not the loggers.error("Test Log") which is inside the closure. Basically everything in driver is going in closure but not the exceutioners closures
Please help