How to append Spark ApplicationID in filename of log4j log file - Scala - scala

Question

I am trying to append the Spark applicationId to the filename of log4j log file. Below is log4j.properties file
log4j.rootLogger=info,file
# Redirect log messages to console
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L -%m%n
# Redirect log messages to log file, support file rolling
log4j.appender.file=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.file.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
log4j.appender.file.rollingPolicy.FileNamePattern=log4j//Data_Quality.%d{yyyy-MM-dd}.log
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L -%m%n
# set the immediate flush to true
log4j.appender.FILE.ImmediateFlush=true
# set the threshold to debug mode INFO
log4j.appender.FILE.Threshold=INFO
#Set the append to false, overwrite
log4j.appender.FILE.Append=true
Spark-submit command:
spark2-submit -conf "spark.driver.extraJavaOptions=-Dconfig.file=./input.conf -Dlog4j.configuration=log4j.properties" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties" --files "input.conf,log4j.properties" --master yarn --class "DataCheckImplementation" Data_quality.jar
Logfiles are created with name : Data_Quality.2020-07-21.log, which is working correctly.
I want to add Spark ApplicationID to filename
Expected filename : Data_Quality.(ApplicationID).2020-07-21.log
Example: Data_Quality.(application_1595144411765_20000).2020-07-21.log
Is it possible? Need help!

Answer 1

I don't know/think if this can be at configuration level (e.g lo4j.properties, etc), but there are ways we can achieve this. Here is one approach:
You will need to have a logger class/trait where you deal with all you logger management, something like :
trait SparkContextProvider {
def spark: SparkSession
}
trait Logger extends SparkContextProvider {
lazy val log = Logger.getLogger(...)
lazy val applicationId = spark.sparkContext.applicationId
val appender = new RollingFileAppender();
appender.setAppend(true);
appender.setMaxFileSize("1MB");
appender.setMaxBackupIndex(1);
appender.setFile("Data_Quality" + applicationId + "_" + dateFormat.format(date) + ".log");
appender.activateOptions();
val layOut = new PatternLayout();
layOut.setConversionPattern("%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n");
appender.setLayout(layOut);
log.addAppender(appender);
}