I have tried using log4j and slf4j with akka in scala and I am able to get log files. Can I achieve the same thing without using any external api other than akka APIs? By using akka.event.Logging I am able to print logs in console, but I want to print it in a file.
I have already tried setting log4j.properties file for my project in classpath and its not working when I am using akka.event.Logging.
Please suggest.
Accordingly to this http://doc.akka.io/docs/akka/current/java/logging.html
you have 3 options:
Use akka.event.Logging$DefaultLogger (to stdout, not for production)
Use akka.event.slf4j.Slf4jLogger (logger by akka for SLF4J)
Use the SLF4J API directly (with async appender)
Your case is 2 or 3 (you use log4j.properties).
Therefore you should properly configure file log4j.properties for output in file.
And
in case 2 (your desired case), you should use akka.event.Logging, for example: Logging.getLogger(system.eventStream(), "my.string")
in case 3, you should use SLF4J API, for example: org.slf4j.LoggerFactory.getLogger(...)
Your case is 2, if you use in your akka config something like this:
akka {
loggers = ["akka.event.slf4j.Slf4jLogger"]
loglevel = "DEBUG"
logging-filter = "akka.event.slf4j.Slf4jLoggingFilter"
}
Related
I have defined some akka remote settings in my application.conf:
akka {
actor {
provider = "akka.remote.RemoteActorRefProvider"
warn-about-java-serializer-usage = false
}
remote {
enabled-transports = ["akka.remote.netty.tcp"]
netty.tcp {
hostname = "myhost"
port = 2561
maximum-frame-size = 256000b
}
}
}
But then I have another program that needs to access other configuration settings from my application.conf. But I need to ignore the akka config settings. So I've tried the following for the second program:
include "application"
akka {}
But the akka settings from application.conf are still being applied. I know this because I get a bind exception on the akka port eventhough there should be no remote akka in my second app
What is the best way for me clear/ignore the akka config settings from my application.conf?
Let's say you want to override the akka.remote.netty.tcp.port in your another.conf, you simply
include "application.conf"
akka.remote.netty.tcp.port = 2562
It will override the netty tcp port while leaving the rest unchanged and inherited
After some experimentation, the best bet IMO is to factor out your application.conf so that your common settings are in their own conf files.
So you might put the common settings in a common-settings.conf and then in your Akka application you would have
include "common-settings"
akka {
// Akka settings here
}
And the other modules that need common-settings can just, in their application.conf:
include "common-settings"
This may work better with a multi-module build. If they're the same, e.g., sbt module, then you'll probably replace the canonical application.conf with akka-application.conf and other-application.conf and point your ActorSystem setup code to akka-application.conf instead of application.conf (which probably shouldn't exist in this scenario, as you'd want bare ConfigFactory.load() calls to fail very quickly (the alternative here is to have different programs fighting over who owns application.conf).
The issue you are facing is actually by design. From the HOCON first page documentation:
Duplicate keys are allowed; later values override earlier, except for object-valued keys where the two objects are merged recursively
Therefore, when you add in your second file an akka {} it is just being merged, and not overwritten.
As I can see it, you have 2 options.
Copying the configuration, and override all properties with the one you actually want. That means, that in the second program, you can add:
akka {
actor {
provider = "new value"
warn-about-java-serializer-usage = false
}
remote {
enabled-transports = ["completely new values"]
netty.tcp {
hostname = "etc..."
port = 2561
maximum-frame-size = 256000b
}
}
}
The other option, which I like less, is to overwrite the object called akka. For doing that, you need to assign to it something that is not an object. Otherwise it will just be merged. For instance, if you add the the second program akka=4, so it will completely remove all of the other values. But! In this case, you have your program to deal with those properties to be missing. That means, that somewhere in your code you will have to write something like (don't forget that config throws on missing):
Try(config.getString("akka.actor.provider")).getOrElse(Do something here)
You have to do that because now akka is a string, and you cannot look into that as an object.
I have a spark application code written in Scala that runs a series of Spark-SQL statements. These results are calculated by calling an action 'Count' in the end against the final dataframe. I would like to know what is the best way to do logging from within a Spark-scala application job? Since all the dataframes (around 20) in number are computed using a single action in the end, what are my options when it comes to logging the outputs/sequence/success of some statements.
Question is little generic in nature. Since spark works on lazy evaluation, the execeution plan is decided by spark and I want to know till what point application statements ran successfully and what were the intermediate results at that stage.
The intention here being to monitor the long running task and see till which point it was fine and where the the problems creeped in.
If we try to put logging before/after transformations then it gets printed when code is read. So, the logging has to be done with custom messages during the actual execution (calling the action in the end of the scala code). If I try to put count/take/first etc in between the code then the execution of job slows down a lot.
I understand the problem that you are facing. Let me put out a simple solution for this.
You need to make use of org.apache.log4j.Logger. Use following lines of code to generate logger messages.
org.apache.log4j.Logger logger = org.apache.log4j.Logger.getRootLogger();
logger.error(errorMessage);
logger.info(infoMessage);
logger.debug(debugMessage);
Now, in order to redirect these messages to a log file, you need to create a log4j property file with below contents.
# Root logger option
# Set everything to be logged to the console
log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
# Settings to quiet third party logs that are too verbose
log4j.logger.org.eclipse.jetty=OFF
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=OFF
log4j.logger.org.spark-project.jetty.servlet.ServletHandler=OFF
log4j.logger.org.spark-project.jetty.server=OFF
log4j.logger.org.spark-project.jetty=OFF
log4j.category.org.spark_project.jetty=OFF
log4j.logger.Remoting=OFF
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR
# Setting properties to have logger logs in local file system
log4j.appender.rolling=org.apache.log4j.RollingFileAppender
log4j.appender.rolling.encoding=UTF-8
log4j.appender.rolling.layout=org.apache.log4j.PatternLayout
log4j.appender.rolling.layout.conversionPattern=[%d] %p %m (%c)%n
log4j.appender.rolling.maxBackupIndex=5
log4j.appender.rolling.maxFileSize=50MB
log4j.logger.org.apache.spark=OFF
log4j.logger.org.spark-project=OFF
log4j.logger.org.apache.hadoop=OFF
log4j.logger.io.netty=OFF
log4j.logger.org.apache.zookeeper=OFF
log4j.rootLogger=INFO, rolling
log4j.appender.rolling.file=/tmp/logs/application.log
You can name the log file in the last statement. Ensure the folders at every node with appropriate permissions.
Now, we need to pass the configurations while submitting the spark job as follows.
--conf spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties --conf spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
And,
--files "location of spark-log4j.properties file"
Hope this helps!
you can use log4j lib from maven
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>${log4j.version}</version>
</dependency>
For logging, first you need to create a logger object and then you can do logging at different log levels like info, error, warning. Below is the example of logging info in spark scala using log4j:
import org.apache.logging.log4j.LogManager
val logger = LogManager.getLogger(this.getClass.getName)
logger.info("logging message")
So, to add info at some points you can use logger.info("logging message") at that point.
I'm a bit confused about the logging system in Play.
Without importing any logging library, I added this to my code:
Logger.debug("Data is: " + data)
It didn't cause a compilation error but at the same time, it didn't print anything in the terminal window where I started the activator(where I typed activator run).
After looking here https://www.playframework.com/documentation/2.5.x/ScalaLogging, I also tried:
val logger = Logger(this.getClass)
logger.debug("Data is: " + data)
However, again nothing is printed.
Why is this happening?
there is few log level You can set in application.conf according documentation.
# Root logger:
logger.root=ERROR
# Logger used by the framework:
logger.play=INFO
# Logger provided to your application:
logger.application=DEBUG
# Logger for a third party library
logger.org.springframework=INFO
Try set log level to debug in Your application.conf
Currently there is an issue in the default configuration for the logger in DEV mode https://github.com/playframework/playframework/issues/5842
The default level for applications is INFO so debug messages are not shown.
While that issue is not fixed the workaround is to override logback.xml
Following the example in https://www.playframework.com/documentation/2.5.x/SettingsLogger that defines the log level for application as DEBUG
1.Add below in build.sbt:
libraryDependencies += "com.typesafe.scala-logging" %% "scala-logging" % "3.9.0"
2.Import following in your controller:
import com.typesafe.scalalogging.Logger
3.Use
private val logger = Logger(this.getClass)
logger.warn("your messages in here.")
I have a particular JUnit test which processes a big data file and when the logging level is left on TRACE, it kills Eclipse - something to do with the console handling, which is not relevant to this question.
I often switch between running all my tests using m2e, in which case I don't need any debugging log output, and running individual tests, where I want often want to see the TRACE output.
To avoid the necessity of editing my log4j2.xml config every time, I coded the log4j config to increase the logging level to INFO in this particular test, like this from programmatically-change-log-level-in-log4j2:
#Before
public void beforeTest() {
LoggerContext ctx = (LoggerContext) LogManager.getContext(false);
Configuration config = ctx.getConfiguration();
LoggerConfig loggerConfig = config.getLoggerConfig(
LogManager.ROOT_LOGGER_NAME);
initialLogLevel = loggerConfig.getLevel();
loggerConfig.setLevel(Level.INFO);
}
But it has no effect.
If the "ROOT_LOGGER" that I am manipulating here represents the same logger as the <root> in my log4j2.xml, then this is not going to work, is it? I need to override all the other loggers, or shut it down completely, but how?
Could it be influenced by my use of slf4j as the log4j2 wrapper in all of my other classes?
I have tried getting hold of the Appenders and using append.stop() but that doesn't work.
You can put a log4j2 config file in src/test/resources/ directory. During unit tests that file will be used.
I have a class with a custom logger. Here's a trivial example:
package models
import play.Logger
object AModel {
val log = Logger.of("amodel")
def aMethod() {
if (! log.isInfoEnabled) log.error("Can't log info...")
log.info("Logging aMethod in AModel")
}
}
and then we'll enable this logger in application.conf:
logger.amodel=DEBUG
and in development (Play console, use run) this logger does indeed log. But in production, once we hit the message
[info] play - Application started (Prod)
loggers defined like the above logger fail to log any further and instead we go through the error branch. It seems their log level has been changed to ERROR.
Is there anyway to correct this undesirable state of affairs? Is there special configuration for production logs?
edit
Play's handling of logs in production is a source of difficulty to more than a few people... https://github.com/playframework/playframework/issues/1186
For some reason it ships its own logger.xml which overrides application.conf.