In my workflows and activities, I’d like to log some messages for debugging purposes.
I saw the cadence.GetLogger(ctx).Info() function, but don’t know where to find the logs.
Go Client:
You can use the following in the workflow code:
cadence.GetLogger(ctx).Info(...)
In the activity code, you should use the following:
cadence.GetActivityLogger(ctx).Info(...)
By default, the logger will write to console, which may be sufficient for development purposes. However, you should log to a file if you need the logs in production, too. Here is how to setup your cadence worker to do it:
workerOptions := cadence.WorkerOptions{
Logger: myLogger,
}
worker := cadence.NewWorker(service, domain, taskList, workerOptions)
The Cadence client uses zap as the logging framework. You can create the zap logger and specify the log file path per your needs. Check out the zap documentation to learn more about configuring the logs.
Java Client
The Java client uses slf4j for logging. You can get the logger instance by calling Workflow.getLogger() and configure it in logback.xml as usual.
Related
I must be missing something obvious. I've got a couple of jobrunr jobs where i'm using the lambda enqueue format version 5.1.6. Like this:
JobId jobId = BackgroundJob.<MyService>enqueue(x -> x.doWork());
I would like to validate the plumbing and work in the jobs is executing via some integration tests with Spring, but don't see the options to run now, eager mode, etc? Thanks
You can't, I'm afraid.
You can mock the JobScheduler and capture the args. JobRunr itself is also tested very well so if you pass a job, you can rest assured it will be enqueued.
You could also put the pollIntervalInSeconds to 5 and use awaitility then to verify your job executed. There are many examples of this in the JobRunr repo.
The Play framework requires (by default) that you configure logging through a logback.xml file. I'd like to build my log appenders through code so I can fetch parameters at runtime (e.g. the graylog destination for the logs is fetched from the deployment environment, rather than baking it in statically through an XML file).
This sort of thing is fairly easy to achieve in Java (by overriding logging factories and the like), I wondered if the same were possible in Play.
Yes, you can configure logback programmatically, see: https://akhikhl.wordpress.com/2013/07/11/programmatic-configuration-of-slf4jlogback/
But I wouldn't recommend it. For starters it's a verbose API that isn't pleasant to work with. Beyond that, it generally nice for configuration to be declarative (even if it is in XML in this case).
For your usecase, Logback's XML does support variables which can come from System properties or Environment variables: https://logback.qos.ch/manual/configuration.html#definingProps
However, you probably want a different config across environments (no greylog locally). I think many projects do that by specifying the logback XML location as a system property at startup: https://logback.qos.ch/manual/configuration.html#configFileProperty
Alternatively, I suspect greylog has some method of watching a file to pickup your logging. That's what we do for picking up logs in Splunk in my team. We don't want to make a change to our code when someone reconfigures Splunk/Greylog.
The solution I used in the end was to use a logback contextlistener to populate the context with the parameters pulled from the environment. The listener can be added as follows to the logback.xml:
<contextListener class="LoggerStartup"/>
The LoggerStartup can then populate the context, which I achieved through AWS SSM (see the simplified code below).
class LoggerStartup extends ContextAwareBase with LoggerContextListener with LifeCycle {
override def start() = {
val context = getContext()
val graylogUrl = ... // Go get value from remote store
context.putProperty("GRAYLOG_URL", graylogUrl)
}
}
And then referenced this context variable in the logback file:
<appender name="GELF UDP APPENDER" class="me.moocar.logbackgelf.GelfUDPAppender">
<remoteHost>${GRAYLOG_URL}</remoteHost>
...
</appender>
What is the way to do message tracing for each request made to the JBoss Fuse 6.2 server? In my case most of the entry points are CXF REST service with the processing delegated to Camel routes in some cases. I would like to do end-to-end tracing with same message id that can correlate the request processing.
In my project, there was a similar requirement. Customer wanted to see all e2e log by executing grep command to system logs with a transaction id.
I used CXF interceptors and MDC logging capability for this as below:
Create a common CXF request and response interceptors. Add them to all your Camel's CXF Server/Client configuration
With your request interceptor, extract transaction id from request (or generte it yourself) then put it into MDC map. MDC is a thread local variable that log4j, slf4j,.. uses.
Print request, it'll have your transaction id as prefix thanks to MDC.
Dont forget to add your MDC key in logging format configuration
All logs you print until the end of operation with this transaction Id until the end.
If you're always using direct-vm, direct for routing then it wont be problem. However as you may know using seda, multi processing, etc. your execution is handled by other threads. Since MDC is thread local variable, you need to manually handle the trouble by transferring it.
With your response interceptor, log response message then clear MDC values.
If you're using CXF as client, you should use same interceptor approach to be able to print client request/reponses with transaction id.
See CXF-RS and MDC links as entry points
How do I configure how the format of the logging output?
E.g. the time format, thread name level etc.
How do I get the following?
17:15:00.154 Main INFO : Here is a log...
log4s is merely a wrapper for slf4j. So its purpose is to provide a logging facade, not an implementation. You'll have to decide which actual logger implementation to use. A popular choice is logback, which you would configure using a logback.xml file. See http://www.slf4j.org/manual.html#swapping and http://logback.qos.ch/manual/configuration.html.
I have been of late trying out apache spark. My question is more specific to trigger spark jobs. Here I had posted question on understanding spark jobs. After getting dirty on jobs I moved on to my requirement.
I have a REST end point where I expose API to trigger Jobs, I have used Spring4.0 for Rest Implementation. Now going ahead I thought of implementing Jobs as Service in Spring where I would submit Job programmatically, meaning when the endpoint is triggered, with given parameters I would trigger the job.
I have now few design options.
Similar to the below written job, I need to maintain several Jobs called by a Abstract Class may be JobScheduler .
/*Can this Code be abstracted from the application and written as
as a seperate job. Because my understanding is that the
Application code itself has to have the addJars embedded
which internally sparkContext takes care.*/
SparkConf sparkConf = new SparkConf().setAppName("MyApp").setJars(
new String[] { "/path/to/jar/submit/cluster" })
.setMaster("/url/of/master/node");
sparkConf.setSparkHome("/path/to/spark/");
sparkConf.set("spark.scheduler.mode", "FAIR");
JavaSparkContext sc = new JavaSparkContext(sparkConf);
sc.setLocalProperty("spark.scheduler.pool", "test");
// Application with Algorithm , transformations
extending above point have multiple versions of jobs handled by service.
Or else use an Spark Job Server to do this.
Firstly, I would like to know what is the best solution in this case, execution wise and also scaling wise.
Note : I am using a standalone cluster from spark.
kindly help.
It turns out Spark has a hidden REST API to submit a job, check status and kill.
Check out full example here: http://arturmkrtchyan.com/apache-spark-hidden-rest-api
Just use the Spark JobServer
https://github.com/spark-jobserver/spark-jobserver
There are a lot of things to consider with making a service, and the Spark JobServer has most of them covered already. If you find things that aren't good enough, it should be easy to make a request and add code to their system rather than reinventing it from scratch
Livy is an open source REST interface for interacting with Apache Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in Apache Hadoop YARN.
Here is a good client that you might find helpful: https://github.com/ywilkof/spark-jobs-rest-client
Edit: this answer was given in 2015. There are options like Livy available now.
Even I had this requirement I could do it using Livy Server, as one of the contributor Josemy mentioned. Following are the steps I took, hope it helps somebody:
Download livy zip from https://livy.apache.org/download/
Follow instructions: https://livy.apache.org/get-started/
Upload the zip to a client.
Unzip the file
Check for the following two parameters if doesn't exists, create with right path
export SPARK_HOME=/opt/spark
export HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
Enable 8998 port on the client
Update $LIVY_HOME/conf/livy.conf with master details any other stuff needed
Note: Template are there in $LIVY_HOME/conf
Eg. livy.file.local-dir-whitelist = /home/folder-where-the-jar-will-be-kept/
Run the server
$LIVY_HOME/bin/livy-server start
Stop the server
$LIVY_HOME/bin/livy-server stop
UI: <client-ip>:8998/ui/
Submitting job:POST : http://<your client ip goes here>:8998/batches
{
"className" : "<ur class name will come here with package name>",
"file" : "your jar location",
"args" : ["arg1", "arg2", "arg3" ]
}