I am trying to do some actions at the Driver side in Spark while an application is running. The Driver needs to know the tasks progress before making any decision. I know that tasks progress can be accessed within each executor or task from RecordReader class by calling getProgress().
The question is, how can I let the Driver call or have an access to getProgress() method of each task? I thought about using broadcast variables, but I don't know how the Driver would distinguish between different tasks.
Note that I am not looking for results displayed in Spark UI.
Any help is appreciated!
One way to do this is to send the progress from each Executor thread to a listening thread in the Driver. This has to be a seperate thread as the main thread is blocked while the action is in progress.
Related
I am using quartz to schedule jobs to be executed daily as a part of a much larger web application. However, after a couple of days, the administrator would like to stop the execution of a particular job (maybe because it is no longer needed). How do I go about doing this? I read the api docs for the Scheduler and it has a method called interrupt(JobKey jobkey) but that method would work only with the same instance of the scheduler that was used to schedule the job.
interrupt(JobKey jobKey)
Request the interruption, within this Scheduler instance, of all
currently executing instances of the identified Job, which must be an
implementor of the InterruptableJob interface.
Is there anyway of getting the instance of an existing scheduler? Or maybe use singletons?
Should definitely use a singleton instance of your scheduler. I recommend the use of an IoC container to manage this in a clean and efficient way.
I just started learning about Play framework lately and really like it so far.
There is just one thing that isn't clear to me. I'm a Java developer, with an blocking
and multi-threaded way of thinking so async programming is a little new to me.
So, Play framework uses an asynchronous WS API which doesn't block thread that calls it, nor
it blocks or spawns any other thread. When doing some async programming usually you have
to spawn a new thread and then make a WS call so you don't block your main thread. So my question here is how does Play's WS API manages to do async call to a web service without blocking current thread and without spawning a new one? Does it fire a request and then the main thread every once in a little checks if there is a Response available in the Future object? I'm aware of how to use it but I want to know what's the deal "under the hood". How would it be implemented in Scala? A simple example if possible would be a nice help.
Thanks in advance and best regards!
Under the hood play uses async-http-client which is based on NIO. A request will be dispatched and when the server responds, a callback (in this case the completion of the future) will be executed on a thread from the threadpool. This way no thread has to be blocked.
Play uses an "execution context", which is typically a thread pool, as described in the documentation:
It’s important to understand which thread code runs on with futures.
In the two code blocks above, there is an import on Plays default
execution context. This is an implicit parameter that gets passed to
all methods on the future API that accept callbacks. The execution
context will often be equivalent to a thread pool, though not
necessarily.
So no new threads are spawned, but only because there is already a pool of threads available for such work.
I want to put an Abort Label or button below...the processing message is shown in ZK. or is there any way to load my custom components instead of the default Processing. message in ZK.
Want will happen if do abort while processing is it ideal to do that, I want to do that anyways as few times my application sleeps while loading
ZK is built on Servlets. When the busy button is shown on the ui awaiting the ajax response then the servlet thread is doing long running work on the server. Perhaps it would be possible to send another thread to interrupt the first thread but really that is high risk as all the servlet threads can end up doing long running work and no new ones will be available to cancel them.
The best solution is that long running work should not be on the servlet thread but handed off to a background thread or message queue. See zk-asynchronous-ui-updates-and-background-processing. In that example the work is offloaded to a java.util.concurrent.ExecutorService which has an API to cancel the work.
Note that cancelling of a working thread uses interruption and it is not guaranteed that code which doing the work will respond to the interrupt properly such that it is actually cancellable. The answers on the thread cancel with executor service outline some of the issues and you should test whether the work you are doing can be interrupted using the API.
I have a GTK+ application with a GtkTextBuffer. It uses several worker threads which perform some lengthy operations. Those threads can report their progress in a textual format. I want to show details on how a worker thread is performing. Can you please tell me how to receive input from a worker thread?
I thought that I can emit a signal "insert-text" from the worker thread, but cannot find a relevant example while googling. Any help is much appreciated.
You could use GAsyncQueue for messaging between your UI and your workers thread, it is a very finished API for communication.
Classic quick and dirty: Write your status updates to a linked list (protected by some sync primitive), have the GTK main loop poll it via a timer.
I have two threads running one id UI and other is worker thread. Worker thread continuously reads on port for some data from server, when appears I need to update my webview in UI thread. Worker thread again continues to read and never ends. Plz suggest how to accomplish this??
it may be likely as 'signals in C++' which causes a method in called thread to be invoked..!!
I tried: 1)As running worker thread on UI (runOnUIThread)may degrade UI webview performance, and if I put the thread on sleep it may miss data appeared at port when sleeping..(i m not sure!)
2)using Handler, I hav to specify time before calling thread again n again which may cause to miss data if appeared on port as like sleep().
3)Having a separate thread in same class,it gives: any other thread cant update view of main UI thread.
Plz help.. :(
Maybe I am missing something, but it seems like creating a Handler object and then using the post() method is what you need: http://developer.android.com/reference/android/os/Handler.html#post%28java.lang.Runnable%29
I had a similar problem. I have a UI thread and a few threads that do backend serverices. I use the observer Pattern to have the services update the UI and to do so I have the update method in a new thread and call runOnUiThread() to have it update the UI. It seems to work well for something quick. If you want something better than a quick band-aid I would suggest reading this android threading guide from Google and trying to incorporate it with your design.