I've problem with running big number of concurrent Jobs in my Eclipse RCP application. When I start 100 Jobs (like the one bellow), they execute normally but after they are finished they are still visible in Progress View for about 10 seconds. I would like Progress View to clear Jobs immediately after they are finished. If I start less Jobs (eg. 10) Progress View refreshes it self immediately.
class MyJob extends Job
{
public MyJob(String name) {
super(name);
}
public IStatus run(IProgressMonitor monitor){
//SOME EXPENSIVE COMPUTATIONS
BigInteger bigint = new BigInteger("0");
for ( int i=0 ; i<100 ; i++ ) {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
bigint = bigint.add(new BigInteger(Integer.toString(i)));
}
monitor.done();
System.out.println(new Random().nextInt());
return new Status(IStatus.OK, "foo", "OK");
}
}
each job is scheduled as a separate thread and in case there are lots of them locking/synchronization/thread management mechanisms take some overhead. for example some internal Eclipse Jobs code explicitly puts progress-bar-feedback mechanism into sleep for 250ms. such solutions probably pile up into long delays.
anyway having 100 entries in Progress View is not user firendly. I'd suggest grouping those jobs with Job.setProgressGroup(). perhaps having less entries in the view will cover up slow responsiveness.
In a case like this, I'd say it's better to allocate your own Threads than to use Jobs. Your 100 Jobs are going to start using the Job Threads that would be used to do Workbench maintenance. Allocating your own Threads means the Jobs Thread pool doesn't grow to the max value and stay there.
I'd have one Job that starts your Threads and reports progress to the Progress View. You could use SubMonitor to split the main IProgressMonitor into 100 so each of the Thread/Runnable pairs you start can report progress back to the main Job.
Related
I need to achieve the ability to monitor and be able to cancel an ALREADY RUNNING job on queue.
There's a lot of answers about deleting QUEUED jobs, but not on an already running one.
This is the situation: I have a "job", which consists of HUNDREDS OF THOUSANDS rows on a database, that need to be queried ONE BY ONE against a web service.
Every row needs to be picked up, queried against a web service, stored the response and its status updated.
I had that already working as a Command (launching from / outputting to console), but now I need to implement queues in order to allow piling up more jobs from more users.
So far I've seen Horizon (which doesn't runs on Windows due to missing process control libs). However, in some demos seen around it lacks (I believe) a couple things I need:
Dynamically configurable timeout (the whole job may take more than 12 hours, depending on the number of rows to process on the selected job)
Ability to CANCEL an ALREADY RUNNING job.
I also considered the option to generate EACH REQUEST as a new job instead of seeing a "job" as the whole collection of rows (this would overcome the timeout thing), but that would give me a Horizon "pending jobs" list of hundreds of thousands of records per job, and that would kill the browser (I know Redis can handle this without itching at all). Further, I guess is not possible to cancel "all jobs belonging to X tag".
I've been thinking about hitting an API route, fire the job and decouple it from the app, but I'm seeing that this requires forking processes.
For the ability to cancel, I would implement a database with job_id, and when the user hits an API to cancel a job, I'd mark it as "halted". On every loop I would check its status and if it finds "halted" then kill itself.
If I've missed any aspect just holler and I'll add it or clarify about it.
So I'm asking for an advice here since I'm new to Laravel: how could I achieve this?
So I finally came up with this (a bit clunky) solution:
In Controller:
public function cancelJob()
{
$jobs = DB::table('jobs')->get();
# I could use a specific ID and user owner filter, etc.
foreach ($jobs as $job) {
DB::table('jobs')->delete($job->id);
}
# This is a file that... well, it's self explaining
touch(base_path(config('files.halt_process_signal')));
return "Job cancelled - It will stop soon";
}
In job class (inside model::chunk() function)
# CHECK FOR HALT SIGNAL AND [OPTIONALLY] STOP THE PROCESS
if ($this->service->shouldHaltProcess()) {
# build stats, do some cleanup, log, etc...
$this->halted = true;
$this->service->stopProcess();
# This FALSE is what it makes the chunk() method to stop looping
return false;
}
In service class:
/**
* Checks the existence of the 'Halt Process Signal' file
*
* #return bool
*/
public function shouldHaltProcess() :bool
{
return file_exists($this->config['files.halt_process_signal']);
}
/**
* Stop the batch process
*
* #return void
*/
public function stopProcess() :void
{
logger()->info("=== HALT PROCESS SIGNAL FOUND - STOPPING THE PROCESS ===");
$this->deleteHaltProcessSignalFile();
return ;
}
It doesn't looks quite elegant, but it works.
I've surfed the whole web and many goes for Horizon or other tools that doesn't fit my case.
If anyone has a better way to achieve this, it's welcome to share.
Laravel queue have 3 important config:
1. retry_after
2. timeout
3. tries
See more: https://laravel.com/docs/5.8/queues
Dynamically configurable timeout (the whole job may take more than 12
hours, depending on the number of rows to process on the selected job)
I think you can config timeout + retry_after about 24h.
Ability to CANCEL an ALREADY RUNNING job.
Delete job in jobs table
Delete process by process id in your server
Hope it help you :)
How do I create a Quartz Scheduler job that terminates automatically after given amount of time (if running job takes too much time)?
A Quartz scheduler has no built-in functionality to interrupt a job by itself after a given amount of time.
If you dont want to interrupt Jobs (see the interface InterruptableJob) manually (for example with rmi), you could easily establish such a automatically termination.
Either:
When starting a scheduler, fork a deamon-thread that runs periodically and checks whether some of the currently running jobs must be interrupted. For Example you could use a JobDataMap to store the maximum execution time on a per job instance basis.
Each Job could control its maximum execution time in a similar way.
To stop a job from the inside of the job itself the easiest way is to throw an exception after a specific amount of time. For example:
public class MyJob : IJob
{
Timer _t;
public MyJob()
{
TimeSpan maxRunningTime = TimeSpan.FromMinutes(1);
_t = new Timer(delegate { throw new JobExecutionException("took to long"); }, null, (int) maxRunningTime.TotalMilliseconds,
-1);
}
public void Execute(IJobExecutionContext context)
{
// do your word
// destroy T before leaving
_t = null;
}
}
Hope it helps :)
This code does what I need, with the exception that there is a quick switch between the normal cursor, and the busy cursor happening extremely fast.
/* this code is run inside the createPartControl(Composite parent) method of a ViewPart */
Job job = new Job("refreshing")
{
#Override
protected IStatus run(IProgressMonitor monitor)
{
while (data.isReading())
{
Display.getDefault().syncExec(new Runnable()
{
#Override
public void run()
{
treeViewer.setInput(DataView.this.dataModel.getArray());
}
});
}
return Status.OK_STATUS;
}
};
job.schedule();
So is there a way of disabling the busy cursor of a Job in eclipse?
Also, could this happen because the Job is called in a GUI class?
Call
job.setSystem(true);
before you schedule the job. From the JavaDoc:
Sets whether or not this job is a system job. System jobs are
typically not revealed to users in any UI presentation of jobs. Other
than their UI presentation, system jobs act exactly like other jobs.
If this value is not explicitly set, jobs are treated as non-system
jobs. This method must be called before the job is scheduled.
However I think the busy cursor is coming from the BusyIndicator.showWhile call in AbstractTreeViewer.createChildren which the tree setInput will call. I don't think you can do anything about.
i have a long-running non-UI Job with a number of sub-tasks. each sub-task updates caption for the Job's progress entry in the Progress View via IProgressMonitor.subTask(some-caption). the Job goes through three sub-tasks before it contributes to the UI in its fourth sub-task (via Display.syncExec() of course) by creating a new IEditorPart instance. the problem is that those initial three sub-tasks are never reflected in the Job's progress entry in the Progress View. in fact, the Job progress entry never actually appears in the Progress View until fourth sub-task. this, of course, creates a problem for a user who is left to wonder if anything is even happening. my guess is that once the Job starts, the vm thread "broker" (sorry, not well versed in this matter) never gets a chance to switch from a non-UI Job thread to the UI thread until the former actually makes a call to the latter via Display.syncExec(). my only solution at this point is to put Job thread to sleep for about half a second right after each sub-task kicks in, thereby allowing UI thread to take over long enough to update Progress View. and while this has worked most of the time, i still get instances here and there where the Porgress View still remains empty until fourth sub-task, leading me to believe that my solution is not of a robust kind. hence, i am looking for a robust one. thank you for your time!
EDIT: example code
Job job = new Job("multipleTasks") {
public IStatus run(IProgressMonitor monitor) {
monitor.beginTask("multiple tasks", IProgressMonitor.UNKNOWN);
monitor.subTask("task1");
monitor.worked(1);
try { Thread.sleep(1000); } catch (Exception e) { }
monitor.subTask("task2");
monitor.worked(1);
try { Thread.sleep(1000); } catch (Exception e) { }
monitor.subTask("task3");
monitor.worked(1);
try { Thread.sleep(1000); } catch (Exception e) { }
monitor.done();
return Status.OK_STATUS;
}
};
job.schedule();
We have a fairly simple program that's used for creating backups. I'm attempting to parallelize it but am getting an OutOfMemoryException within an AggregateException. Some of the source folders are quite large, and the program doesn't crash for about 40 minutes after it starts. I don't know where to start looking so the below code is a near exact dump of all code the code sans directory structure and Exception logging code. Any advice as to where to start looking?
using System;
using System.Diagnostics;
using System.IO;
using System.Threading.Tasks;
namespace SelfBackup
{
class Program
{
static readonly string[] saSrc = {
"\\src\\dir1\\",
//...
"\\src\\dirN\\", //this folder is over 6 GB
};
static readonly string[] saDest = {
"\\dest\\dir1\\",
//...
"\\dest\\dirN\\",
};
static void Main(string[] args)
{
Parallel.For(0, saDest.Length, i =>
{
try
{
if (Directory.Exists(sDest))
{
//Delete directory first so old stuff gets cleaned up
Directory.Delete(sDest, true);
}
//recursive function
clsCopyDirectory.copyDirectory(saSrc[i], sDest);
}
catch (Exception e)
{
//standard error logging
CL.EmailError();
}
});
}
}
///////////////////////////////////////
using System.IO;
using System.Threading.Tasks;
namespace SelfBackup
{
static class clsCopyDirectory
{
static public void copyDirectory(string Src, string Dst)
{
Directory.CreateDirectory(Dst);
/* Copy all the files in the folder
If and when .NET 4.0 is installed, change
Directory.GetFiles to Directory.Enumerate files for
slightly better performance.*/
Parallel.ForEach<string>(Directory.GetFiles(Src), file =>
{
/* An exception thrown here may be arbitrarily deep into
this recursive function there's also a good chance that
if one copy fails here, so too will other files in the
same directory, so we don't want to spam out hundreds of
error e-mails but we don't want to abort all together.
Instead, the best solution is probably to throw back up
to the original caller of copy directory an move on to
the next Src/Dst pair by not catching any possible
exception here.*/
File.Copy(file, //src
Path.Combine(Dst, Path.GetFileName(file)), //dest
true);//bool overwrite
});
//Call this function again for every directory in the folder.
Parallel.ForEach(Directory.GetDirectories(Src), dir =>
{
copyDirectory(dir, Path.Combine(Dst, Path.GetFileName(dir)));
});
}
}
The Threads debug window shows 417 Worker threads at the time of the exception.
EDIT: The copying is from one server to another. I'm now trying to run the code with the last Paralell.ForEach changed to a regular foreach.
Making a few guesses here as I haven't yet had feedback from the comment to your question.
I am guessing that the large amount of worker threads is happening here as actions (an action being the unit of work carried out on the parallel foreach) are taking longer than a specified amount of time, so the underlying ThreadPool is growing the number of threads. This will happen as the ThreadPool follows an algorithm of growing the pool so that new tasks are not blocked by existing long running tasks e.g. if all my current threads have been busy for half a second, I'll start adding more threads to the pool. However, you are going to get into trouble if all tasks are long-running and new tasks that you add are going to make existing tasks run even longer. This is why you are probably seeing a large number of worker threads - possibly because of disk thrashing or slow network IO (if networked drives are involved).
I am also guessing that files are being copied from one disk to another, or they are being copied from one location to another on the same disk. In this case, adding threads to the problem is not going to help out much. The source and destination disks only have one set of heads, so trying to make them do multiple things at once is likely to actually slow things down:
The disk heads will be lurching all over the place.
Your disk\OS caches may be frequently invalidated.
This may not be a great problem for parallelization.
Update
In answer to your comment, if you are getting a speed-up using multiple threads on smaller datasets, then you could experiment with lowering the maximum number of threads used in your parallel foreach, e.g.
ParallelOptions options = new ParallelOptions { MaxDegreeOfParallelism = 2 };
Parallel.ForEach(Directory.GetFiles(Src), options, file =>
{
//Do stuff
});
But please do bear in mind that disk thrashing may negate any benefits from parallelization in the general case. Play about with it and measure your results.