I am trying to run a child job in tLoop. The child job connects to salesforce and downloads "Account" object to local SQL Server table. There are problems with connection to Salesforce, it takes few attempts to connect. Hence, I put the connection stuff in child job and now trying to call the child job in a loop. Below is the image of my parent job.
As you can see in image the tRunJob_1 has error because of Salesforce connection problem in child job. This is correct behaviour.
The setRetryConnect that is connected to OnComponentError has this code: context.retryConnect = true;
The setRetryConnect that is connected to OnComponentOk has this code: context.retryConnect = false;
So, I am tripping this context variable depending on whether child job succeeds or fails.
My tLoop looks as below:
I want the tLoop to run as many times till the condition remains true. That is till the time it continues to error out. However, it just iterates once and then stops. Could anyone please let me know what correction need to be done here to make the tLoop work?
I couldn't re-pro your issue with SalesForce but by looking at your job what I feel is that when you say - "it just iterates once and then stops" is the expected behavior.
As per your job flow after the tRunJob you are using OnComponentOk/OnComponentError trigger which would process and stop the job run as it would have completed the job execution. What it would have ideal was to keep everything in a subjob post tLoop so that it will iterate till the condition is met.
Sample job for explanation -
Here used tSetGlobalVar to define a global variable (in place of your context variable). Then use the globalMap variable as ((Boolean)globalMap.get("tLoop")) in your "Condition" for the tLoop.
And then finally run some code in the tJava component that does something and conditionally sets the global variable to false to mark the ending of loop.
tRunJob provides an Return Code ((Integer)globalMap.get("tRunJob_1_CHILD_RETURN_CODE"))
If you're running your child Job a number of times and want your Job to exit with non-Zero if one of these iterations fails, then after each iteration, you should test this return code and store it in your own globalMap Object if it is non-zero
int returnCode = ((Integer)globalMap.get("tRunJob_1_CHILD_RETURN_CODE"));
if (returnCode > 0) {
globalMap.put("tLoop", false);
}
else {
System.out.println(returnCode);
};
Found the answer myself, posting it here so that it may help others. It appears like OnComponentError breaks the tLoop. Disabled the OnComponentError flow and un-checked the 'Die on Child Error' checkbox in tRunJob.
The tLoop remains as it is. No changes here.
The retryConnect will use the below code. It uses CHILD_RETURN_CODE to check whether the child job threw error. In case of success, its value is 0. I am tripping the variable when the child job succeeds, so the loop will stop. As you can see, the tLoop shows 2 iterations, it is working as expected now. Thanks.
Related
I want to store process times in my model for each agent of type Box. For this purpose I have created a Java class ProcessData.
My problem is that I always get NullPointerExceptions. I don't know if it's because of my code or the functionality of the batch block.
When the wait block reaches a certain size, all agents of type Box are routed to the batch block. The batch agent then waits in the seize block for staff member. If the agent is then seized, each agent of type Box contained in the batch agent should then get a start processing time. This is my code at enter of the delay block: box.getProcessData().add(new ProcessData(date(), duration()));. Using a constructor, I add the start time and duration as new variables to a new ProcessData entry.
To determine the end time of each Box agent I have the following code at unbatch block entry: box.getProcessData().getLast().setEnd(date());
Since I have a NullPointerException I can't tell exactly where it came from. My guess is that I have a bug with addressing the code: should I use agent. (from batch) or box. (name of original agent before batch)? Or is it not possible to give the batch agent properties that are inherited by the box agent?
First error is at unbatch, so the second code from above and second error is the following public void onExit( Box batch, Box agent ) { _unbatch_onExit_xjal( this, batch, agent ); }.
Following the screenshot of my error:
The problem you have with this model first,is that you are seizing a resource for the batch without releasing that resource... you need to release before unbatch
your error nevertheless is related to some code that you wrote in the on exit action of the unbatch block that you are unfortunately not telling us
But to discover what is null, check with a traceln() the value of the variables that are present in the on exit section of the unbatch in order to print on the screen what is null
I have the model which I posted before on Stack. I am currently running the iterations through 5 Flow Chart blocks contain enter block and service block. when agent fill service block 5 in flow chart 5, the exit block should start to fill block one and so on. I have used While infinite loop to loop between the five flow chart blocks but it isn't working.
while(true)
{
for (Curing_Drying currProcess : collection) {
if (currProcess.allowedDay == (int)time(DAY)) {
currProcess.enter.take(agent);
}
}
if (queue10.size() <= Throughtput1){
break;
}
}
Image for further illustration 1
Image for further illustration 2
Wondering if someone can tell me what is wrong in the code.
Based on the description and the pictures provided, it isn't clear why the while loop is necessary. The On exit action is executed for each Agent arrival to the Exit block. It seems that the intention is to find the appropriate Curing_Drying block based on number of days since the model start time? If so, then just iterating through the collection is enough.
Also, it is generally a good practice to provide more meaningful names to collections. Using simply collection doesn't say anything about the contents and can get pretty confusing later on.
I want abort 2nd job activity if 1st job activity aborts due env. issue or manual abortion.
I gone through the triggering option but couldn't get.
Can anyone help me ?
OK, You can't abort a job without running it. So you would need to be able to invoke Job 2 with knowledge of Job 1 status (e.g. from activity variable Job1.$JobStatus). This could be used to cause Job 2 to abort.
The cleanest solution would be a before-job subroutine, which sets its return code to 0 if Job 1's status was DSJS.RUNOK or DSJS.RUNWARN (these are DataStage constants), or to a non-zero value if Job 1's status was DSJS.RUNFATAL or any other value. This is the cleanest approach because the before-job subroutine could write a message to the job log indicating precisely why the job was being aborted.
A less clean way would be to have a parameter in Job 2 of type, say, Integer (anything other than string), and set its value in the Job Activity using an expression such as If Job1.$JobStatus = DSJS.RUNFATAL Then "" Else 1 - setting a non-string parameter to "" will cause the job to abort with DSJE.PARAMBADVALUE error.
You will need to show us exactly what you did. The use of trigger, which I explained earlier, is the correct solution. If you did it correctly, the arrow on the design canvas should be red.
Alternatively you could change your sequence to bypass Job 2 entirely, and instead simply log a message that Job 2 was being bypassed because Job 1 aborted.
I am trying to write a code for one state to another state transition where the system will first store the initial model time and then will check a condition (used while loop). It will continue to run the loop until the condition is false and when the condition is false it will record the final model time. So, my main objective is to get the total time that while loop condition is true. The problem is, I don't know how to check the while loop every 1 time step, For example, I tried "wait (1);" in place of "???" section of the below code which is not correct. Can anyone please suggest how I can do this?
My transition code as below:
...
...
initialTime=time();
while ((thisPed.inState(walking) && thisPed.fieldOfVision.contains(pedX, pedY));
{
???
}
finalTime= time();
exposureTime = finalTime - initialTime;
...
...
you can't put while statements in a model that at the same time run with time steps... to do that you have many other ways..
For instance you can generate a transition that goes from that state to the same state (internally) and generate your code there every time step.
Another option is to use a conditional transition in the same way
BUt NOT a while loop
I need to achieve the ability to monitor and be able to cancel an ALREADY RUNNING job on queue.
There's a lot of answers about deleting QUEUED jobs, but not on an already running one.
This is the situation: I have a "job", which consists of HUNDREDS OF THOUSANDS rows on a database, that need to be queried ONE BY ONE against a web service.
Every row needs to be picked up, queried against a web service, stored the response and its status updated.
I had that already working as a Command (launching from / outputting to console), but now I need to implement queues in order to allow piling up more jobs from more users.
So far I've seen Horizon (which doesn't runs on Windows due to missing process control libs). However, in some demos seen around it lacks (I believe) a couple things I need:
Dynamically configurable timeout (the whole job may take more than 12 hours, depending on the number of rows to process on the selected job)
Ability to CANCEL an ALREADY RUNNING job.
I also considered the option to generate EACH REQUEST as a new job instead of seeing a "job" as the whole collection of rows (this would overcome the timeout thing), but that would give me a Horizon "pending jobs" list of hundreds of thousands of records per job, and that would kill the browser (I know Redis can handle this without itching at all). Further, I guess is not possible to cancel "all jobs belonging to X tag".
I've been thinking about hitting an API route, fire the job and decouple it from the app, but I'm seeing that this requires forking processes.
For the ability to cancel, I would implement a database with job_id, and when the user hits an API to cancel a job, I'd mark it as "halted". On every loop I would check its status and if it finds "halted" then kill itself.
If I've missed any aspect just holler and I'll add it or clarify about it.
So I'm asking for an advice here since I'm new to Laravel: how could I achieve this?
So I finally came up with this (a bit clunky) solution:
In Controller:
public function cancelJob()
{
$jobs = DB::table('jobs')->get();
# I could use a specific ID and user owner filter, etc.
foreach ($jobs as $job) {
DB::table('jobs')->delete($job->id);
}
# This is a file that... well, it's self explaining
touch(base_path(config('files.halt_process_signal')));
return "Job cancelled - It will stop soon";
}
In job class (inside model::chunk() function)
# CHECK FOR HALT SIGNAL AND [OPTIONALLY] STOP THE PROCESS
if ($this->service->shouldHaltProcess()) {
# build stats, do some cleanup, log, etc...
$this->halted = true;
$this->service->stopProcess();
# This FALSE is what it makes the chunk() method to stop looping
return false;
}
In service class:
/**
* Checks the existence of the 'Halt Process Signal' file
*
* #return bool
*/
public function shouldHaltProcess() :bool
{
return file_exists($this->config['files.halt_process_signal']);
}
/**
* Stop the batch process
*
* #return void
*/
public function stopProcess() :void
{
logger()->info("=== HALT PROCESS SIGNAL FOUND - STOPPING THE PROCESS ===");
$this->deleteHaltProcessSignalFile();
return ;
}
It doesn't looks quite elegant, but it works.
I've surfed the whole web and many goes for Horizon or other tools that doesn't fit my case.
If anyone has a better way to achieve this, it's welcome to share.
Laravel queue have 3 important config:
1. retry_after
2. timeout
3. tries
See more: https://laravel.com/docs/5.8/queues
Dynamically configurable timeout (the whole job may take more than 12
hours, depending on the number of rows to process on the selected job)
I think you can config timeout + retry_after about 24h.
Ability to CANCEL an ALREADY RUNNING job.
Delete job in jobs table
Delete process by process id in your server
Hope it help you :)