Notice that when the Rundeck job triggers a timed out due to running too long, it fails to remove the temp file from my remote window node. Due to this action, the Temp folder has an unnecessary amount of dispatch temp files in the folder. I wondered if their a config setting to remove said files if the job triggers a timed out within Rundeck.
Rundeck is running on Community 3.4.10
There is no specific option for that, a good idea you can do is to run a periodic job that cleans that directory.
Related
I am new to spring-batch, got few questions:-
I have got a question about the restart. As per documentation, the restart feature is enabled by default. What I am not clear is do I need to do any extra code for a restart? If so, I am thinking of adding a scheduled job that looks at failed processes and restarts them?
I understand spring-batch-admin is deprecated. However, we cannot use spring-cloud-data-flow right now. Is there any other alternative to monitor and restart jobs on demand?
The restart that you mention only means if a job is restartable or not .It doesn't mean Spring Batch will help you to restart the failed job automatically.
Instead, it provides the following building blocks for developers for achieving this task on their own :
JobExplorer to find out the id of the job execution that you want to restart
JobOperator to restart a job execution given a job execution id
Also , a restartable job can only be restarted if its status is FAILED. So if you want to restart a running job that was stop running because of the server breakdown , you have to first find out this running job and update its job execution status and all of its task execution status to FAILED first in order to restart it. (See this for more information). One of the solution is to implement a SmartLifecycle which use the above building blocks to achieve this goal.
Whenever we restart a failed task, it will ALWAYS pick up the config it had at the time of the failure, and run with that.. and THEN it picks up the new config.. and runs that as well.
We have connect jobs that we pause, update config, and then resume. This works fine, unless the task has failed.
If we restart a failed task, even if the connector has an updated config, the task will launch with the old config.. run to completion/failure.. then a new task will be launched with the new config.
This can cause various data/etc issues.. if you really don't want that old task to run with that config.
Any ideas how to restart a connector with a failed task.. with a new config.. and NOT have the old config get invoked?
(running Kafka v2.5, btw)
I don't know if it would make sense for the task to pick up the latest config.
For instance, let's assume that your connector fires up 10 distinct tasks and 1 of them fails. It won't make sense to have the remaining 9 tasks of the connector running with the older config while the failed task runs the newest config once it is restarted.
I would say that in cases you want to use a new/different configuration file when a task fails, it might make more sense to restart the connector and not the individual task(s):
POST /connectors/connector-name/restart HTTP/1.1
I was having this problem and managed to "fix" this by a bit of randomness.
I increased the number of Tasks in the connector and then reduced it again and it seemed to have picked up a new configuration.
Was really random.
I do know the restart did not work for me
I had developed a Job in Talend and built the job and automated to run the Windows Batch file from the below build
On the Execution of the Job Start Windows Batch file it will invoke the dimtableinsert job and then after it finishes it will invoke fact_dim_combine it is taking just minutes to run in the Talend Open Studio but when I invoke the batch file via the Task Scheduler it is taking hours for the process to finish
Time Taken
Manual -- 5 Minutes
Automation -- 4 hours (on invoking Windows batch file)
Can someone please tell me what is wrong with this Automation Process
The reason of the delay in the execution would be a latency issue. Talend might be installed in the same server where database instance is installed. And so whenever you execute the job in Talend, it will complete as expected. But the scheduler might be installed in the other server, when you call the job through scheduler, it would take some time to insert the data.
Make sure you scheduler and database instance is on the same server
Execute the job directly in the windows terminal and check if you have same issue
The easiest way to know what is taking so much time is to add some logs to your job.
First, add some tWarn at the start and finish of each of the subjobs (dimtableinsert and fact_dim_combine) to know which one is the longest.
Then add more logs before/after the components inside the jobs.
This way you should have a better idea of what is responsible for the slowdown (DB access, writing of some files, etc ...)
I have a scheduled job to run the following command:
copy 'C:\Users\tdjeilati\Desktop\RDP.rdg' '\\fil03\Dept_1Z\Public\my'
This copies a file to a remote server.
I have a job trigger -AtLogon.
When I log in to my PC, it runs the job.
When I retrieve that job with receive-job, I get the job got an an access is denied error:
But then I run the job by hand, and it works correctly! What gives?
I don't understand why it fails when running from the job trigger but works when I run it manually in powershell. I can only assume that the environment/permissions are different.
EDIT: One thing I noticed is that the job that runs from the jobtrigger doesn't have any childJobs, but the job that I start from command line has child jobs. Why should there be a difference?
The scheduled task may not be running under your user account. This could explain why it works when you manually start the job.
Verify that the task is running as a user with rights to the file and the remote share.
When spring batch DB failure happens or server is shut down, a spring batch job which was running at that time will be in a unknown started state.
In spring batch admin, we will not see an option to restart the job. Hence we are not able to resume the job.
How to restart the job from last successful commit?
The old discussions suggest that it had to be dealt manually by updating tables. I was manually able to update end time, status in batch step execution and batch job execution tables. Is it really the best option? It may not be practical to do that manually in a prod region.
As mentioned in the Aborting a Job section of the reference documentation, when a server failure happens, the job repository has no way to know that the process running the job died. Hence a manual intervention is required.
How to restart the job from last successful commit?
Change the status of the job to FAILED and restart the job instance, it should continue from where it left off.