I'm using PBS Torque to run multiple jobs. The idea is simple, each job works on a shunk of data. The PBS_Torque job_script to launch a job is called run_appli.sh
Here is the simple code (code 1) to launch 10 jobs
for i in 1 2 3 4 5 6 7 9 10 do; qsub run_appli.sh ; done
Indeed, I can monitor the execution of each of those jobs using qstat (see the command below) and have elapsed time of each job.
watch -n1 -d `qstat`
However, I am interested by the overall elapsed time. That means the time starting from when I launched all the jobs (code 1) and when the last job finished its execution.
Does anyone have an idea on how to do this ?
If you know the job id of the first job, you can look at it's ctime (creation time, or the time it is queued). You can then check the end time for the last job's comp_time. The difference between the two would be the total time elapsed.
qstat -f $first_job_id | grep ctime # Shows the first job's queued time
qstat -f $last_job_id | grep comp_time # Shows the final job's completion time.
If the last job's isn't completed, then the running elapsed time would just be the current time - the first job's queue time.
Related
I'm working on a Kubernetes CronJob that relies on knowing when the last successfull run happened in order to correctly notify users about new events since said run. That said, I need to know if I can rely on status.lastScheduleTime to be my "last successful run" timestamp.
When does it change? Does it depend on exit codes? Is it even a good idea to use it for that purpose?
No, you can't rely on that as an indicator of a successful run. That value changes whenever your CronJob runs. It doesn't mean that it's your last successful run and it doesn't change depending on exit codes.
A CronJob essentially runs a Job with a name that is <cronjob name>-<unix epoch>. The epoch is in Unix/Linux what you would get from the date +%s command, for example, also that epoch is a timestamp that is slightly later than the timestamp of the lastScheduleTime (It's when the job resource gets created)
To find out if your last cron job ran successfully you can do something like the following.
You can get the last Job run/started name including its epoch with something like this:
$ kubectl get jobs | tail -1 | awk '{print $1}'
Then after that, you could check whether that job is successful with something like:
$ kubectl get job <job-name> -o=jsonpath='{.status.succeeded}'
Should return a 1.
I am looking for some support on creating some way of running a swift command in terminal to run a program and then stop it after 1 hour then restart.
Example of manual process:
Open Termain.
cd my app
swift run my program --with-parameters
ctrl+c (after 1 hours)
Restart with step 3
I am sure there must be some way using a bash script maybe to start the program by command, kill it after 60min and restart it with a continuous loop like that.
Thanks :-)
You can set up a cron job to do this. Basically, you'll have a bash script, say it's located at /Users/ben/scripts/run_my_program.sh that will, at every hour:
Terminate the current running process (kill pid)
Execute the swift run my program --with-parameters and spit out the process ID
you can get the PID of the swift process you launch with echo $!, and then use sleep 1h to sleep for 1 hour and then kill the process with kill -9 and the PID you got in the first step.
I have a job that runs every 15 minutes and gets a datetime. The batch file contains the following PowerShell command to get the current datetime minus 15 minutes:
for /f "delims=" %%a in ('"powershell [DateTime]::Now.AddMinutes(-15).ToString('yyyy-MM-ddTHH:mm:ss')"') do set "dt=%%a"
I write the results of the PowerShell command in my logs. I've been able to notice that every 5 or so job runs, there is a small delay of anywhere between 1-7 seconds. For example, if the job is scheduled to run at 15:00:00, and there is a delay of 5 seconds, my logs will show 14:45:05 as the datetime returned by the PowerShell command.
What could be the reason for this delay? Perhaps the PowerShell command takes longer than expected sometimes? Or could it be that the OS is not running the task on time because it might be struggling to handle resources?
Please let me know if I should provide any more info - I'm having a hard time to be less general/open-ended with the question.
I want to schedule a job to run every 5 minutes. but the question is that
Is there a way (like creating a crontab) to prevent a job from running, when the previous job has not been completed?
You can write a shell script to start the job only when the job is not already running. Configure the shell script in crontab every 5 minutes. this will ensure that the execution happens only when there is no instance of the job running already. This is how i have done for my cron jobs
Note : make use of ps -ef | grep commands in your shell script to identify if there is a process already running
I'm attempting to write a manager in Perl to automate a bioinformatics pipeline my lab has been using. (The REPET pipeline, for anyone who's interested.) The pipeline has eight steps, several of which are broken down into substeps which can be run in parallel. Most notably, step 3 is broken down into three parts, and step 4 into three corresponding parts. Each part of step 3 can be run independently, and its corresponding part in step 4 can be started as soon as its step 3 companion is finished. I'd like my manager to be able to launch step 3 in three parallel threads, and, for each thread, move on to step 4 as soon as step 3 is finished. The best way I can think to do that is to monitor the output of each process. The output from each step looks like this:
START TEdenovo.py (2012-08-23 11:20:10)
version 2.0
project name = dm3_chr2L
project directory = /home/<etc>
beginning of step 1
submitting job(s) with groupid 'dm3_chr2L_TEdenovo_prepareBatches' (2012-08-23 11:20:10)
waiting for 1 job(s) with groupid 'dm3_chr2L_TEdenovo_prepareBatches' (2012-08-23 11:20:10)
execution time per job: n=1 mean=2.995 var=0.000 sd=0.000 min=2.995 med=2.995 max=2.995
step 1 finished successfully
version 2.0
END TEdenovo.py (2012-08-23 11:20:25)
That's the output for step 1, but in step 3, when "step 3 finished successfully" appears in the output, it's safe to move on to step 4. The problem has been successfully tabulating the output for three of these processes as they run at once. Essentially, this is the behavior that I want (pseudocode):
my $log31 = `TEdenovo.py [options] &`;
my $log32 = `TEdenovo.py [options] &`;
my $log33 = `TEdenovo.py [options] &`;
while(1) {
#start step 41 if $log31 =~ /step 3 finished successfully/;
#start step 42 if $log32 =~ /step 3 finished successfully/;
#start step 43 if $log33 =~ /step 3 finished successfully/;
#monitor logs 41, 42, 43 similarly
last if #all logs read "finished successfully"
sleep(5);
}
#move on to step 5
The problem is that evoking a process with backticks causes perl to wait until that process has finished to move on; as I discovered, it isn't like with system(), where you can spin something into a background process with & and then proceed immediately. As far as I know, there isn't a good way to use system() to get the effect I'm looking for. I suppose I could do this:
system("TEdenovo.py [options] & > log31.txt");
And then poll log31.txt periodically to see whether "finished successfully" has appeared, but that seems unecessarily messy.
I've also tried opening the process in a filehandle:
open(my $step3, "TEdenovo.py [options] |");
my #log3;
while(1)
{
push(#log3, <$step3>);
last if grep("step 3 finished successfully", #log3);
sleep(5);
}
...but, once again, Perl waits until the process has finished in order to move on (in this case, at the push()). I tried the above with $| both set and unset.
So, the essence of my question is: Is there a way to capture the standard output of a running background process in perl?
maybe you could try
open(my $step3, "TEdenovo.py [options] |");
while(<$step3>)
{
last if /step 3 finished successfully/;
}
instead of while(1) ?
The approach of using open and reading from the pipehandle is the a correct approach. If Nahuel's suggestion of reading from the handle in scalar context doesn't help, you could still be suffering from buffering.
$| changes the buffering behavior of Perl's output, but not the behavior of any external programs called from Perl. You have to use an external program that doesn't buffer its output. In this case, I believe this is possible by passing the -u option to python:
open(my $step3, "|-", "python -u TEdenovo.py [more options]");