[In Airflow]
Please how I can trigger a task based on the result of another task? If task A result is true run Task 4 , if false ? re-run task A
You're likely looking for the BranchPythonOperator, you can read about it here -- here's some code copied from the documentation that branches based on the return value of a task named start_task:
def branch_func(**kwargs):
ti = kwargs['ti']
xcom_value = int(ti.xcom_pull(task_ids='start_task'))
if xcom_value >= 5:
return 'continue_task'
else:
return 'stop_task'
start_op = BashOperator(
task_id='start_task',
bash_command="echo 5",
xcom_push=True,
dag=dag)
branch_op = BranchPythonOperator(
task_id='branch_task',
provide_context=True,
python_callable=branch_func,
dag=dag)
continue_op = DummyOperator(task_id='continue_task', dag=dag)
stop_op = DummyOperator(task_id='stop_task', dag=dag)
start_op >> branch_op >> [continue_op, stop_op]
Related
We periodically runs jobs and we need to save the output into a PDS and then parse the output to extract parts of it to save into another member. It needs to be done by issuing a REXX command using the percent sign and the REXX member name as an SDSF command line. I've attempted to code a REXX to do this, but it is getting an error when trying to invoke an ISPF service, saying the ISPF environment has not been established. But, this is SDSF running under ISPF.
My code has this in it (copied from several sources and modified):
parse arg PSDSFPARMS "(" PUSERPARMS
parse var PSDSFPARMS PCURRPNL PPRIMPNL PROWTOKEN PPRIMCMD .
PRIMCMD=x2c(PPRIMCMD)
RC = isfquery()
if RC <> 0 then
do
Say "** SDSF environment does not exist, exec ending."
exit 20
end
RC = isfcalls("ON")
Address SDSF "ISFGET" PPRIMPNL "TOKEN('"PROWTOKEN"')" ,
" (" VERBOSE ")"
LRC = RC
if LRC > 0 then
call msgrtn "ISFGET"
if LRC <> 0 then
Exit 20
JOBNAME = value(JNAME.1)
JOBNBR = value(JOBID.1)
SMPDSN = "SMPE.*.OUTPUT.LISTINGS"
LISTC. = ''
SMPODSNS. = ''
SMPODSNS.0 = 0
$ = outtrap('LISTC.')
MSGVAL = msg('ON')
address TSO "LISTC LVL('"SMPDSN"') ALL"
MSGVAL = msg(MSGVAL)
$ = outtrap('OFF')
do LISTCi = 1 to LISTC.0
if word(LISTC.LISTCi,1) = 'NONVSAM' then
do
parse var LISTC.LISTCi . . DSN
SMPODSNS.0 = SMPODSNS.0 + 1
i = SMPODSNS.0
SMPODSNS.i = DSN
end
IX = pos('ENTRY',LISTC.LISTCi)
if IX <> 0 then
do
IX = pos('NOT FOUND',LISTC.LISTCi,IX + 8)
if IX <> 0 then
do
address ISPEXEC "SETMSG MSG(IPLL403E)"
EXITRC = 16
leave
end
end
end
LISTC. = ''
if EXITRC = 16 then
exit 0
address ISPEXEC "TBCREATE SMPDSNS NOWRITE" ,
"NAMES(TSEL TSMPDSN)"
I execute this code by typing %SMPSAVE next to the spool output line on the "H" SDSF panel and it runs fine until it gets to this point in the REXX:
114 *-* address ISPEXEC "TBCREATE SMPDSNS NOWRITE" ,
"NAMES(TSEL TSMPDSN)"
>>> "TBCREATE SMPDSNS NOWRITE NAMES(TSEL TSMPDSN)"
ISPS118S SERVICE NOT INVOKED. A VALID ISPF ENVIRONMENT DOES NOT EXIST.
+++ RC(20) +++
Does anyone know why it says I don't have a valid ISPF environment and how I can get around this?
I've done quite a bit in the past with REXX, including writing REXX code to handle line commands, but this is the first time I've tried to use ISPEXEC commands within this code.
Thank you,
Alan
My VBScript does not show the results of any command I execute. I know the command gets executed but I would like to capture the result.
I have tested many ways of doing this, for example the following:
Const WshFinished = 1
Const WshFailed = 2
strCommand = "ping.exe 127.0.0.1"
Set WshShell = CreateObject("WScript.Shell")
Set WshShellExec = WshShell.Exec(strCommand)
Select Case WshShellExec.Status
Case WshFinished
strOutput = WshShellExec.StdOut.ReadAll
Case WshFailed
strOutput = WshShellExec.StdErr.ReadAll
End Select
WScript.StdOut.Write strOutput 'write results to the command line
WScript.Echo strOutput 'write results to default output
But it does not print any results. How do I capture StdOut and StdErr?
WScript.Shell.Exec() returns immediately, even though the process it starts does not. If you try to read Status or StdOut right away, there won't be anything there.
The MSDN documentation suggests using the following loop:
Do While oExec.Status = 0
WScript.Sleep 100
Loop
This checks Status every 100ms until it changes. Essentially, you have to wait until the process completes, then you can read the output.
With a few small changes to your code, it works fine:
Const WshRunning = 0
Const WshFinished = 1
Const WshFailed = 2
strCommand = "ping.exe 127.0.0.1"
Set WshShell = CreateObject("WScript.Shell")
Set WshShellExec = WshShell.Exec(strCommand)
Do While WshShellExec.Status = WshRunning
WScript.Sleep 100
Loop
Select Case WshShellExec.Status
Case WshFinished
strOutput = WshShellExec.StdOut.ReadAll()
Case WshFailed
strOutput = WshShellExec.StdErr.ReadAll()
End Select
WScript.StdOut.Write(strOutput) 'write results to the command line
WScript.Echo(strOutput) 'write results to default output
You should read both streams INSIDE the loop as well as after it. When your process is verbose then it will block on the I/O buffer when this buffer will not be emptyfied succesively!!!
I think Tomek's answer is good, but incomplete.
Here's a code example.
Private Sub ExecuteCommand(sCommand$)
Dim wsh As Object
Set wsh = CreateObject("WScript.Shell")
Dim oExec As Object, oOut As TextStream
'Exec the command
Set oExec = wsh.Exec(sCommand$)
Set oOut = oExec.StdOut
'Wait for the command to finish
While Not oOut.AtEndOfStream
Call Debug.Print(oOut.ReadLine)
Wend
Select Case oExec.Status
Case WshFinished
Case WshFailed
Err.Raise 1004, "EndesaSemanal.ExecuteCommand", "Error: " & oExec.StdErr.ReadAll()
End Select
End Sub
I am new to these optimisation problems, I just found the or-tools library and saw that cp_model can fix problems that are close to mine.
I have printers and some tasks, that I want to schedule in order to finish the production the earliest. The tasks uses time on machine and raw material, that I must refill at the end of coil. For the moment, I don't consider changing a plastic coil before using all the material.
Here is some information about my situation:
1- The printers are all same, they can do every task, with the same efficiency.
2- A printer can only print one task at a time.
3- A printer cannot start without human around, so tasks can start only at certain hours (in the code below, from 0AM to 10 AM).
4- A task can finish at any time.
5- If a printer has no more material, it needs to be change, this can happen only on opening hours.
6- If a printer has no more material, the task is paused until new material is put.
7- I consider having unlimited quantity of material coil.
Thanks to examples and some search in the documentation, I have been able to fix all the issues that are not related to material. I have been able to set a maximum quantity per machine, but it is not my issue.
I don't understand how I can pause/resume my intervals (for the moment I set the duration to a fixed one).
from ortools.sat.python import cp_model
from ortools.util.python import sorted_interval_list
import random
Domain = sorted_interval_list.Domain
def main():
random.seed(0)
nb_jobs = 10
nb_machine = 2
horizon = 30000000
job_list = [] #Job : (id,time,quantity)
listOfEnds = []
quantityPerMachine = []
maxQuantity = 5
#create the jobs
for i in range(nb_jobs):
time = random.randrange(1,24)
quantity = random.randrange(1,4)
job_list.append([i,time,quantity])
print([job[1] for job in job_list])
print("total time to print = ",horizon)
print("quantity")
print([job[2] for job in job_list])
print("total quantity = ",sum([job[2] for job in job_list]))
model = cp_model.CpModel()
makespan = model.NewIntVar(0, horizon, 'makespan')
machineForJob = {}
#boolean variable for each combination of machine and job. if True then machine works on this job
for machine in range(nb_machine):
for job in job_list:
j = job[0]
machineForJob[(machine,j)]=(model.NewBoolVar(f'x[{machine},{j}]'))
#for each job, apply sum()==1 to ensure one machine works on each job only
for j in range(nb_jobs):
model.Add(sum([machineForJob[(machine,j)] for machine in range(nb_machine)])==1)
#set the affectation of the jobs
time_intervals={}
starts = {}
ends = {}
#Time domain represents working hours when someone can start the taks
timeDomain = []#[[0],[1],[2],[3],[4],[5],[6],[7],[8],[9],[10],[24],[25],[26]]
for i in range(20):
for j in range(10):
t= [j +i*24]
timeDomain.append(t)
for machine in range(nb_machine):
time_intervals[machine] = []
for job in job_list:
j = job[0]
duration = job[1]
starts[(machine,j)] = model.NewIntVarFromDomain(Domain.FromIntervals(timeDomain),f'start {machine},{j}')
ends[(machine,j)] = (model.NewIntVar(0, horizon, f'end {machine},{j}'))
time_intervals[machine].append(model.NewOptionalIntervalVar(starts[(machine,j)], duration, ends[(machine,j)],
machineForJob[(machine,j)],f'interval {machine},{j} '))
#time should not overlap, quantity of raw material is limited,
for machine in range(nb_machine):
#model.Add(quantityPerMachine[machine] <= maxQuantity) Not working as expected as the raw material cannot be refilled
model.AddNoOverlap(time_intervals[machine])
#calculate time per machine
time_per_machine = []
for machine in range(nb_machine):
q = 0
s = 0
for job in job_list:
s+= job[1]*machineForJob[(machine,job[0])]
listOfEnds.append(ends[(machine,job[0])])
q+= job[2]*machineForJob[(machine,job[0])]
time_per_machine.append(s)
quantityPerMachine.append(q)
#Goal is to finish all taks earliest
model.AddMaxEquality(makespan,listOfEnds)
model.Minimize(makespan)
solver = cp_model.CpSolver()
# Sets a time limit of 10 seconds.
solver.parameters.max_time_in_seconds = 600.0
#Solve and prints
status = solver.Solve(model)
if status == cp_model.OPTIMAL or status == cp_model.FEASIBLE:
print("optimal =",status == cp_model.OPTIMAL )
print(f'Total cost = {solver.ObjectiveValue()}')
for i in range(nb_machine):
for j in range(nb_jobs):
if solver.BooleanValue(machineForJob[(i,j)]):
print(
f'Machine {i} assigned to Job {j} Time = {job_list[j][1]},Quantity = {job_list[j][2]}')
print(f"[{solver.Value(starts[(i,j)])} ,{solver.Value(ends[(i,j)])}]")
else:
print('No solution found.')
# Statistics.
print('\nStatistics')
print(' - conflicts: %i' % solver.NumConflicts())
print(' - branches : %i' % solver.NumBranches())
print(' - wall time: %f s' % solver.WallTime())
I have a celery chain that runs some tasks. Each of the tasks can fail and be retried. Please see below for a quick example:
from celery import task
#task(ignore_result=True)
def add(x, y, fail=True):
try:
if fail:
raise Exception('Ugly exception.')
print '%d + %d = %d' % (x, y, x+y)
except Exception as e:
raise add.retry(args=(x, y, False), exc=e, countdown=10)
#task(ignore_result=True)
def mul(x, y):
print '%d * %d = %d' % (x, y, x*y)
and the chain:
from celery.canvas import chain
chain(add.si(1, 2), mul.si(3, 4)).apply_async()
Running the two tasks (and assuming that nothing fails), your would get/see printed:
1 + 2 = 3
3 * 4 = 12
However, when the add task fails the first time and succeeds in subsequent retry calls, the rest of the tasks in the chain do not run, i.e. the add task fails, all other tasks in the chain are not run and after a few seconds, the add task runs again and succeeds and the rest of the tasks in the chain (in this case mul.si(3, 4)) does not run.
Does celery provide a way to continue failed chains from the task that failed, onwards? If not, what would be the best approach to accomplishing this and making sure that a chain's tasks run in the order specified and only after the previous task has executed successfully even if the task is retried a few times?
Note 1: The issue can be solved by doing
add.delay(1, 2).get()
mul.delay(3, 4).get()
but I am interested in understanding why chains do not work with failed tasks.
You've found a bug :)
Fixed in https://github.com/celery/celery/commit/b2b9d922fdaed5571cf685249bdc46f28acacde3
will be part of 3.0.4.
I'm also interested in understanding why chains do not work with failed tasks.
I dig some celery code and what I've found so far is:
The implementation happends at app.builtins.py
#shared_task
def add_chain_task(app):
from celery.canvas import chord, group, maybe_subtask
_app = app
class Chain(app.Task):
app = _app
name = 'celery.chain'
accept_magic_kwargs = False
def prepare_steps(self, args, tasks):
steps = deque(tasks)
next_step = prev_task = prev_res = None
tasks, results = [], []
i = 0
while steps:
# First task get partial args from chain.
task = maybe_subtask(steps.popleft())
task = task.clone() if i else task.clone(args)
i += 1
tid = task.options.get('task_id')
if tid is None:
tid = task.options['task_id'] = uuid()
res = task.type.AsyncResult(tid)
# automatically upgrade group(..) | s to chord(group, s)
if isinstance(task, group):
try:
next_step = steps.popleft()
except IndexError:
next_step = None
if next_step is not None:
task = chord(task, body=next_step, task_id=tid)
if prev_task:
# link previous task to this task.
prev_task.link(task)
# set the results parent attribute.
res.parent = prev_res
results.append(res)
tasks.append(task)
prev_task, prev_res = task, res
return tasks, results
def apply_async(self, args=(), kwargs={}, group_id=None, chord=None,
task_id=None, **options):
if self.app.conf.CELERY_ALWAYS_EAGER:
return self.apply(args, kwargs, **options)
options.pop('publisher', None)
tasks, results = self.prepare_steps(args, kwargs['tasks'])
result = results[-1]
if group_id:
tasks[-1].set(group_id=group_id)
if chord:
tasks[-1].set(chord=chord)
if task_id:
tasks[-1].set(task_id=task_id)
result = tasks[-1].type.AsyncResult(task_id)
tasks[0].apply_async()
return result
def apply(self, args=(), kwargs={}, **options):
tasks = [maybe_subtask(task).clone() for task in kwargs['tasks']]
res = prev = None
for task in tasks:
res = task.apply((prev.get(), ) if prev else ())
res.parent, prev = prev, res
return res
return Chain
You can see that at the end prepare_steps prev_task is linked to the next task.
When the prev_task failed the next task is not called.
I'm testing with adding the link_error from prev task to the next:
if prev_task:
# link and link_error previous task to this task.
prev_task.link(task)
prev_task.link_error(task)
# set the results parent attribute.
res.parent = prev_res
But then, the next task must take care of both cases (maybe, except when it's configured to be immutable, e.g. not accept more arguments).
I think chain can support that by allowing some syntax likes this:
c = chain(t1, (t2, t1e), (t3, t2e))
which means:
t1 link to t2 and link_error to t1e
t2 link to t3 and link_error to t2e
I want to be able to run a program through command line and I want to start it with VbScript. I also want to get the output of the command line and assign it to a variable and I want all this to be done silently without cmd windows popping up. I have managed two things separately but not together. Here's what I got so far.
Run the command from cmd and get output:
Dim WshShell, oExec
Set WshShell = WScript.CreateObject("WScript.Shell")
Set oExec = WshShell.Exec("C:\snmpget -c public -v 2c 10.1.1.2 .1.3.6.1.4.1.6798.3.1.1.1.5.1")
x = oExec.StdOut.ReadLine
Wscript.Echo x
The above script works and does what I want except that cmd pops up for a brief moment.
Here's a script that will run silently but won't grab the output
Set WshShell = WScript.CreateObject("WScript.Shell")
Return = WshShell.Run("C:\snmpset -c public -v 2c -t 0 10.1.1.2 .1.3.6.1.4.1.6798.3.1.1.1.7.1 i 1", 0, true)
Is there a way to get these two to work together?
Let me give you a background on why I want do to this. I am basically polling a unit every 5-10 minutes and I am going to get the script to email or throw a message box when a certain condition occurs but I don't want to see cmd line popping up all day long on my computer. Any suggestions?
Thanks
You can redirect output to a file and then read the file:
return = WshShell.Run("cmd /c C:\snmpset -c ... > c:\temp\output.txt", 0, true)
Set fso = CreateObject("Scripting.FileSystemObject")
Set file = fso.OpenTextFile("c:\temp\output.txt", 1)
text = file.ReadAll
file.Close
I have taken this and various other comments and created a bit more advanced function for running an application and getting the output.
Example to Call Function: Will output the DIR list of C:\ for Directories only. The output will be returned to the variable CommandResults as well as remain in C:\OUTPUT.TXT.
CommandResults = vFn_Sys_Run_CommandOutput("CMD.EXE /C DIR C:\ /AD",1,1,"C:\OUTPUT.TXT",0,1)
Function
Function vFn_Sys_Run_CommandOutput (Command, Wait, Show, OutToFile, DeleteOutput, NoQuotes)
'Run Command similar to the command prompt, for Wait use 1 or 0. Output returned and
'stored in a file.
'Command = The command line instruction you wish to run.
'Wait = 1/0; 1 will wait for the command to finish before continuing.
'Show = 1/0; 1 will show for the command window.
'OutToFile = The file you wish to have the output recorded to.
'DeleteOutput = 1/0; 1 deletes the output file. Output is still returned to variable.
'NoQuotes = 1/0; 1 will skip wrapping the command with quotes, some commands wont work
' if you wrap them in quotes.
'----------------------------------------------------------------------------------------
On Error Resume Next
'On Error Goto 0
Set f_objShell = CreateObject("Wscript.Shell")
Set f_objFso = CreateObject("Scripting.FileSystemObject")
Const ForReading = 1, ForWriting = 2, ForAppending = 8
'VARIABLES
If OutToFile = "" Then OutToFile = "TEMP.TXT"
tCommand = Command
If Left(Command,1)<>"""" And NoQuotes <> 1 Then tCommand = """" & Command & """"
tOutToFile = OutToFile
If Left(OutToFile,1)<>"""" Then tOutToFile = """" & OutToFile & """"
If Wait = 1 Then tWait = True
If Wait <> 1 Then tWait = False
If Show = 1 Then tShow = 1
If Show <> 1 Then tShow = 0
'RUN PROGRAM
f_objShell.Run tCommand & ">" & tOutToFile, tShow, tWait
'READ OUTPUT FOR RETURN
Set f_objFile = f_objFso.OpenTextFile(OutToFile, 1)
tMyOutput = f_objFile.ReadAll
f_objFile.Close
Set f_objFile = Nothing
'DELETE FILE AND FINISH FUNCTION
If DeleteOutput = 1 Then
Set f_objFile = f_objFso.GetFile(OutToFile)
f_objFile.Delete
Set f_objFile = Nothing
End If
vFn_Sys_Run_CommandOutput = tMyOutput
If Err.Number <> 0 Then vFn_Sys_Run_CommandOutput = "<0>"
Err.Clear
On Error Goto 0
Set f_objFile = Nothing
Set f_objShell = Nothing
End Function
I am pretty new to all of this, but I found that if the script is started via CScript.exe (console scripting host) there is no window popping up on exec(): so when running:
cscript myscript.vbs //nologo
any .Exec() calls in the myscript.vbs do not open an extra window, meaning
that you can use the first variant of your original solution (using exec).
(Note that the two forward slashes in the above code are intentional, see cscript /?)
Here I found a solution, which works for me:
set wso = CreateObject("Wscript.Shell")
set exe = wso.Exec("cmd /c dir /s /b d:\temp\*.jpg")
sout = exe.StdOut.ReadAll
Look for assigning the output to Clipboard (in your first script) and then in second script parse Clipboard value.
#Mark Cidade
Thanks Mark! This solved few days of research on wondering how should I call this from the PHP WshShell. So thanks to your code, I figured...
function __exec($tmppath, $cmd)
{
$WshShell = new COM("WScript.Shell");
$tmpf = rand(1000, 9999).".tmp"; // Temp file
$tmpfp = $tmppath.'/'.$tmpf; // Full path to tmp file
$oExec = $WshShell->Run("cmd /c $cmd -c ... > ".$tmpfp, 0, true);
// return $oExec == 0 ? true : false; // Return True False after exec
return $tmpf;
}
This is what worked for me in my case. Feel free to use and modify as per your needs. You can always add functionality within the function to automatically read the tmp file, assign it to a variable and/or return it and then delete the tmp file.
Thanks again #Mark!
Dim path As String = GetFolderPath(SpecialFolder.ApplicationData)
Dim filepath As String = path + "\" + "your.bat"
' Create the file if it does not exist.
If File.Exists(filepath) = False Then
File.Create(filepath)
Else
End If
Dim attributes As FileAttributes
attributes = File.GetAttributes(filepath)
If (attributes And FileAttributes.ReadOnly) = FileAttributes.ReadOnly Then
' Remove from Readonly the file.
attributes = RemoveAttribute(attributes, FileAttributes.ReadOnly)
File.SetAttributes(filepath, attributes)
Console.WriteLine("The {0} file is no longer RO.", filepath)
Else
End If
If (attributes And FileAttributes.Hidden) = FileAttributes.Hidden Then
' Show the file.
attributes = RemoveAttribute(attributes, FileAttributes.Hidden)
File.SetAttributes(filepath, attributes)
Console.WriteLine("The {0} file is no longer Hidden.", filepath)
Else
End If
Dim sr As New StreamReader(filepath)
Dim input As String = sr.ReadToEnd()
sr.Close()
Dim output As String = "#echo off"
Dim output1 As String = vbNewLine + "your 1st cmd code"
Dim output2 As String = vbNewLine + "your 2nd cmd code "
Dim output3 As String = vbNewLine + "exit"
Dim sw As New StreamWriter(filepath)
sw.Write(output)
sw.Write(output1)
sw.Write(output2)
sw.Write(output3)
sw.Close()
If (attributes And FileAttributes.Hidden) = FileAttributes.Hidden Then
Else
' Hide the file.
File.SetAttributes(filepath, File.GetAttributes(filepath) Or FileAttributes.Hidden)
Console.WriteLine("The {0} file is now hidden.", filepath)
End If
Dim procInfo As New ProcessStartInfo(path + "\" + "your.bat")
procInfo.WindowStyle = ProcessWindowStyle.Minimized
procInfo.WindowStyle = ProcessWindowStyle.Hidden
procInfo.CreateNoWindow = True
procInfo.FileName = path + "\" + "your.bat"
procInfo.Verb = "runas"
Process.Start(procInfo)
it saves your .bat file to "Appdata of current user" ,if it does not exist and remove the attributes
and after that set the "hidden" attributes to file after writing your cmd code
and run it silently and capture all output saves it to file
so if u wanna save all output of cmd to file just add your like this
code > C:\Users\Lenovo\Desktop\output.txt
just replace word "code" with your .bat file code or command and after that the directory of output file
I found one code recently after searching alot
if u wanna run .bat file in vb or c# or simply
just add this in the same manner in which i have written