Invoking an argo workflow container with optional flags? - argo-workflows

Say I have a dockerized command that can take an optional flag, similar to how ls can take a flag -l.
How can I make argo invoke it properly, allowing the caller to choose whether to include the flag or not?
This does not work, but hopefully it is somewhat close and demonstrates my intent:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: do-work-
spec:
arguments:
parameters:
- name: long-format
value: false
entrypoint: do-work
templates:
- name: do-work
container:
image: alpine:3.7
command: [ls]
args:
- "{{workflow.parameters.long-format == 'true' ? '-l' : '' }}"
- /etc
If I invoke it like argo submit do-work.yaml or argo submit do-work.yaml -p long-format=false then I want it to run ls /etc.
If I invoke it like argo submit do-work.yaml -p long-format=true then I want it to run ls -l /etc.

I figured out that something like this can work, using python to jam them all together into one string... kind of ugly.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: do-work-
spec:
arguments:
parameters:
- name: dir
value: /etc
- name: long-format
value: false
entrypoint: main
templates:
- name: main
dag:
tasks:
- name: build-cli-args
template: build-cli-args
- name: do-work
depends: build-cli-args
template: do-work
arguments:
parameters:
- name: cli-args
value: '{{tasks.build-cli-args.outputs.result}}'
- name: build-cli-args
script:
image: python:alpine3.6
command: [python]
source: |
cli_args = [
"-l" if "{{workflow.parameters.long-format}}" == "true" else None,
"{{workflow.parameters.dir}}",
]
print(" ".join([a for a in cli_args if a is not None]))
- name: do-work
inputs:
parameters:
- name: cli-args
script:
image: alpine:3.7
command: [sh, -ex]
source: |
ls {{inputs.parameters.cli-args}}
I will leave the question open in case someone can tell me how to do it more minimally, with an expression like I was attempting.

Related

Argoworkflow can't call template inputs parameters within steps

kind: Workflow
metadata:
generateName: small-
spec:
entrypoint: fan-out-in-params-workflow
arguments:
parameters:
- name: jira-ticket
value: INFRA-000
templates:
- name: fan-out-in-params-workflow
steps:
- - name: generate
template: gen-host-list
- - name: pre-conditions
template: pre-conditions
arguments:
parameters:
- name: host
value: "{{item}}"
withParam: "{{steps.generate.outputs.result}}"
- name: gen-host-list
inputs:
artifacts:
- name: host
path: /tmp/host.txt
s3:
key: host.txt
script:
image: python:alpine3.6
command: [python]
source: |
import json
import sys
filename="{{ inputs.artifacts.host.path }}"
with open(filename, 'r', encoding='UTF-8') as f:
json.dump([line.rstrip() for line in f], sys.stdout)
- name: pre-conditions
inputs:
parameters:
- name: host
steps:
- - name: online-check
template: online-check
arguments:
parameters:
- name: host
value: {{inputs.parameters.host}}
- name: online-check
inputs:
parameters:
- name: host
script:
image: python:alpine3.6
command:
- python
source: |
print({{inputs.parameters.host}})
Hi there, I'm quite new to argoworkflow. Now I'm trying to call the template pre-conditions inputs parameters host like I posted above. But it seems the host params passes to template pre-conditions successfully but I can't get it in steps online-check, anyone can give me some advice ? Anything will be appreciated !

Tekton YAML TriggerTemplate - string substitution

I have this kind of yaml file to define a trigger
`
apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
name: app-template-pr-deploy
spec:
params:
- name: target-branch
- name: commit
- name: actor
- name: pull-request-number
- name: namespace
resourcetemplates:
- apiVersion: tekton.dev/v1alpha1
kind: PipelineRun
metadata:
generateName: app-pr-$(tt.params.actor)-
labels:
actor: $(tt.params.actor)
spec:
serviceAccountName: myaccount
pipelineRef:
name: app-pr-deploy
podTemplate:
nodeSelector:
location: somelocation
params:
- name: branch
value: $(tt.params.target-branch)
** - name: namespace
value: $(tt.params.target-branch)**
- name: commit
value: $(tt.params.commit)
- name: pull-request-number
value: $(tt.params.pull-request-number)
resources:
- name: app-cluster
resourceRef:
name: app-location-cluster
`
The issue is that sometime target-branch is like "integration/feature" and then the namespace is not valid
I would like to check if there is an unvalid character in the value and replace it if there is.
Any way to do it ?
Didn't find any valuable way to do it beside creating a task to execute this via shell script later in the pipeline.
This is something you could do from your EventListener, using something such as:
apiVersion: triggers.tekton.dev/v1alpha1
kind: EventListener
metadata:
name: xx
spec:
triggers:
- name: demo
interceptors:
- name: addvar
ref:
name: cel
params:
- name: overlays
value:
- key: branch_name
expression: "body.ref.split('/')[2]"
bindings:
- ref: your-triggerbinding
template:
ref: your-triggertemplate
Then, from your TriggerTemplate, you would add the "branch_name" param, parsed from your EventListener.
Note: payload from git notification may vary. Sample above valid with github. Translating remote/origin/master into master, or abc/def/ghi/jkl into ghi.
I've created a separate task that is doing all the magic I needed and output a valid namespace name into a different variable.
Then instead of use namespace variable, i use valid-namespace all the way thru the pipeline.
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: validate-namespace-task-v1
spec:
description: >-
This task will validate namespaces
params:
- name: namespace
type: string
default: undefined
results:
- name: valid-namespace
description: this should be a valid namespace
steps:
- name: triage-validate-namespace
image: some-image:0.0.1
script: |
#!/bin/bash
echo -n "$(params.namespace)" | sed "s/[^[:alnum:]-]/-/g" | tr '[:upper:]' '[:lower:]'| tee $(results.valid-namespace.path)
Thanks

How to pass list of arguments from argo workflow and use it in argo workflowtemplate

apiVersion: argoproj.io/v1alpha1
kind: Workflow
.
.
- name: mytemplate
steps:
- - name: mytask
templateRef:
name: ABCDworkflowtemplate
template: taskA
arguments:
parameters:
- name: mylist
value: [10,"some",false]
....................
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: ABCDworkflowtemplate
spec:
templates:
- name: taskA
inputs:
parameters:
- name: mylist
.
My question is how to use every element of this list {{input.parameters.?}} ? Help me with some reference. Thank you
You didn't really specify what exactly you want to do with these values so i'll explain both ways to use this input.
The more common usage with arrays is to iterate over them using "withParam", With this syntax a new task (pod) will be created for each of the items.
templates:
- name: taskA
inputs:
parameters:
- name: mylist
steps:
- - name: doSomeWithItem
template: doSomeWithItem
arguments:
parameters:
- name: item
value: "{{item}}"
withParam: "{{inputs.parameters.mylist}}"
- name: doSomeWithItem
inputs:
parameters:
- name: item
container:
image: python:alpine3.6
command: [ python ]
source: |
print("{{inputs.parameters.item}}")
The other option is just to pass the entire array as a variable to the pod, and use custom logic based on needs:
templates:
- name: taskA
inputs:
parameters:
- name: mylist
steps:
- - name: doSomethingWithList
template: doSomethingWithList
arguments:
parameters:
- name: list
value: "{{inputs.parameters.mylist}}"
- name: doSomethingWithList
inputs:
parameters:
- name: list
container:
image: python:alpine3.6
command: [ python ]
source: |
if (list[2] == 'some'):
// do somehting
else if (list[0] == 10]:
// do something

How to run multiple files in one container image and use the output as input in another container in Argo workflow?

I am trying to run 2 python files in one container , save the output as parameters:
- name: training
serviceAccountName: argo-service
outputs:
parameters:
- name: output-param-1
valueFrom:
path: ./tmp/output.txt
- name: output-param-2
valueFrom:
path: ./tmp/output2.txt
container:
image: (image name)
command: [python]
args: ["/data/argo/model_build.py","/data/argo/model_build_2.py"]
Use that output as input in another container :
- name: evaluate
serviceAccountName: argo-service
inputs:
parameters:
- name: value-1
- name: value-2
container:
image: (image name)
command: ["python", "/data/argo/evaluate.py"]
args:
- '{{inputs.parameters.value-1}}'
- '{{inputs.parameters.value-2}}'
and have defined the chain as :
- name: dag-chain
dag:
tasks:
- name: src
template: clone-repo
- name: prep
template: data-prep
dependencies: [src]
- name: train
template: training
dependencies: [prep]
- name: eval
template: evaluate
dependencies: [train]
args:
parameters:
- name: value-1
value: '{{dag-chain.tasks.train.outputs.parameters.output-param-1}}'
- name: value-2
value: '{{dag-chain.tasks.train.outputs.parameters.output-param-2}}'
But with these steps I'm getting the error :
" Internal Server Error: templates.dag-chain.tasks.eval templates.evaluate inputs.parameters.value-1 was not supplied: "
Please help me identify the mistakes I'm making.
I have tried the steps mentioned above but it's not working.
I don't have Argo accessible just now to test, but a couple of things to try:
args in DAG tasks should be arguments (see field reference for a DAG task here).
Try removing dag-chain from the parameters (see example here).
- name: dag-chain
dag:
tasks:
- name: src
template: clone-repo
- name: prep
template: data-prep
dependencies: [src]
- name: train
template: training
dependencies: [prep]
- name: eval
template: evaluate
dependencies: [train]
arguments:
parameters:
- name: value-1
value: '{{tasks.train.outputs.parameters.output-param-1}}'
- name: value-2
value: '{{tasks.train.outputs.parameters.output-param-2}}'
If that doesn't work I'll try some more steps with Argo.

Unable to pass output parameters from one workflowTemplate to a workflow via another workflowTemplate

I've Two workflowTemplates generate-output, lib-read-outputs and One workflow output-paramter as follows
generate-output.yaml
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: generate-output
spec:
entrypoint: main
templates:
- name: main
dag:
tasks:
# Generate Json for Outputs
- name: read-outputs
arguments:
parameters:
- name: outdata
value: |
{
"version": 4,
"terraform_version": "0.14.11",
"serial": 0,
"lineage": "732322df-5bd43-6e92-8f46-56c0dddwe83cb4",
"outputs": {
"key_alias_arn": {
"value": "arn:aws:kms:us-west-2:123456789:alias/tetsing-key",
"type": "string",
"sensitive": true
},
"key_arn": {
"value": "arn:aws:kms:us-west-2:123456789:alias/tetsing-key",
"type": "string",
"sensitive": true
}
}
}
template: retrieve-outputs
# Create Json
- name: retrieve-outputs
inputs:
parameters:
- name: outdata
script:
image: python
command: [python]
env:
- name: OUTDATA
value: "{{inputs.parameters.outdata}}"
source: |
import json
import os
OUTDATA = json.loads(os.environ["OUTDATA"])
with open('/tmp/templates_lst.json', 'w') as outfile:
outfile.write(str(json.dumps(OUTDATA['outputs'])))
volumeMounts:
- name: out
mountPath: /tmp
volumes:
- name: out
emptyDir: { }
outputs:
parameters:
- name: message
valueFrom:
path: /tmp/templates_lst.json
lib-read-outputs.yaml
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: lib-read-outputs
spec:
entrypoint: main
templates:
- name: main
dag:
tasks:
# Read Outputs
- name: lib-wft
templateRef:
name: generate-output
template: main
output-paramter.yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: output-paramter-
spec:
entrypoint: main
templates:
- name: main
dag:
tasks:
# Json Output data task1
- name: wf
templateRef:
name: lib-read-outputs
template: main
- name: lib-wf2
dependencies: [wf]
arguments:
parameters:
- name: outputResult
value: "{{tasks.wf.outputs.parameters.message}}"
template: whalesay
- name: whalesay
inputs:
parameters:
- name: outputResult
container:
image: docker/whalesay:latest
command: [cowsay]
args: ["{{inputs.parameters.outputResult}}"]
I am trying to pass the output parameters generated in workflowTemplate generate-output to workflow output-paramter via lib-read-outputs
When I execute them, it's giving the following error - Failed: invalid spec: templates.main.tasks.lib-wf2 failed to resolve {{tasks.wf.outputs.parameters.message}}
DAG and steps templates don't produce outputs by default
DAG and steps templates do not automatically produce their child templates' outputs, even if there is only one child template.
For example, the no-parameters template here does not produce an output, even though it invokes a template which does have an output.
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
spec:
templates:
- name: no-parameters
dag:
tasks:
- name: get-a-parameter
template: get-a-parameter
This lack of outputs makes sense if you consider a DAG template with multiple tasks:
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
spec:
templates:
- name: no-parameters
dag:
tasks:
- name: get-a-parameter
template: get-a-parameter
- name: get-another-parameter
depends: get-a-parameter
template: get-another-parameter
Which task's outputs should no-parameters produce? Since it's unclear, DAG and steps templates simply do not produce outputs by default.
You can think of templates as being like functions. You wouldn't expect a function to implicitly return the output of a function it calls.
def get_a_string():
return "Hello, world!"
def call_get_a_string():
get_a_string()
print(call_get_a_string()) # This prints nothing.
But a DAG or steps template can forward outputs
You can make a DAG or a steps template forward an output by setting its outputs field.
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: get-parameters-wftmpl
spec:
templates:
- name: get-parameters
dag:
tasks:
- name: get-a-parameter
template: get-a-parameter
- name: get-another-parameter
depends: get-a-parameter
template: get-another-parameter
# This is the critical part!
outputs:
parameters:
- name: parameter-1
valueFrom:
expression: "tasks['get-a-parameter'].outputs.parameters['parameter-name']"
- name: parameter-2
valueFrom:
expression: "tasks['get-another-parameter'].outputs.parameters['parameter-name']"
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
spec:
templates:
- name: print-parameter
dag:
tasks:
- name: get-parameters
templateRef:
name: get-parameters-wftmpl
template: get-parameters
- name: print-parameter
depends: get-parameters
template: print-parameter
arguments:
parameters:
- name: parameter
value: "{{tasks.get-parameters.outputs.parameters.parameter-1}}"
To continue the Python analogy:
def get_a_string():
return "Hello, world!"
def call_get_a_string():
return get_a_string() # Add 'return'.
print(call_get_a_string()) # This prints "Hello, world!".
So, in your specific case...
Add an outputs section to the main template in the generate-parameter WorkflowTemplate to forward the output parameter from the retrieve-parameters template.
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: generate-parameter
spec:
entrypoint: main
templates:
- name: main
outputs:
parameters:
- name: message
valueFrom:
expression: "tasks['read-parameters'].outputs.parameters.message"
dag:
tasks:
# ... the rest of the file ...
Add an outputs section to the main template in the lib-read-parameters WorkflowTemplate to forward generate-parameter's parameter.
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: lib-read-parameters
spec:
entrypoint: main
templates:
- name: main
outputs:
parameters:
- name: message
valueFrom:
expression: "tasks['lib-wft'].outputs.parameters.message"
dag:
tasks:
# ... the rest of the file ...