I'm trying out Argo workflow and would like to understand how to freeze a step. Let's say that I have 3 step workflow and a workflow failed at step 2. So I'd like to resubmit the workflow from step 2 using successful step 1's artifact. How can I achieve this? I couldn't find the guidance anywhere on the document.
I think you should consider using Conditions and Artifact passing in your steps.
Conditionals provide a way to affect the control flow of a
workflow at runtime, depending on parameters. In this example
the 'print-hello' template may or may not be executed depending
on the input parameter, 'should-print'. When submitted with
$ argo submit examples/conditionals.yaml
the step will be skipped since 'should-print' will evaluate false.
When submitted with:
$ argo submit examples/conditionals.yaml -p should-print=true
the step will be executed since 'should-print' will evaluate true.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: conditional-
spec:
entrypoint: conditional-example
arguments:
parameters:
- name: should-print
value: "false"
templates:
- name: conditional-example
inputs:
parameters:
- name: should-print
steps:
- - name: print-hello
template: whalesay
when: "{{inputs.parameters.should-print}} == true"
- name: whalesay
container:
image: docker/whalesay:latest
command: [sh, -c]
args: ["cowsay hello"]
If you use conditions in each step you will be able to start from a step you like with appropriate condition.
Also have a loot at this article Argo: Workflow Engine for Kubernetes as author explains the use of conditions on coinflip example.
You can see many examples on their GitHub page.
Related
I want to execute a task in Argo workflow if a string starts with a particular substring.
For example, my string is tests/dev-or.yaml and I want to execute task if my string starts with tasks/
Here is my workflow but the condition is not being validated properly
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: conditional-
spec:
entrypoint: conditional-example
arguments:
parameters:
- name: should-print
value: "tests/dev-or.yaml"
templates:
- name: conditional-example
inputs:
parameters:
- name: should-print
steps:
- - name: print-hello
template: whalesay
when: "{{inputs.parameters.should-print }} startsWith 'tests/'"
- name: whalesay
container:
image: docker/whalesay:latest
command: [sh, -c]
args: ["cowsay hello"]
Below is the error it is giving when I run the workflow
WorkflowFailed 7s workflow-controller Invalid 'when' expression 'tests/dev-or.yaml startsWith 'tests/'': Unable to access unexported field 'yaml' in token 'or.yaml'
Seems it is not accepting -, .yaml and / while evaluating the when condition.
Any mistake am making in my workflow? What's the right way to use this condition?
tl;dr - use this: when: "'{{inputs.parameters.should-print}}' =~ '^tests/'"
Parameter substitution happens before the when expression is evaluated. So the when expression is actually tests/dev-or.yaml startsWith 'tests/'. As you can see, the first string needs quotation marks.
But even if you had when: "'{{inputs.parameters.should-print}}' startsWith 'tests/'" (single quotes added), the expression would fail with this error: Cannot transition token types from STRING [tests/dev-or.yaml] to VARIABLE [startsWith].
Argo Workflows conditionals are evaluated as govaluate expressions. govaluate does not have any built-in functions, and Argo Workflows does not augment it with any functions. So startsWith is not defined.
Instead, you should use govaluate's regex comparator. The expression will look like this: when: "'{{inputs.parameters.should-print}}' =~ '^tests/'".
This is the functional Workflow:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: conditional-
spec:
entrypoint: conditional-example
arguments:
parameters:
- name: should-print
value: "tests/dev-or.yaml"
templates:
- name: conditional-example
inputs:
parameters:
- name: should-print
steps:
- - name: print-hello
template: whalesay
when: "'{{inputs.parameters.should-print}}' =~ '^tests/'"
- name: whalesay
container:
image: docker/whalesay:latest
command: [sh, -c]
args: ["cowsay hello"]
I have a Tekton Pipeline and PipelineRun definitions. But, I couldn't achieve to run Pipeline via passing parameter.
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: build-deploy-
labels:
tekton.dev/pipeline: build-deploy
spec:
serviceAccountName: tekton-build-bot
pipelineRef:
name: build-deploy
params:
- name: registry-address
value: $(REG_ADDRESS)
- name: repo-address
#value: $(REPO_ADDRESS)
value: $(REPO_ADDRESS)
- name: repo-name
value: $(REPO_NAME)
- name: version
value: $(VERSION)
workspaces:
- name: source
persistentVolumeClaim:
claimName: my-pvc
How can I pass while parameters while trying to run that runner with following command kubectl create -f pipelinerun.yaml?
Example:
value: $(REG_ADDRESS) -> I wanted to pass registry address as right before the running pipeline instead of giving hard-coded constant.
Any ideas?
You cannot pass those parameters when using kubectl create.
There are two alternatives:
Use tkn cli
You can use tkn, a purpose made CLI for Tekton. Then you can start a run of a Pipeline with, e.g.:
tkn pipeline start build-deploy \
--param registry-address=yay \
--param repo-name=nay \
--workspace name=source,claimName=my-pvc
Initiate pipeline with Trigger
You can setup a Trigger that initiates runs of your Pipeline on certain events, e.g. when you push to Git.
Then your PipelineRun template with parameter mapping is done using a TriggerTemplate
I'm exploring an easy way to read K8S resources in the Argo workflow. The current documentation is focusing mainly on create/patch with conditions (https://argoproj.github.io/argo/examples/#kubernetes-resources), while I'm curious if it's possible to perform "action: get", extra part of the resource state (or full resource) and pass it downstream as artifact or result output. Any ideas?
action: get is not a feature available from Argo.
However, it's easy to use kubectl from within a Pod and then send the JSON output to an output parameter. This uses a BASH script to send the JSON to the result output parameter, but an explicit output parameter or an output artifact are also viable options.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: kubectl-bash-
spec:
entrypoint: kubectl-example
templates:
- name: kubectl-example
steps:
- - name: generate
template: get-workflows
- - name: print
template: print-message
arguments:
parameters:
- name: message
value: "{{steps.generate.outputs.result}}"
- name: get-workflows
script:
image: bitnami/kubectl:latest
command: [bash]
source: |
some_workflow=$(kubectl get workflows -n argo | sed -n 2p | awk '{print $1;}')
kubectl get workflow "$some_workflow" -ojson
- name: print-message
inputs:
parameters:
- name: message
container:
image: alpine:latest
command: [sh, -c]
args: ["echo result was: '{{inputs.parameters.message}}'"]
Keep in mind that kubectl will run with the permissions of the Workflow's ServiceAccount. Be sure to submit the Workflow using a ServiceAccount which has access to the resource you want to get.
As per the Argo DAG template documentation.
tasks.<TASKNAME>.outputs.parameters: When the previous task uses
'withItems' or 'withParams', this contains a JSON array of the output
parameter maps of each invocation
When trying with the following simple workflow:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: test-workflow-
spec:
entrypoint: start
templates:
- name: start
dag:
tasks:
- name: with-items
template: hello-letter
arguments:
parameters:
- name: input-letter
value: "{{item}}"
withItems:
- A
- B
- C
- name: show-result
dependencies:
- with-items
template: echo-result
arguments:
parameters:
- name: input
value: "{{tasks.with-items.outputs.parameters}}"
- name: hello-letter
inputs:
parameters:
- name: input-letter
outputs:
parameters:
- name: output-letter
value: "{{inputs.parameters.input-letter}}"
script:
image: alpine
command: ["sh"]
source: |
echo "{{inputs.parameters.input-letter}}"
- name: echo-result
inputs:
parameters:
- name: input
outputs:
parameters:
- name: output
value: "{{inputs.parameters.input}}"
script:
image: alpine
command: ["sh"]
source: |
echo {{inputs.parameters.input}}
I get the following error :
Failed to submit workflow: templates.start.tasks.show-result failed to resolve {{tasks.with-items.outputs.parameters}}
Argo version (running in a minikube cluster)
argo: v2.10.0+195c6d8.dirty
BuildDate: 2020-08-18T23:06:32Z
GitCommit: 195c6d8310a70b07043b9df5c988d5a62dafe00d
GitTreeState: dirty
GitTag: v2.10.0
GoVersion: go1.13.4
Compiler: gc
Platform: darwin/amd64
Same error in Argo 2.8.1, although, using .result instead of .parameters in the show-result task worked fine (result was [A,B,C]), but doesn't work in 2.10 anymore
- name: show-result
dependencies:
- with-items
template: echo-result
arguments:
parameters:
- name: input
value: "{{tasks.with-items.outputs.result}}"
The result:
STEP TEMPLATE PODNAME DURATION MESSAGE
⚠ test-workflow-parallelism-xngg4 start
├-✔ with-items(0:A) hello-letter test-workflow-parallelism-xngg4-3307649634 6s
├-✔ with-items(1:B) hello-letter test-workflow-parallelism-xngg4-768315880 7s
├-✔ with-items(2:C) hello-letter test-workflow-parallelism-xngg4-2631126026 9s
└-⚠ show-result echo-result invalid character 'A' looking for beginning of value
I also tried to change the show-result task as:
- name: show-result
dependencies:
- with-items
template: echo-result
arguments:
parameters:
- name: input
value: "{{tasks.with-items.outputs.parameters.output-letter}}"
Executes without no errors:
STEP TEMPLATE PODNAME DURATION MESSAGE
✔ test-workflow-parallelism-qvp72 start
├-✔ with-items(0:A) hello-letter test-workflow-parallelism-qvp72-4221274474 8s
├-✔ with-items(1:B) hello-letter test-workflow-parallelism-qvp72-112866000 9s
├-✔ with-items(2:C) hello-letter test-workflow-parallelism-qvp72-1975676146 6s
└-✔ show-result echo-result test-workflow-parallelism-qvp72-3460867848 3s
But the parameter is not replaced by the value:
argo logs test-workflow-parallelism-qvp72
test-workflow-parallelism-qvp72-1975676146: 2020-08-25T14:52:50.622496755Z C
test-workflow-parallelism-qvp72-4221274474: 2020-08-25T14:52:52.228602517Z A
test-workflow-parallelism-qvp72-112866000: 2020-08-25T14:52:53.664320195Z B
test-workflow-parallelism-qvp72-3460867848: 2020-08-25T14:52:59.628892135Z {{tasks.with-items.outputs.parameters.output-letter}}
I don't understand what to expect as the output of a loop! What did I miss? Is there a way to find out what's happening?
There was a bug which caused this error before Argo version 3.2.5. Upgrade to latest and try again.
It looks like the problem was in the CLI only. I submitted the workflow with kubectl apply, and it ran fine. The error only appeared with argo submit.
The argo submit error was resolved when I upgraded to 3.2.6.
This is quite a common problem I've faced, I have not come across this in any bug report or feature documentation so far so it's yet to be determined if this is a feature or bug. However argo is clearly not capable in performing a "map-reduce" flow OOB.
The only "real" workaround I've found is to attach an artifact, write the with-items task output to it, and pass it along to your next step where you'll do the "reduce" yourself in code/script by reading values from the artifact.
---- edit -----
As mentioned by another answer this was indeed a bug which was resolved for the latest version, this resolves the usage of parameters as you mentioned as an option but outputs.result still causes an error post bugfix.
This issue is currently open on Argo Workflows Github: issue #6805
You could use a nested DAG to workaround this issue. This helps the parallel executions artifact resolution problem because each task output artifact is scoped to its inner nested DAG only, so there's only one upstream branch in the dependency tree. The error in issue #6805 happens when artifacts exist in the previous step and there's more than one upstream branch in the dependency tree.
I have a fairly large playbook that is capable of updating up to 10 services on a given host.
Let's say I have the services a b c d and I'd like to be to be able to selectively update the services by passing command line arguments, but default to updating everything when no arguments are passed. How could you do this in Ansible without being able to drop into arbitrary scripting?
Right now what I have is a when check on each service and define whether or not the service is true at playbook invocation. Given I may have as many as 10 services, I can't write boolean logic to accommodate every possibility.
I was hoping there is maybe a builtin like $# in bash that lists all arguments and I can do a check along the lines of when: $#.length = 0
ansible-playbook deploy.yml -e "a=true b=true d=true"
when: a == "true"
when: b == "true"
when: c == "true"
when: d == "true"
I would suggest to use tags. Lets say we have two services for example nginx and fpm. Then tag the play for nginx with nginx and for fpm with fpm. Below is an example for task level tagging, let say its named play.yml
- name: tasks for nginx
service: name=nginx state=reloaded
tags:
- nginx
- name: tasks for php-fpm
service: name=php-fpm state=reloaded
tags:
- fpm
Exeucting ansible-playbook play.yml will by default run both the tasks. But, If i change the command to
ansible-playbook play.yml --tags "nginx"
then only the task with nginx tag is executed. Tags can also be applied over play level or role level.
Play level tagging would look like
- hosts: all
remote_user: user
tasks:
- include: play1.yml
tags:
- play1
- include: play2.yml
tags:
- play2
In this case, all tasks inside the playbook play1.yml will inherit the tag play1 and the same for play2. While running ansible-playbook with the tag play1 then all tasks inside play1.yml are executed. Rather if we dont specify any tag all tasks from play1 and play2 are executed.
Note: A tasks is not limited to just one tag.
If you have a single play that you want to loop over the services, define that list in group_vars/all or somewhere else that makes sense:
services:
- first
- second
- third
- fourth
Then your tasks in start_services.yml playbook can look like this:
- name: Ensure passed variables are in services list
fail:
msg: "{{ item }} not in services list"
when: item not in services
with_items: "{{ varlist | default(services) }}"
- name: Start services
service:
name: "{{ item }}"
state: started
with_items: "{{ varlist | default(services) }}"
Pass in varlist as a JSON array:
$ ansible-playbook start_services.yml --extra-vars='{"varlist":[first,third]}'