Rundeck - loop on referenced job - rundeck

I break my head with a problem of referenced job on my workflow. I'm not sur this is possible with Rundeck :
I have a job who call a second. I want run this second for all nodes but only over one server.
With this exemple maybe it's more simple to understand:
Workflow : Select Nodes
Referenced job 1
NodeA > Website www.exempleA.com < restore DB with default value
NodeB > Website www.exempleB.com < restore DB with default value
NodeC > Website www.exempleC.com < restore DB with default value
NodeD > Website www.exempleD.com < restore DB with default value
This run perfectly
Referenced Job 2 : Use Cypress server to test websites. Node filter have only Cypress server.
NodeE > Cypress -url https://${node.name} = NodeA > www.exempleA.com
NodeE > Cypress -url https://${node.name} = NodeB > www.exempleB.com
NodeE > Cypress -url https://${node.name} = NodeC > www.exempleC.com
NodeE > Cypress -url https://${node.name} = NodeD > www.exempleD.com
So I want to make a loop with a referenced job who execute on only one server but for all nodes name.
Someone know if this configuration is possible with Rundeck ?
Thank you for your knowledge.
Erwan

An excellent way to do that is to play with parent job options in two ways: first, against the first child job as a node filter (to dispatch to remote nodes), and second, against the second child job (to create an array and run the Cypress command in a bash loop).
Here is an example to test.
Parent Job. Contains an option that should be used for child jobs.
- defaultTab: nodes
description: ''
executionEnabled: true
id: db051872-7d5f-4506-bd49-17719af9785b
loglevel: INFO
name: ParentJob
nodeFilterEditable: false
options:
- name: nodes
value: node00 node01 node02
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- jobref:
args: -myfilter ${option.nodes}
group: ''
name: FirstChildJob
nodeStep: 'true'
uuid: f7271fc4-3ccb-41a5-9de4-a12e65093a3d
- jobref:
args: -myarray ${option.nodes}
childNodes: true
group: ''
name: SecondChildJob
nodeStep: 'true'
uuid: 1b8b1d82-a8dc-4949-9245-e973a8c37f5a
keepgoing: false
strategy: sequential
uuid: db051872-7d5f-4506-bd49-17719af9785b
First Child Job. Takes the parent job option and uses that as a job filter, the node filter is an own option called ${option.myfilter}.
- defaultTab: nodes
description: ''
executionEnabled: true
id: f7271fc4-3ccb-41a5-9de4-a12e65093a3d
loglevel: INFO
name: FirstChildJob
nodeFilterEditable: false
nodefilters:
dispatch:
excludePrecedence: true
keepgoing: false
rankOrder: ascending
successOnEmptyNodeFilter: false
threadcount: '1'
filter: ${option.myfilter}
nodesSelectedByDefault: true
options:
- name: myfilter
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- exec: echo "hi"
keepgoing: false
strategy: node-first
uuid: f7271fc4-3ccb-41a5-9de4-a12e65093a3d
Second Child Job. Contains an inline-script step that takes the parent's job option as an array and runs in a bash loop.
- defaultTab: nodes
description: ''
executionEnabled: true
id: 1b8b1d82-a8dc-4949-9245-e973a8c37f5a
loglevel: INFO
name: SecondChildJob
nodeFilterEditable: false
nodefilters:
dispatch:
excludePrecedence: true
keepgoing: false
rankOrder: ascending
successOnEmptyNodeFilter: false
threadcount: '1'
filter: 'name: node02'
nodesSelectedByDefault: true
options:
- name: myarray
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- script: "#!/bin/bash\narray=(#option.myarray#)\nfor i in \"${array[#]}\"\ndo\n\
\techo \"execute $i\"\ndone"
keepgoing: false
strategy: node-first
uuid: 1b8b1d82-a8dc-4949-9245-e973a8c37f5a
Here is the loop script (inside the second child job as inline-script):
#!/bin/bash
array=(#option.myarray#)
for i in "${array[#]}"
do
echo "$i"
done
And here you can see the result.

Related

Rundeck: Pass data between jobs

I'm trying to follow the instructions provided at https://stackoverflow.com/a/61802154 to pass output from one job as input into another job.
Job1 sets up the k/v data
- defaultTab: output
description: ''
executionEnabled: true
id: b6656d3b-2b32-4554-b224-52bd3702c305
loglevel: INFO
name: job1
nodeFilterEditable: false
nodefilters:
dispatch:
excludePrecedence: true
keepgoing: false
rankOrder: ascending
successOnEmptyNodeFilter: false
threadcount: '1'
filter: 'name: rdnode01'
nodesSelectedByDefault: true
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- description: output k/v
exec: echo RUNDECK:DATA:MYNUM=123
- description: test k/v
exec: echo ${data.MYNUM}
keepgoing: false
pluginConfig:
LogFilter:
- config:
invalidKeyPattern: \s|\$|\{|\}|\\
logData: 'true'
regex: ^RUNDECK:DATA:\s*([^\s]+?)\s*=\s*(.+)$
replaceFilteredResult: 'false'
type: key-value-data
strategy: node-first
uuid: b6656d3b-2b32-4554-b224-52bd3702c305
Job2 will output that k/v data
- defaultTab: output
description: ''
executionEnabled: true
id: c069e7d3-2d1f-46f2-a4d8-15eb19761daf
loglevel: INFO
name: job2
nodeFilterEditable: false
nodefilters:
dispatch:
excludePrecedence: true
keepgoing: false
rankOrder: ascending
successOnEmptyNodeFilter: false
threadcount: '1'
filter: 'name: rdnode01'
nodesSelectedByDefault: true
options:
- name: option_for_receive
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- exec: echo ${option.option_for_receive}
keepgoing: false
strategy: node-first
uuid: c069e7d3-2d1f-46f2-a4d8-15eb19761daf
Wrapper runs the job references as node steps and passes the data from job1 to job2
- defaultTab: output
description: ''
executionEnabled: true
id: 5a62cabf-ffc2-45d1-827b-156f4134a082
loglevel: INFO
name: wrapper job
nodeFilterEditable: false
nodefilters:
dispatch:
excludePrecedence: true
keepgoing: false
rankOrder: ascending
successOnEmptyNodeFilter: false
threadcount: '1'
filter: 'name: rdnode01'
nodesSelectedByDefault: true
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- description: job1
jobref:
childNodes: true
group: ''
name: job1
nodeStep: 'true'
uuid: b6656d3b-2b32-4554-b224-52bd3702c305
- description: job2
jobref:
args: -option_for_receive ${data.MYNUM}
childNodes: true
group: ''
name: job2
nodeStep: 'true'
uuid: c069e7d3-2d1f-46f2-a4d8-15eb19761daf
keepgoing: false
strategy: node-first
uuid: 5a62cabf-ffc2-45d1-827b-156f4134a082
This the formatted text from the execution log
11:26:39 [rundeck#rdnode01 1#node=rdnode01/1][NORMAL] RUNDECK:DATA:MYNUM=123
11:26:40 [rundeck#rdnode01 1#node=rdnode01/1][NORMAL] {"MYNUM":"123"}
11:26:40 [rundeck#rdnode01 1#node=rdnode01/2][NORMAL] 123
11:26:41 [rundeck#rdnode01 2#node=rdnode01/1][NORMAL] '${data.MYNUM}'
This is what it looks like on the screen:
As you can see, job2 is outputting '${data.MYNUM}' instead of the actual contents. Thus I think there's a syntax issue somewhere.
The data values are generated in the job context, in that case, the "Wrapper Job" (Parent Job in the Rundeck terminology) doesn't know about that data variable in their context (generated in the first job).
If you want to pass that data value to another job, call the second one from the first one in the following way (Workflow Node Step):
JobA:
- defaultTab: output
description: ''
executionEnabled: true
id: b6656d3b-2b32-4554-b224-52bd3702c305
loglevel: INFO
name: job1
nodeFilterEditable: false
nodefilters:
dispatch:
excludePrecedence: true
keepgoing: false
rankOrder: ascending
successOnEmptyNodeFilter: false
threadcount: '1'
filter: 'name: localhost '
nodesSelectedByDefault: true
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- description: output k/v
exec: echo RUNDECK:DATA:MYNUM=123
- description: test k/v
exec: echo ${data.MYNUM}
- jobref:
args: -option_for_receive ${data.MYNUM}
childNodes: true
group: ''
name: job2
nodeStep: 'true'
uuid: c069e7d3-2d1f-46f2-a4d8-15eb19761daf
keepgoing: false
pluginConfig:
LogFilter:
- config:
invalidKeyPattern: \s|\$|\{|\}|\\
logData: 'true'
regex: ^RUNDECK:DATA:\s*([^\s]+?)\s*=\s*(.+)$
replaceFilteredResult: 'false'
type: key-value-data
strategy: node-first
uuid: b6656d3b-2b32-4554-b224-52bd3702c305
JobB:
- defaultTab: output
description: ''
executionEnabled: true
id: c069e7d3-2d1f-46f2-a4d8-15eb19761daf
loglevel: INFO
name: job2
nodeFilterEditable: false
nodefilters:
dispatch:
excludePrecedence: true
keepgoing: false
rankOrder: ascending
successOnEmptyNodeFilter: false
threadcount: '1'
filter: 'name: localhost '
nodesSelectedByDefault: true
options:
- name: option_for_receive
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- exec: echo ${option.option_for_receive}
keepgoing: false
strategy: node-first
uuid: c069e7d3-2d1f-46f2-a4d8-15eb19761daf

How to make a Rundeck parent job with different nodes for each workflow step?

I have three jobs in the same project with their own node filters. And the matched nodes do not overlap between these jobs. I want to create a parent job that runs these three jobs instead of me running them individually. How do I configure the nodes on this parent job? Each step has it's own list of nodes.
Nothing is needed in the Parent Job, just edit the Job Reference Steps and click on the "Use referenced job's nodes." checkbox.
A basic example:
Parent Job:
- defaultTab: nodes
description: ''
executionEnabled: true
id: a0d5834d-4b62-44d9-bd1e-f00a6befb990
loglevel: INFO
name: ParentJob
nodeFilterEditable: false
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- jobref:
childNodes: true
group: ''
name: JobA
uuid: 63fb953c-53e0-4233-ba28-eabd69a0e41c
- jobref:
childNodes: true
group: ''
name: JobB
uuid: 8936db73-9bd4-4912-ae07-c5fc8500ee9d
- jobref:
childNodes: true
group: ''
name: JobC
uuid: 16fa66d3-fbda-439a-9a2b-14f90e99f72b
keepgoing: false
strategy: node-first
uuid: a0d5834d-4b62-44d9-bd1e-f00a6befb990
JobA:
- defaultTab: nodes
description: ''
executionEnabled: true
id: 63fb953c-53e0-4233-ba28-eabd69a0e41c
loglevel: INFO
name: JobA
nodeFilterEditable: false
nodefilters:
dispatch:
excludePrecedence: true
keepgoing: false
rankOrder: ascending
successOnEmptyNodeFilter: false
threadcount: '1'
filter: 'name: node00 '
nodesSelectedByDefault: true
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- exec: hostname
keepgoing: false
strategy: node-first
uuid: 63fb953c-53e0-4233-ba28-eabd69a0e41c
JobB:
- defaultTab: nodes
description: ''
executionEnabled: true
id: 8936db73-9bd4-4912-ae07-c5fc8500ee9d
loglevel: INFO
name: JobB
nodeFilterEditable: false
nodefilters:
dispatch:
excludePrecedence: true
keepgoing: false
rankOrder: ascending
successOnEmptyNodeFilter: false
threadcount: '1'
filter: 'name: node01'
nodesSelectedByDefault: true
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- exec: hostname
keepgoing: false
strategy: node-first
uuid: 8936db73-9bd4-4912-ae07-c5fc8500ee9d
JobC:
- defaultTab: nodes
description: ''
executionEnabled: true
id: 16fa66d3-fbda-439a-9a2b-14f90e99f72b
loglevel: INFO
name: JobC
nodeFilterEditable: false
nodefilters:
dispatch:
excludePrecedence: true
keepgoing: false
rankOrder: ascending
successOnEmptyNodeFilter: false
threadcount: '1'
filter: 'name: node02'
nodesSelectedByDefault: true
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- exec: hostname
keepgoing: false
strategy: node-first
uuid: 16fa66d3-fbda-439a-9a2b-14f90e99f72b
Check the Result.

YAML pipeline : How to disable a template depending on variable

I have a yaml pipeline calling a step AndroidSigning#3 which needs apksignerKeystoreFile input. I want to disable this step depending on a variable in the pipeline's library.
I get this error when starting the pipeline :
/build/pipelines/build-mobile-android.yml (Line: 42, Col: 14): Unexpected value 'ne(, 'release')'
Any help is appreciated
- group: variables-group
stages:
- stage: build
jobs:
- job: buil_app
steps:
- task: AndroidSigning#3
enabled: ne(${{ variables.env }}, 'release')
inputs:
apkFiles: 'blabla'
apksign: true
Looking at YAML schema reference it looks that this is not possible here
steps:
- script: string # contents of the script to run
displayName: string # friendly name displayed in the UI
name: string # identifier for this step (A-Z, a-z, 0-9, and underscore)
workingDirectory: string # initial working directory for the step
failOnStderr: boolean # if the script writes to stderr, should that be treated as the step failing?
condition: string
continueOnError: boolean # 'true' if future steps should run even if this step fails; defaults to 'false'
enabled: boolean # whether to run this step; defaults to 'true'
enabled must be boolean. You may use condition and then ne( variables['env'], 'release') it should do the job.
steps:
- task: AndroidSigning#3
condition: eq(variables['env'], 'release')
displayName: AndroidSigning - enabled
enabled: true
inputs:
apkFiles: 'blabla'
apksign: true
- task: AndroidSigning#3
condition: ne(variables['env'], 'release')
displayName: AndroidSigning - disable
enabled: false
inputs:
apkFiles: 'blabla'
apksign: true

Azure Devops: Template use sequence typed parameter to generate jobs and stages

It seems the template variable does not support sequence type but the template parameter does support it(called object type in Microsoft doc), together with JobList type and StageList type, but they seem to be just sequences with defined schema.
When iterating through object-typed parameter to generate step I didn't meet any problem, but I got error called 'A sequence was not expected' when I was trying to do such to generate jobs and stages.
So the result have to be like this:
# main.yml:
trigger: none
extends:
template: template.yml
parameters:
PROJECTNAME: foo.bar
DEPLOYMENTTARGETS:
- stage:
jobs:
- deployment:
environment:
name: TEST
tags: Web # tag must be comma separated string which is also kinda weird
- stage:
jobs:
- deployment:
environment:
name: PROD
tags: Web
# template.yml:
parameters:
- name: PROJECTNAME
type: string
default: ""
- name: DEPLOYMENTTARGETS
type: stageList # I was expecting object to be working
default: []
variables:
- group: LoginSecrets
stages:
- ${{ each deploymentTarget in parameters.DEPLOYMENTTARGETS }}: # iterating through an object here will result in error: 'A sequence was not expected', iterating through a StageList is OK
- stage: Deploy_${{replace(parameters.PROJECTNAME,'.','_')}}_${{replace(deploymentTarget.jobs[0].environment.name,'-','')}}_${{replace(replace(deploymentTarget.jobs[0].environment.tags,',','_'),'-','')}}_Stage # assuming each stage contains only one deployment job. We also iterate through jobs here if required.
dependsOn: BuildAndPublish_${{replace(parameters.PROJECTNAME,'.','_')}}_Stage
jobs:
- deployment: Deploy_${{replace(parameters.PROJECTNAME,'.','_')}}_Job
environment:
name: ${{deploymentTarget.jobs[0].environment.name}}
resourceType: VirtualMachine
tags: ${{deploymentTarget.jobs[0].environment.tags}}
strategy:
runOnce:
deploy:
steps:
- pwsh: $(Pipeline.Workspace)/${{parameters.PROJECTNAME}}/pipelineRelease.ps1
env:
LOGINNAME: $(loginName)
LOGINPASSWORD: $(loginPassword)
PROJECTNAME: ${{parameters.PROJECTNAME}}
But I was expecting something like this to be working:
# main.yml:
trigger: none
extends:
template: template.yml
parameters:
PROJECTNAME: foo.bar
DEPLOYMENTTARGETS:
- EnvName: TEST
Tags: Web
- EnvName: PROD
Tags: Web
# template.yml:
parameters:
- name: PROJECTNAME
type: string
default: ""
- name: DEPLOYMENTTARGETS
type: object # instead of StageList, I passed an object here
default: []
variables:
- group: LoginSecrets
stages:
- ${{ each deploymentTarget in parameters.DEPLOYMENTTARGETS }}: # error here
- stage: Deploy_${{replace(parameters.PROJECTNAME,'.','_')}}_${{replace(deploymentTarget.EnvName,'-','')}}_${{replace(replace(deploymentTargetTags,',','_'),'-','')}}_Stage
dependsOn: BuildAndPublish_${{replace(parameters.PROJECTNAME,'.','_')}}_Stage
jobs:
- deployment: Deploy_${{replace(parameters.PROJECTNAME,'.','_')}}_Job
environment:
name: ${{deploymentTarget.EnvName}}
resourceType: VirtualMachine
tags: ${{deploymentTarget.Tags}}
strategy:
runOnce:
deploy:
steps:
- pwsh: $(Pipeline.Workspace)/${{parameters.PROJECTNAME}}/pipelineRelease.ps1
env:
LOGINNAME: $(loginName)
LOGINPASSWORD: $(loginPassword)
PROJECTNAME: ${{parameters.PROJECTNAME}}
Azure Devops: Template use sequence typed parameter to generate jobs and stages
I am afraid we have to use the stageList instead of object to transfer two-dimensional arrays.
That because the parameters is defined as two-dimensional arrays in the main.yml, But if we define this arrays as object in the template.yml file. In this case, we could not loop the object in the template file.
You could check the document Loop through parameters for some more details.
Also, you could check the similar thread for some details.

Rundeck global variable in meta job not resolved

we have a Rundeck (3.1.2-20190927) job which triggers multiple other jobs in Rundeck. In the /etc/rundeck/framework.properties are global variables defined that are used in the jobs. It's used to build the url of our Icinga so Rundeck can submit th job result to the monitoring. The variable is used in the notifications tab (edit job -> notification) of every single job.
When the meta job has run, it submits the result successfully to the monitoring. The same applies to the 'sub' job, if you trigger them manualy. BUT if they are triggert by the meta job, they throw an error:
Error calling the endpoint: Illegal character in authority at index 8: https://icinga-master-${globals.environment}.some.domain.
It looks like the global variable is not resolved correctly when the job are triggert by the meta job. Strange to say, other meta jobs don't have this problem. I can't find any configuration differences. Has anybody an idea what the cause could be?
Thanks for any help!
Update: Here's my job definition
- defaultTab: output
description: "Call Patchday Jobs in a row \n**Attention! This will reboot the servers**"
executionEnabled: true
group: CloudServices/SomeSoftWare
id: 5cf2966c-3e5f-4a32-8cce-b3e82b6fd036
loglevel: INFO
multipleExecutions: true
name: Patchday SomeSoftWare - Meta Job
nodeFilterEditable: false
nodefilters:
dispatch:
excludePrecedence: true
keepgoing: false
rankOrder: ascending
successOnEmptyNodeFilter: false
threadcount: '1'
filter: 'tags: role_SomeSoftWare'
nodesSelectedByDefault: false
notification:
onfailure:
plugin:
configuration:
_noSSLVerification: ''
_printResponseToFile: ''
_proxySettings: ''
authentication: Basic
body: |-
{
"type": "Service",
"filter": "service.name==\"Rundeck-Job - ${job.name}\"",
"exit_status": 2,
"plugin_output": "'${job.name}' failed"
}
contentType: application/json
file: ''
headers: 'Accept: application/json'
method: POST
noSSLVerification: 'true'
oauthTokenEndpoint: ''
oauthValidateEndpoint: ''
password: ******
proxyIP: ''
proxyPort: ''
remoteUrl: https://icinga-master-${globals.environment}.some.domain:5665/v1/actions/process-check-result
timeout: '30000'
username: rundeck_process-check-result
type: HttpNotification
onsuccess:
plugin:
configuration:
_noSSLVerification: ''
_printResponseToFile: ''
_proxySettings: ''
authentication: Basic
body: |-
{
"type": "Service",
"filter": "service.name==\"Rundeck-Job - ${job.name}\"",
"exit_status": 0,
"plugin_output": "'${job.name}' succeeded"
}
contentType: application/json
file: ''
headers: 'Accept: application/json'
method: POST
noSSLVerification: 'true'
oauthTokenEndpoint: ''
oauthValidateEndpoint: ''
password: ******
proxyIP: ''
proxyPort: ''
remoteUrl: https://icinga-master-${globals.environment}.some.domain:5665/v1/actions/process-check-result
timeout: '30000'
username: rundeck_process-check-result
type: HttpNotification
notifyAvgDurationThreshold: null
options:
- description: Addition paramater to give to the ansible-playbook call
name: AdditionalParameter
scheduleEnabled: true
sequence:
commands:
- jobref:
args: 'branch: ${globals.defaultbranch}'
group: Infrastructure
name: Icinga Service Downtime
nodefilters:
dispatch:
nodeIntersect: true
uuid: 6eec5749-ef35-481e-aea8-674f233c32ac
- description: Pause Bamboo Server
jobref:
args: -branch ${globals.defaultbranch} -bamboo_command pause
group: Infrastructure
name: Bamboo Control
nodefilters:
dispatch:
nodeIntersect: true
uuid: 87bc7f1c-d133-4d7e-9df9-2b40fb935fd4
- configuration:
ansible-become: 'false'
ansible-disable-limit: 'false'
ansible-playbook-inline: |
- name: +++ Stop Tomcat ++++
hosts: tag_role_SomeSoftWare
gather_facts: true
remote_user: ec2-user
become: yes
tasks:
- name: stop tomcat
service:
name: tomcat
state: stopped
nodeStep: false
type: com.batix.rundeck.plugins.AnsiblePlaybookInlineWorkflowStep
- description: SomeSoftWare Update
jobref:
args: -branch ${globals.defaultbranch}
group: Infrastructure
name: SomeSoftWare Server - Setup
nodefilters:
dispatch:
nodeIntersect: true
uuid: f01a4483-d8b2-43cf-99fd-6a610d25c3a4
- description: Install/Update fs-cli
jobref:
args: -branch ${globals.defaultbranch}
group: Infrastructure
name: firstspirit-cli - Setup
nodefilters:
dispatch:
nodeIntersect: true
uuid: c7c54433-be96-4d85-b2c1-32d0534b5c60
- description: Install/Update Modules
jobref:
args: -branch ${globals.defaultbranch}
group: Infrastructure
name: SomeSoftWare Modules - Setup
nodefilters:
dispatch:
nodeIntersect: true
uuid: f7a8929b-2bc3-4abe-8c69-e0d2acf62159
- description: restart SomeSoftWare
exec: sudo service SomeSoftWare restart
- description: Resume Bamboo Server
jobref:
args: -branch ${globals.defaultbranch} -bamboo_command resume
group: Infrastructure
name: Bamboo Control
nodefilters:
dispatch:
nodeIntersect: true
uuid: 87bc7f1c-d133-4d7e-9df9-2b40fb935fd4
keepgoing: false
strategy: node-first
timeZone: Europe/Berlin
uuid: 5cf2966c-3e5f-4a32-8cce-b3e82b6fd036