Getting global log filter to capture key/value data from another job in Rundeck - rundeck

I use Rundeck to execute the Job A. In Job A, I have a Global Log Filter that captures key/value data using the default regex (RUNDECK:DATA:...).
I call another job - Job B - that prints stuff starting with RUNDECK:DATA:key = value. I execute Job B from Job A. Somehow the Global Log Filter that I setup in Job A doesn't parse the output of Job B.
I added the same Global Log Filter to Job B, and see that the data is captured correctly, but it is not available on Job A so that it can be used on the following steps.
Does anyone know if it is possible to capture key/value from the output of a another job? I managed to capture values from steps of the same job, but not when the output comes from an execution of another called job.

Checking your last comment the easiest way to do that is to use a file that stores the value (on JobB) and later is processed by the JobA next steps. Basically the JobA launches JobB via Job Reference Step, then JobB generates the data on a file, and the JobA recovers that value from the temporal file on the next steps.
JobA:
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>a0daf1e3-e918-43dc-b232-bc46a7a287b6</id>
<loglevel>INFO</loglevel>
<name>JobA</name>
<nodeFilterEditable>false</nodeFilterEditable>
<plugins />
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>echo "starting Job A"</exec>
</command>
<command>
<description>Job B call that geneates the temp value</description>
<jobref name='JobB' nodeStep='true'>
<uuid>ba183d3b-67d9-4499-b863-da8b7ac8aef3</uuid>
</jobref>
</command>
<command>
<exec>echo "printing the Job B value"</exec>
</command>
<command>
<exec>cat tempfile.txt</exec>
<plugins>
<LogFilter type='key-value-data'>
<config>
<invalidKeyPattern>\s|\$|\{|\}|\\</invalidKeyPattern>
<logData>true</logData>
<regex>^(mykey)\s*=\s*(.+)$</regex>
</config>
</LogFilter>
</plugins>
</command>
<command>
<exec>echo "the value is ${data.mykey}"</exec>
</command>
</sequence>
<uuid>a0daf1e3-e918-43dc-b232-bc46a7a287b6</uuid>
</job>
</joblist>
JobB:
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>ba183d3b-67d9-4499-b863-da8b7ac8aef3</id>
<loglevel>INFO</loglevel>
<name>JobB</name>
<nodeFilterEditable>false</nodeFilterEditable>
<plugins />
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>echo "Generating the value..."</exec>
</command>
<command>
<exec>echo "mykey=myvalue" > tempfile.txt</exec>
</command>
<command>
<exec>echo "Value generated on temp file"</exec>
</command>
</sequence>
<uuid>ba183d3b-67d9-4499-b863-da8b7ac8aef3</uuid>
</job>
</joblist>

Related

is there any way for rundeck opensource to trigger jobs at failed workflow step?

Do we have any plugin or is there any way to allow the jobs to resume execution at failed workflow step?
If my job 1 and job 2 got passed, then job 3 have to trigger.If job 1 got passed and job 2 got failed and when we fix it for job 2 and triggered it manually and if it got passes, then job 3 should know the job 2 got passes and job 1 previous state is passed.then job3 should automatically trigger. Do we have any plugin or a way to do this for rundeck opensource?
The best approach is to attach an error handler on your step (in your case job reference step), that error handler calls another job, I leave a basic example:
RemediationJob (the job that must be called if any ParentJob job reference fails through error handler):
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>67c56627-9dbd-4ad2-b6c1-04436ab3d0b2</id>
<loglevel>INFO</loglevel>
<name>RemediationJob</name>
<nodeFilterEditable>false</nodeFilterEditable>
<plugins />
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<fileExtension>.sh</fileExtension>
<script><![CDATA[echo "fixing..."
echo "fixed! try run the job again"]]></script>
<scriptargs />
<scriptinterpreter>/bin/bash</scriptinterpreter>
</command>
</sequence>
<uuid>67c56627-9dbd-4ad2-b6c1-04436ab3d0b2</uuid>
</job>
</joblist>
JobA:
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>5d19ea42-0431-4a68-a061-623f498222f6</id>
<loglevel>INFO</loglevel>
<name>ChildA</name>
<nodeFilterEditable>false</nodeFilterEditable>
<plugins />
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>echo "hi"</exec>
</command>
</sequence>
<uuid>5d19ea42-0431-4a68-a061-623f498222f6</uuid>
</job>
</joblist>
JobB (with an intentional error on the step):
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>00f5aaa5-5726-4f65-86a3-2a3c404bff08</id>
<loglevel>INFO</loglevel>
<name>ChildB</name>
<nodeFilterEditable>false</nodeFilterEditable>
<plugins />
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>eco "hi"</exec>
</command>
</sequence>
<uuid>00f5aaa5-5726-4f65-86a3-2a3c404bff08</uuid>
</job>
</joblist>
ParentJob (which calls JobA and JobB and uses error handler on any job reference step)
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>c2e0a05e-4fdd-4c33-9b4e-c9109d3667c6</id>
<loglevel>INFO</loglevel>
<name>ParentJob</name>
<nodeFilterEditable>false</nodeFilterEditable>
<plugins />
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<errorhandler>
<jobref name='RemediationJob' nodeStep='true'>
<uuid>67c56627-9dbd-4ad2-b6c1-04436ab3d0b2</uuid>
</jobref>
</errorhandler>
<jobref name='ChildA' nodeStep='true'>
<uuid>5d19ea42-0431-4a68-a061-623f498222f6</uuid>
</jobref>
</command>
<command>
<errorhandler>
<jobref name='RemediationJob' nodeStep='true'>
<uuid>67c56627-9dbd-4ad2-b6c1-04436ab3d0b2</uuid>
</jobref>
</errorhandler>
<jobref name='ChildB' nodeStep='true'>
<uuid>00f5aaa5-5726-4f65-86a3-2a3c404bff08</uuid>
</jobref>
</command>
</sequence>
<uuid>c2e0a05e-4fdd-4c33-9b4e-c9109d3667c6</uuid>
</job>
</joblist>
In this example, the second job reference always fails intentionally, in that way you can see how to trigger another job if the step (job reference step in your case) fails.

Job Conditional Reports Previous Job "Never Run"

I have a Job that calls other jobs as sort of a main run-file. I execute a preliminary check, which sets some values on a management server that I have to ensure I don't get any erroneous alerts due to the updates being run.
The preliminary job executes fine, but the conditional that comes right after it, ALWAYS outputs the status of:
{{UUID}} is NOT RUNNING AND previously NEVER. Expected NOT RUNNING AND 'Succeeded'
No matter what I do, I can't figure out if either I'm doing something wrong or if I just have something misconfigured, or if the conditionals are just broken. This is a small install for my homelab of about 10 servers, so I'm running with the default database back end, not MySQL or something else, so I don't know if that's an issue.
My Rundeck detail
Rundeck version: Rundeck 3.2.8-20200608
Install type: deb
OS Name/version: Ubuntu 18.04
DB Type/version: H2? (whatever the default is)
Verify the job state conditional expectation, I leave an example on 3.2.8 that works. Basically Job Step Conditional expects another job last execution as successfully, anything different of that makes the condition fails.
HelloWorld job:
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<group>test1</group>
<id>0bdd6fe5-addb-4051-b072-bb4430130a80</id>
<loglevel>INFO</loglevel>
<name>HelloWorld</name>
<nodeFilterEditable>false</nodeFilterEditable>
<plugins />
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>echo "hello world"</exec>
</command>
</sequence>
<uuid>0bdd6fe5-addb-4051-b072-bb4430130a80</uuid>
</job>
</joblist>
CheckerJob (With a Job Step Conditional step, expects that HelloWorld job with a successful execution):
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<group>test1</group>
<id>c6f2be75-81e8-4f8e-ba35-234d5c8a97d6</id>
<loglevel>INFO</loglevel>
<name>CheckerJob</name>
<nodeFilterEditable>false</nodeFilterEditable>
<plugins />
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>echo "starting..."</exec>
</command>
<command>
<step-plugin type='job-state-conditional'>
<configuration>
<entry key='condition' value='Equals' />
<entry key='executionState' value='Succeeded' />
<entry key='fail' value='true' />
<entry key='halt' value='true' />
<entry key='jobName' value='test1/HelloWorld' />
<entry key='jobProject' value='${job.project}' />
<entry key='jobUUID' value='0bdd6fe5-addb-4051-b072-bb4430130a80' />
<entry key='running' value='false' />
</configuration>
</step-plugin>
</command>
</sequence>
<uuid>c6f2be75-81e8-4f8e-ba35-234d5c8a97d6</uuid>
</job>
</joblist>
If the job never runs before.
If the job runs before (Job State Conditional expectation).
Another way
If you want to run Job State conditional in the same parent job, you need an individual execution of the "child" job, the way to do that is executing that job "externally" (using RD-CLI or API, I leave an example):
HelloWord:
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>dd33434e-5b6e-40e1-88b0-e9b5b52a3801</id>
<loglevel>INFO</loglevel>
<name>HelloWorld</name>
<nodeFilterEditable>false</nodeFilterEditable>
<plugins />
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>echo "hello world!"</exec>
</command>
</sequence>
<uuid>dd33434e-5b6e-40e1-88b0-e9b5b52a3801</uuid>
</job>
</joblist>
CheckerFakeParentJob (Executes HelloWorld vi RD-CLI and evaluate that execution later):
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>a90ec790-b856-4f00-8b66-2f37da1d00cb</id>
<loglevel>INFO</loglevel>
<name>CheckerFakeParentJob</name>
<nodeFilterEditable>false</nodeFilterEditable>
<plugins />
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>echo "starting"</exec>
</command>
<command>
<fileExtension>.sh</fileExtension>
<script><![CDATA[# print a message
echo "running HelloWorld job"
# run the hello world job via RD-CLI to get the individual execution and evaluate later
rd run -j HelloWorld -p ProjectEXAMPLE
# some time depending of job
sleep 3
# print a message
echo "done"]]></script>
<scriptargs />
<scriptinterpreter>/bin/bash</scriptinterpreter>
</command>
<command>
<step-plugin type='job-state-conditional'>
<configuration>
<entry key='condition' value='Equals' />
<entry key='executionState' value='Succeeded' />
<entry key='fail' value='true' />
<entry key='halt' value='true' />
<entry key='jobUUID' value='dd33434e-5b6e-40e1-88b0-e9b5b52a3801' />
<entry key='running' value='false' />
</configuration>
</step-plugin>
</command>
</sequence>
<uuid>a90ec790-b856-4f00-8b66-2f37da1d00cb</uuid>
</job>
</joblist>
Result here.
You have more information here.
EDIT: If you're referencing a Job Reference Step based workflow, you must take the parent job status (even the child job is executed through parent job).

my rundeck job is not letting me pass all the parameters, just one, and I need all

I use this but only get a SINGLE PARAMETER, passed to the rundeck job:
... ${option.ticketnumber} ...
I want ALL the parameters (not just ONE) so that I can parse them within my code
free and CLEAR of Rundeck.
Anyone want to comment out there?
You can wrap your steps using key/value data with a regex to generate a set of data variables like this to use in your workflow (in this way you avoid to define individual options to pass later). Here a job definition example:
The job that generates the data values:
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description>Regex and data passing demo.</description>
<executionEnabled>true</executionEnabled>
<id>a0daf1e3-e918-43dc-b232-bc46a7a287b6</id>
<loglevel>INFO</loglevel>
<name>Regex</name>
<nodeFilterEditable>false</nodeFilterEditable>
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>env</exec>
</command>
<command>
<exec>echo ${data.SHELL}</exec>
</command>
<command>
<exec>echo ${data.USER}</exec>
</command>
<command>
<exec>echo ${data.PWD}</exec>
</command>
<command>
<jobref name='AnotherJob' nodeStep='true'>
<arg line='-option1 ${data.USER}' />
<uuid>4932f9c7-a435-4332-8b1a-5ade41da9edd</uuid>
</jobref>
</command>
<pluginConfig>
<LogFilter type='key-value-data'>
<config>
<logData>true</logData>
<regex>^(SHELL|USER|PWD)\s*=\s*(.+)$</regex>
</config>
</LogFilter>
</pluginConfig>
</sequence>
<uuid>a0daf1e3-e918-43dc-b232-bc46a7a287b6</uuid>
</job>
</joblist>
The job that gets the data values:
<joblist>
<job>
<context>
<options preserveOrder='true'>
<option name='option1' />
</options>
</context>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>4932f9c7-a435-4332-8b1a-5ade41da9edd</id>
<loglevel>INFO</loglevel>
<name>AnotherJob</name>
<nodeFilterEditable>false</nodeFilterEditable>
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>echo "the user from Regex Job is ${option.option1}"</exec>
</command>
</sequence>
<uuid>4932f9c7-a435-4332-8b1a-5ade41da9edd</uuid>
</job>
</joblist>
But to get it (passing) from another job, you need to set individual options using a job reference step with option as arguments.
I programmed around all of this to avoid the restrictions of the interface.

Can rundeck jobs be executed sequentially?

I am migrating a client from Control-M to a more modern solution. Client is running workloads in AWS, and has been considering using Rundeck to replace Control-M.
The control-m jobs run sequentially (one after another) only when the previous jobs terminates successfully.
A few of the jobs run in parallel (1 or more of the parallel jobs do not have dependent "child" jobs), but almost all run sequentially.
Does rundeck support running jobs sequentially? How can this be achieved?
Edit 1: client has over 500 jobs, with one command (shell) each. Target node is ec2 instance in same VPC with ssh public key authn configured.
You can set the Strategy of your Job, for that Create or Edit your job and go to Workflow tab > Strategy = Sequential, check this.
You have a complete explanation of strategies here and here.
EDIT 2: I leave an example focused on jobs using Job Reference Step.
Job A
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>7e85b6ff-4813-4f94-85f2-2b91ff359fd4</id>
<loglevel>INFO</loglevel>
<name>JobA</name>
<nodeFilterEditable>false</nodeFilterEditable>
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>sleep 2; echo "i am the job a"</exec>
</command>
</sequence>
<uuid>7e85b6ff-4813-4f94-85f2-2b91ff359fd4</uuid>
</job>
</joblist>
Job B
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>859788d6-1fef-4467-8ce9-bc34ef6735e2</id>
<loglevel>INFO</loglevel>
<name>JobB</name>
<nodeFilterEditable>false</nodeFilterEditable>
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>sleep 2; echo "i am the job b"</exec>
</command>
</sequence>
<uuid>859788d6-1fef-4467-8ce9-bc34ef6735e2</uuid>
</job>
</joblist>
Job C
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>458d13d1-7436-4d7e-b7a2-382a5bea449f</id>
<loglevel>INFO</loglevel>
<name>JobC</name>
<nodeFilterEditable>false</nodeFilterEditable>
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>sleep 2; echo "i am the job c"</exec>
</command>
</sequence>
<uuid>458d13d1-7436-4d7e-b7a2-382a5bea449f</uuid>
</job>
</joblist>
The parent job with Job Reference Step:
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>af4b4937-f4c2-4d73-ba8e-0ccc50bc6479</id>
<loglevel>INFO</loglevel>
<name>ParentJob</name>
<nodeFilterEditable>false</nodeFilterEditable>
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='sequential'>
<command>
<jobref name='JobA' nodeStep='true'>
<uuid>7e85b6ff-4813-4f94-85f2-2b91ff359fd4</uuid>
</jobref>
</command>
<command>
<jobref name='JobB' nodeStep='true'>
<uuid>859788d6-1fef-4467-8ce9-bc34ef6735e2</uuid>
</jobref>
</command>
<command>
<jobref name='JobC' nodeStep='true'>
<uuid>458d13d1-7436-4d7e-b7a2-382a5bea449f</uuid>
</jobref>
</command>
</sequence>
<uuid>af4b4937-f4c2-4d73-ba8e-0ccc50bc6479</uuid>
</job>
</joblist>
For example, if any job fails, the execution stops. With Rundeck 3.2.0 Enterprise, you can use the Job Resume feature to retake the execution at the failed step (only for sequential strategy).

Running a job on multiple nodes in parallel in Rundeck

Is there a possibility that a job can be run in parallel [it has only one task] on all the target nodes [15 nodes precisely]?
Also, will the Thread Count option in rundeck GUI solve the above purpose?
PS: I only have GUI access to Rundeck
You need to dispatch your job to your 15 nodes, and set ThreadCount = 15. I did an example with 3 nodes (node00,node01 and node02) to understand it:
<joblist>
<job>
<defaultTab>summary</defaultTab>
<description></description>
<dispatch>
<excludePrecedence>true</excludePrecedence>
<keepgoing>false</keepgoing>
<rankOrder>ascending</rankOrder>
<successOnEmptyNodeFilter>false</successOnEmptyNodeFilter>
<threadcount>3</threadcount>
</dispatch>
<executionEnabled>true</executionEnabled>
<id>c4a37a41-db35-4a76-b407-be6170c6502d</id>
<loglevel>INFO</loglevel>
<name>JobRemoteParallel</name>
<nodeFilterEditable>false</nodeFilterEditable>
<nodefilters>
<filter>node.*</filter>
</nodefilters>
<nodesSelectedByDefault>true</nodesSelectedByDefault>
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>sleep 2; echo "starting"</exec>
</command>
<command>
<exec>sleep 5; echo "processing"</exec>
</command>
<command>
<exec>echo "done!"</exec>
</command>
</sequence>
<uuid>c4a37a41-db35-4a76-b407-be6170c6502d</uuid>
</job>
</joblist>
Check how works.