Rundeck multiple job executions, queued or discarded? - rundeck

Rundeck docs for creating a job,
https://docs.rundeck.com/docs/manual/creating-jobs.html#creating-a-job
says:
Multiple Executions
By default, a job runs as a "Single Execution" -- it can only have a single execution running at a time. This is useful if the steps the Job performs might be interfered with if another separate process was also performing them on the same Node(s).
However, in some cases it is useful to allow a Job to be executed more than once simultaneously.
You can make a job allow "Multiple Executions" by toggling the value to Yes in the Job editor field shown below:
my questions:
For the default "Single Execution" mode, what happens to subsequent simultaneous job runs? Are they discarded or queued?
For "Multiple Execution" mode, what happens to subsequent simultaneous job runs that exceed the "Max number of multiple executions"? Are they discarded or queued?
Thanks for any guidance.

You can test with a simple job definition like this:
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>72fe54d5-26da-4cd4-a487-0955bd3b7f67</id>
<loglevel>INFO</loglevel>
<name>HelloWorld</name>
<nodeFilterEditable>false</nodeFilterEditable>
<plugins />
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>sleep 15; echo "hello"</exec>
</command>
</sequence>
<uuid>72fe54d5-26da-4cd4-a487-0955bd3b7f67</uuid>
</job>
</joblist>
In case of execute again the job you can see this error.
So, if you modify the job to allowing "two multiple execution" you can see that works normally with 2 executions of the same job:
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>72fe54d5-26da-4cd4-a487-0955bd3b7f67</id>
<loglevel>INFO</loglevel>
<maxMultipleExecutions>2</maxMultipleExecutions>
<multipleExecutions>true</multipleExecutions>
<name>HelloWorld</name>
<nodeFilterEditable>false</nodeFilterEditable>
<plugins />
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>sleep 15; echo "hello"</exec>
</command>
</sequence>
<uuid>72fe54d5-26da-4cd4-a487-0955bd3b7f67</uuid>
</job>
</joblist>
But if you execute three times the same job, you can see the same error of first scenario.
Short answer: if you exceed the default value or defined value, the execution is discarded.

Related

Rundeck - flow execution with error handler

Default error handler works in a way, if any job sequence does not end with the non-zero status it went into error handler flow.
I wanted to implement the flow, if the API endpoint returns 204, I need to perform some dependent operation.
I was thinking of doing it using an error handler but how I can deal with the return status 204 as it just handles non-zero status flow in the error handler.
On Rundeck Enterprise you can do it using the Ruleset Strategy (for example, catch the exit code and put it on a data option to decides if runs or not another job), take a look at this.
On the Community version you can do it scripting, manage the behavior on an script step in the following way (on my example i want to detect only the 127 error, defined on an option):
A job definition example:
<joblist>
<job>
<context>
<options preserveOrder='true'>
<option name='code_to_catch' value='127' />
</options>
</context>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>ab58e8e3-78f8-4a60-b6fc-4ad1167aa3a4</id>
<loglevel>INFO</loglevel>
<name>HelloWorld</name>
<nodeFilterEditable>false</nodeFilterEditable>
<plugins />
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<fileExtension>.sh</fileExtension>
<script><![CDATA[date-foo-bar # intentional error
mycode=$(printf '%d\n' $?)
case $mycode in
#option.code_to_catch#)
echo "Executing actions for #option.code_to_catch# error"
## also, you can run any job using RD-CLI (https://rundeck.github.io/rundeck-cli/)
## or Rundeck API (https://docs.rundeck.com/docs/api/rundeck-api.html)
;;
0)
echo "all ok!"
exit 0
;;
esac
]]></script>
<scriptargs />
<scriptinterpreter>/bin/bash</scriptinterpreter>
</command>
</sequence>
<uuid>ab58e8e3-78f8-4a60-b6fc-4ad1167aa3a4</uuid>
</job>
</joblist>
As you see, you can use the inline script to caught the error code and later run another job using RD-CLI if you like.

Getting global log filter to capture key/value data from another job in Rundeck

I use Rundeck to execute the Job A. In Job A, I have a Global Log Filter that captures key/value data using the default regex (RUNDECK:DATA:...).
I call another job - Job B - that prints stuff starting with RUNDECK:DATA:key = value. I execute Job B from Job A. Somehow the Global Log Filter that I setup in Job A doesn't parse the output of Job B.
I added the same Global Log Filter to Job B, and see that the data is captured correctly, but it is not available on Job A so that it can be used on the following steps.
Does anyone know if it is possible to capture key/value from the output of a another job? I managed to capture values from steps of the same job, but not when the output comes from an execution of another called job.
Checking your last comment the easiest way to do that is to use a file that stores the value (on JobB) and later is processed by the JobA next steps. Basically the JobA launches JobB via Job Reference Step, then JobB generates the data on a file, and the JobA recovers that value from the temporal file on the next steps.
JobA:
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>a0daf1e3-e918-43dc-b232-bc46a7a287b6</id>
<loglevel>INFO</loglevel>
<name>JobA</name>
<nodeFilterEditable>false</nodeFilterEditable>
<plugins />
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>echo "starting Job A"</exec>
</command>
<command>
<description>Job B call that geneates the temp value</description>
<jobref name='JobB' nodeStep='true'>
<uuid>ba183d3b-67d9-4499-b863-da8b7ac8aef3</uuid>
</jobref>
</command>
<command>
<exec>echo "printing the Job B value"</exec>
</command>
<command>
<exec>cat tempfile.txt</exec>
<plugins>
<LogFilter type='key-value-data'>
<config>
<invalidKeyPattern>\s|\$|\{|\}|\\</invalidKeyPattern>
<logData>true</logData>
<regex>^(mykey)\s*=\s*(.+)$</regex>
</config>
</LogFilter>
</plugins>
</command>
<command>
<exec>echo "the value is ${data.mykey}"</exec>
</command>
</sequence>
<uuid>a0daf1e3-e918-43dc-b232-bc46a7a287b6</uuid>
</job>
</joblist>
JobB:
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>ba183d3b-67d9-4499-b863-da8b7ac8aef3</id>
<loglevel>INFO</loglevel>
<name>JobB</name>
<nodeFilterEditable>false</nodeFilterEditable>
<plugins />
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>echo "Generating the value..."</exec>
</command>
<command>
<exec>echo "mykey=myvalue" > tempfile.txt</exec>
</command>
<command>
<exec>echo "Value generated on temp file"</exec>
</command>
</sequence>
<uuid>ba183d3b-67d9-4499-b863-da8b7ac8aef3</uuid>
</job>
</joblist>

How can I run a job on Rundeck for each user in the list?

I want to create a scheduled job on Rundeck. Rundeck Job runs the script that does some staff for a single user I have in my list.
So job has an input parameter - username. I need to run that job for all users in my list, but not to process it in one job execution. I need a separate job execution for each user.
Providing an example to make it more clear:
For instance, I have 3 users: user1, user2, user3.
I have a job that does some processing for that users and running it with parameter (unsername).
I need to create a scheduled job that will be run for each user.
1. Running job with user1 parameter
2. Running job with user2 parameter
3. Running job with user3 parameter
Is there anyway to do that?
You can design a workflow with a list option to use on any step (like command, inline-script or script), I did an example with a list option (users) and example inline-script that do some action for each user, the job is scheduled.
<joblist>
<job>
<context>
<options preserveOrder='true'>
<option name='users' required='true' value='alice,bob,charlie,david' />
</options>
</context>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>e62a25a0-b67c-4bb7-8721-c96b7e4cb9f5</id>
<loglevel>INFO</loglevel>
<name>JobEXAMPLE</name>
<nodeFilterEditable>false</nodeFilterEditable>
<plugins />
<schedule>
<month month='*' />
<time hour='12' minute='0' seconds='0' />
<weekday day='*' />
<year year='*' />
</schedule>
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<fileExtension>.sh</fileExtension>
<script><![CDATA[#!/bin/bash
myusers=#option.users#
Field_Separator=$IFS
# set comma as the internal field separator for the string list
IFS=,
for value in $myusers;
do
echo "action for $value"
done
IFS=$Field_Separator]]></script>
<scriptargs />
<scriptinterpreter>/bin/bash</scriptinterpreter>
</command>
</sequence>
<uuid>e62a25a0-b67c-4bb7-8721-c96b7e4cb9f5</uuid>
</job>
</joblist>
Here the result.
Maybe this interest you. Also, data values work in this case.

my rundeck job is not letting me pass all the parameters, just one, and I need all

I use this but only get a SINGLE PARAMETER, passed to the rundeck job:
... ${option.ticketnumber} ...
I want ALL the parameters (not just ONE) so that I can parse them within my code
free and CLEAR of Rundeck.
Anyone want to comment out there?
You can wrap your steps using key/value data with a regex to generate a set of data variables like this to use in your workflow (in this way you avoid to define individual options to pass later). Here a job definition example:
The job that generates the data values:
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description>Regex and data passing demo.</description>
<executionEnabled>true</executionEnabled>
<id>a0daf1e3-e918-43dc-b232-bc46a7a287b6</id>
<loglevel>INFO</loglevel>
<name>Regex</name>
<nodeFilterEditable>false</nodeFilterEditable>
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>env</exec>
</command>
<command>
<exec>echo ${data.SHELL}</exec>
</command>
<command>
<exec>echo ${data.USER}</exec>
</command>
<command>
<exec>echo ${data.PWD}</exec>
</command>
<command>
<jobref name='AnotherJob' nodeStep='true'>
<arg line='-option1 ${data.USER}' />
<uuid>4932f9c7-a435-4332-8b1a-5ade41da9edd</uuid>
</jobref>
</command>
<pluginConfig>
<LogFilter type='key-value-data'>
<config>
<logData>true</logData>
<regex>^(SHELL|USER|PWD)\s*=\s*(.+)$</regex>
</config>
</LogFilter>
</pluginConfig>
</sequence>
<uuid>a0daf1e3-e918-43dc-b232-bc46a7a287b6</uuid>
</job>
</joblist>
The job that gets the data values:
<joblist>
<job>
<context>
<options preserveOrder='true'>
<option name='option1' />
</options>
</context>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>4932f9c7-a435-4332-8b1a-5ade41da9edd</id>
<loglevel>INFO</loglevel>
<name>AnotherJob</name>
<nodeFilterEditable>false</nodeFilterEditable>
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>echo "the user from Regex Job is ${option.option1}"</exec>
</command>
</sequence>
<uuid>4932f9c7-a435-4332-8b1a-5ade41da9edd</uuid>
</job>
</joblist>
But to get it (passing) from another job, you need to set individual options using a job reference step with option as arguments.
I programmed around all of this to avoid the restrictions of the interface.

Can rundeck jobs be executed sequentially?

I am migrating a client from Control-M to a more modern solution. Client is running workloads in AWS, and has been considering using Rundeck to replace Control-M.
The control-m jobs run sequentially (one after another) only when the previous jobs terminates successfully.
A few of the jobs run in parallel (1 or more of the parallel jobs do not have dependent "child" jobs), but almost all run sequentially.
Does rundeck support running jobs sequentially? How can this be achieved?
Edit 1: client has over 500 jobs, with one command (shell) each. Target node is ec2 instance in same VPC with ssh public key authn configured.
You can set the Strategy of your Job, for that Create or Edit your job and go to Workflow tab > Strategy = Sequential, check this.
You have a complete explanation of strategies here and here.
EDIT 2: I leave an example focused on jobs using Job Reference Step.
Job A
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>7e85b6ff-4813-4f94-85f2-2b91ff359fd4</id>
<loglevel>INFO</loglevel>
<name>JobA</name>
<nodeFilterEditable>false</nodeFilterEditable>
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>sleep 2; echo "i am the job a"</exec>
</command>
</sequence>
<uuid>7e85b6ff-4813-4f94-85f2-2b91ff359fd4</uuid>
</job>
</joblist>
Job B
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>859788d6-1fef-4467-8ce9-bc34ef6735e2</id>
<loglevel>INFO</loglevel>
<name>JobB</name>
<nodeFilterEditable>false</nodeFilterEditable>
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>sleep 2; echo "i am the job b"</exec>
</command>
</sequence>
<uuid>859788d6-1fef-4467-8ce9-bc34ef6735e2</uuid>
</job>
</joblist>
Job C
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>458d13d1-7436-4d7e-b7a2-382a5bea449f</id>
<loglevel>INFO</loglevel>
<name>JobC</name>
<nodeFilterEditable>false</nodeFilterEditable>
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>sleep 2; echo "i am the job c"</exec>
</command>
</sequence>
<uuid>458d13d1-7436-4d7e-b7a2-382a5bea449f</uuid>
</job>
</joblist>
The parent job with Job Reference Step:
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>af4b4937-f4c2-4d73-ba8e-0ccc50bc6479</id>
<loglevel>INFO</loglevel>
<name>ParentJob</name>
<nodeFilterEditable>false</nodeFilterEditable>
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='sequential'>
<command>
<jobref name='JobA' nodeStep='true'>
<uuid>7e85b6ff-4813-4f94-85f2-2b91ff359fd4</uuid>
</jobref>
</command>
<command>
<jobref name='JobB' nodeStep='true'>
<uuid>859788d6-1fef-4467-8ce9-bc34ef6735e2</uuid>
</jobref>
</command>
<command>
<jobref name='JobC' nodeStep='true'>
<uuid>458d13d1-7436-4d7e-b7a2-382a5bea449f</uuid>
</jobref>
</command>
</sequence>
<uuid>af4b4937-f4c2-4d73-ba8e-0ccc50bc6479</uuid>
</job>
</joblist>
For example, if any job fails, the execution stops. With Rundeck 3.2.0 Enterprise, you can use the Job Resume feature to retake the execution at the failed step (only for sequential strategy).