Rundeck - Using Tags in Nodes gives an error when running the job - tags

In Rundeck within a Job I used the following value under Nodes - Node Filter.
tags:${option.server_env}
My tags are defined under Workflow - Options:
Option type: Text
Option name: server_env
Allowed values: List (Defined my tags over there)
List delimiter: ,
My tags are defined as well under the Project settings - Edit nodes..
When I open my job, select my chosen tag and run my job I retrieve the
following error.
The following plugin is installed.
How can I get it to work that nodes are a variable value so I can use 1 job for multiple tags (environments)?

I made an example based on this three-nodes model source.
<?xml version="1.0" encoding="UTF-8"?>
<project>
<node name="node00" description="Node 00" tags="db" hostname="192.168.56.20" osArch="amd64" osFamily="unix" osName="Linux" osVersion="3.10.0-1062.4.1.el7.x86_64" username="vagrant" ssh-key-storage-path="keys/rundeck" />
<node name="node01" description="Node 01" tags="db" hostname="192.168.56.21" osArch="amd64" osFamily="unix" osName="Linux" osVersion="3.10.0-1062.4.1.el7.x86_64" username="vagrant" ssh-key-storage-path="keys/rundeck" />
<node name="node02" description="Node 02" tags="bk" hostname="192.168.56.22" osArch="amd64" osFamily="unix" osName="Linux" osVersion="3.10.0-1062.4.1.el7.x86_64" username="vagrant" ssh-key-storage-path="keys/rundeck" />
</project>
It works in this way.
Create a job with an option, the option must contain a list of values, a default value and, a delimiter (a comma in my example). Also, the job must be dispatched to a node using the node tag in this way.
Save the job.
First test: run the job just using the default value, the job is dispatched only to the bk tagged nodes.
Second test: select all options, then the job is dispatched to all nodes (db and bk tagged nodes).
Here the job definition example to test.
- defaultTab: nodes
description: ''
executionEnabled: true
id: 3ebd683d-4f0a-42ae-8d32-2be22fbb9ae5
loglevel: INFO
name: JobDispatchedToNodes
nodeFilterEditable: false
nodefilters:
dispatch:
excludePrecedence: true
keepgoing: false
rankOrder: ascending
successOnEmptyNodeFilter: false
threadcount: '1'
filter: tags:${option.mynodes}
nodesSelectedByDefault: true
options:
- delimiter: ','
enforced: true
multivalued: true
name: mynodes
required: true
value: db
values:
- db
- bk
valuesListDelimiter: ','
plugins:
ExecutionLifecycle: {}
scheduleEnabled: true
schedules: []
sequence:
commands:
- exec: uname -a
keepgoing: false
strategy: node-first
uuid: 3ebd683d-4f0a-42ae-8d32-2be22fbb9ae5

Related

Running GitHub scripts directly from Rundeck

I have a Windows server running the community version of Rundeck, and I want to be able to run PowerShell scripts directly from my private GitHub repository, by providing the URL of the script.
On GitHub, I have created a personal access token that doesn't expire, because I'm running multiple jobs on Rundeck and I don't want to be modifying each job, every time a token expires.
I've tried various ways to do this but nothing works.
providing the username and password in the URL
providing the username and access token in the URL
https://$username:$token#raw.githubusercontent.com/$username/general/main/rundeck/$script.ps1
https://$username:$password#raw.githubusercontent.com/$username/general/main/rundeck/$script.ps1
I know that at some point GitHub disabled password authentication in the URL, so I'm looking for an alternative way to do it.
If it's possible I don't want to clone the git repository. I want to use a link to the raw script and run it directly.
It works in the following way:
On GitHub Go to your Avatar icon (up to right) > Settings > Developer Settings (left menu) > Personal Access Token > Tokens.
Click on the "Generate new token" button (up to right) > Generate new token (classic). Make sure to create a token with enough rights to read your repository. Save this token value.
On Rundeck, create a new job and add "Script file or URL", on the "File Path/URL" field put the remote path with the following format: https://<your_github_user_name>:<your_user_token>#raw.githubusercontent.com/username/repository/branch/your_script.sh
I made a job definition example that works (tested on Rundeck 4.8.0):
- defaultTab: nodes
description: ''
executionEnabled: true
id: 5f74a444-7b2f-4e50-af96-ca770c9d038e
loglevel: INFO
name: RemoteScriptTest
nodeFilterEditable: false
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- fileExtension: .sh
interpreterArgsQuoted: false
scriptInterpreter: /bin/bash
scripturl: https://user:my_user_token#raw.githubusercontent.com/user/scripts/main/myscript.sh
keepgoing: false
strategy: node-first
uuid: 5f74a444-7b2f-4e50-af96-ca770c9d038e
This example calls a bash script but you can use Powershell to define powershell.exe on the "Invocation String" and .ps1 on the "File extension" text boxes.
Note: if you want to use an option to store the token, use the ${option.myoption} format, here is another example:
- defaultTab: nodes
description: ''
executionEnabled: true
id: 5f74a444-7b2f-4e50-af96-ca770c9d038e
loglevel: INFO
name: RemoteScriptTest
nodeFilterEditable: false
options:
- name: mytoken
value: abc123
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- fileExtension: .sh
interpreterArgsQuoted: false
scriptInterpreter: /bin/bash
scripturl: https://user:${option.mytoken}#raw.githubusercontent.com/user/scripts/main/myscript.sh
keepgoing: false
strategy: node-first
uuid: 5f74a444-7b2f-4e50-af96-ca770c9d038e

YAML-based deployment not working from ADO, can do a manual deploy from VS Code

I am building a yaml-based release pipeline to deploy an app service from ADO to Azure App Services. I can perform a manual deploy and all of the code deploys as a zip and then it works with WEBSITE_RUN_FROM_PACKAGE="1".
for example from VS Code:
az webapp config appsettings set --resource-group RGP --name sitename --settings WEBSITE_RUN_FROM_PACKAGE="1"
az webapp deployment source config-zip --resource-group RGP --name sitename --src sitename.zip
It returns
"author": "N/A",
"author_email": "N/A",
"complete": true,
"deployer": "ZipDeploy",
"end_time": "2022-02-02T04:28:38.6143958Z",
"id": "xxx",
"is_readonly": true,
"is_temp": false,
"last_success_end_time": "2022-02-02T04:28:38.6143958Z",
"log_url": "https://sitename.scm.azurewebsites.net/api/deployments/latest/log",
"message": "Created via a push deployment",
"progress": "",
"provisioningState": "Succeeded",
"received_time": "2022-02-02T04:28:10.9806495Z",
"site_name": "sitename",
"start_time": "2022-02-02T04:28:11.3244119Z",
"status": 4,
"status_text": "",
"url": "https://sitename.scm.azurewebsites.net/api/deployments/latest"
However, in my YAML release, I am using the following, and it both over-writes WEBSITE_RUN_FROM_PACKAGE="1" and won't run with or without that value added after the fact. It return a site permission error.
Here is the relevant YAML section. I'm wondering if I need to use deployer: zipDeploy somehow, based on the above success and the below failing even though it puts the same zip file in the wwwroot dir of the app service. FYI - I am using terraform to provision the site and Configuration Items and then running the deploy afterwards.
- stage: deploy
dependsOn: [infrastructure]
displayName: DEPLOY APP
jobs:
# Track deployments on the environment.
- deployment: DeployWeb
displayName: deploy Web App
pool:
vmImage: 'windows-latest'
# Creates an environment if it doesn't exist.
environment: '$(workstream)-$(env_identifier)'
strategy:
# Default deployment strategy
runOnce:
deploy:
steps:
- checkout: self
- bash: ls $BUILD_ARTIFACTSTAGINGDIRECTORY
- bash: ls $PIPELINE_WORKSPACE
- task: AzureRmWebAppDeployment#4
displayName: 'deploy $(tier_prefix)-$(region_prefix)-$(workstream)-UI-01'
inputs:
ConnectionType: 'AzureRM'
azureSubscription: 'subname'
# SlotName: 'slot'
appType: 'webApp'
WebAppName: '$(tier_prefix)-$(region_prefix)-$(workstream)-UI-01'
ResourceGroupName: 'XZ$(region_prefix)-E-N-$(rgp_identifier)-$(tier_prefix)-RGP-10'
# DeployToSlotOrASEFlag: true
Package: '$(Pipeline.Workspace)/build/eu-web-$(workstream)/'
UseWebDeploy: true
enableCustomDeployment: true
deploymentMethod: runFromPackage
ExcludeFilesFromAppDataFlag: false
TakeAppOfflineFlag: false
removeAdditionalFilesFlag: true
additionalArguments: '-useCheckSum'
Here was my solution, so far it seems to work.
I completely dropped everything around UseWebDeploy
UseWebDeploy: true
enableCustomDeployment: true
deploymentMethod: runFromPackage
ExcludeFilesFromAppDataFlag: false
TakeAppOfflineFlag: false
removeAdditionalFilesFlag: true
additionalArguments: '-useCheckSum'
and change this
Package: '$(Pipeline.Workspace)/build/eu-web-$(workstream)/'
to this
Package: '$(Pipeline.Workspace)/build/eu-web-$(workstream)/eu-web-$(workstream)-$(env_identifier).zip'

Rundeck passing key-data value as command line argument

How I can pass the log filter key-value data into one of the reference job as an argument.
I was able to use the options available on the job as an argument to pass on the child reference job use,
But when I passed my log filter key-value data variable as ${data.my_log_filer_var} from the parent job argument, it does not evaluate in the child job but other argument passing is working for like for other option variables, I passed as: ${option.my_another_var}
Please let me know if there is any other way to pass the ${data.my_log_filer_var} to my child job.
Following this, I made a little example. Basically, you need an option on the child's job to "receive" the argument from the parent's job. This example is a couple of job definitions that you can import to your instance.
Child job (contains an option, this option is called from parent job passing the data variable):
- defaultTab: nodes
description: ''
executionEnabled: true
id: 0aeaa0f4-d090-4083-b0a5-2878c5f558d1
loglevel: INFO
name: ChildJob
nodeFilterEditable: false
options:
- name: opt1
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- exec: 'echo "the argument is: ${option.opt1}"'
keepgoing: false
strategy: node-first
uuid: 0aeaa0f4-d090-4083-b0a5-2878c5f558d1
And the Parent job (creates a data value and passes it to a child using arguments -opt1 ${data.mydata})
- defaultTab: nodes
description: ''
executionEnabled: true
id: 5c3d4248-9761-45d4-b164-6e38d5146007
loglevel: INFO
name: ParentJob
nodeFilterEditable: false
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- exec: env
plugins:
LogFilter:
- config:
invalidKeyPattern: \s|\$|\{|\}|\\
logData: 'true'
regex: ^(USER)\s*=\s*(.+)$
type: key-value-data
- exec: 'echo "the data value is ${data.USER}, sending to child job using arguments
on job reference step..."'
- jobref:
args: -opt1 ${data.USER}
group: ''
name: ChildJob
nodeStep: 'true'
uuid: 0aeaa0f4-d090-4083-b0a5-2878c5f558d1
keepgoing: false
strategy: node-first
uuid: 5c3d4248-9761-45d4-b164-6e38d5146007
The global variable step can capture data variable captured by key-value data log filter.
For example, if you capture the key-value data to export group, you can create a global variable {export.keyname*}, which can be used on other steps or in notifications. Note that if you only want the value from a single node, you should use {export.keyname#nodename} instead.
Rundeck Docs

Azure DevOps file transform with vault secrets

I need to deploy linux azure app service via azure devops. My configuration is stored in appsettings file and I need to substitute configuration values to the values stored in azure vault.
So I created variable group in artifacts, linked it to variables in pipeline and used FileTransform#2 to substitute appsettings values.
But it substitutes to null values. If I explicitly define variable value in pipeline by assigning some string value - it works fine.
Also I cannot use AzureRmWebAppDeployment#4 with JSONFiles, it does not work for linux deployment
What is the way of solving this?
here is pipeline code:
trigger:
branches:
include:
- master
- develop
- release/*
paths:
include:
- src/ConsumerBackEnd/*
- src/ConsumerShared/*
variables:
- name: poolName
value: 'Private-Windows10'
- name: azureRegisteredApp
value: 'portal-devops'
- name: workingDirectory
value: '$(System.DefaultWorkingDirectory)/src/ConsumerBackEnd'
- name: solutionDirectory
value: '$(System.DefaultWorkingDirectory)/src'
stages:
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
pool:
name: $(poolName)
variables:
- group: ConsumerDevVariableGroup
- name: 'Graph.GraphAppTenantId'
value: '**************' #works fine
- name: 'Graph.GraphAppClientId'
value: '$[variables.GraphAppClientId]' #should take value from vault but injects null
- task: FileTransform#2
inputs:
folderPath: '$(workingDirectory)'
xmlTransformationRules:
jsonTargetFiles: '**/appsettings.json'
- task: DotNetCoreCLI#2
displayName: Nuget Restore
inputs:
command: 'restore'
projects: '$(workingDirectory)/*.csproj'
feedsToUse: 'config'
nugetConfigPath: '$(solutionDirectory)/NuGet.config'
- task: DotNetCoreCLI#2
displayName: Build
inputs:
command: 'build'
projects: |
$(workingDirectory)/ConsumerBackEnd.csproj
arguments: --output $(System.DefaultWorkingDirectory)/output
- task: DotNetCoreCLI#2
displayName: Publish
inputs:
command: 'publish'
publishWebProjects: false
projects: '$(workingDirectory)/ConsumerBackEnd.csproj'
arguments: '-c Release -r linux-x64 --self-contained true --output $(System.DefaultWorkingDirectory)/publish_output'
#requires approval on pipeline
- stage: DeployDev
displayName: DeployDev
dependsOn: Build
condition: succeeded()
jobs:
- deployment: DeployConsumerBackendAPIDev
displayName: DeployConsumerBackendAPIDev
environment: ConsumerBackendAPIDev
pool:
name: $(poolName)
strategy:
runOnce:
deploy:
steps:
- task: AzureRmWebAppDeployment#4
inputs:
ConnectionType: 'AzureRM'
azureSubscription: '$(azureRegisteredApp)'
appType: 'webAppLinux'
WebAppName: 'my-backend-dev'
packageForLinux: '$(System.DefaultWorkingDirectory)/publish_output/**/*.zip'
RuntimeStack: 'DOTNETCORE|LTS --configuration Release'
it appears that referencing group variable using runtime expression does not work with file transform task but macro syntax works fine
Microsoft documentation does not describe it well
so here is how it should be defined:
variables:
#secrets
- group: ConsumerDevVariableGroup
- name: Graph.GraphAppTenantId
value: $(GraphAppTenantId) #works fine
- name: 'Graph.GraphAppClientId'
value: '$[variables.GraphAppClientId]' #does not work
Azure DevOps file transform with vault secrets
To achieve this, you could use the task Replace Tokens to update the key's values with Azure vault secrets.
As test, I created a vault secret LeoVar1 in the Azure portal with test value 123456:
Then connect Azure key vault into azure devops pipeline.
Now, I use the Replace Tokens to replace the value of key webpages:Version in the web.config file with the vault secret in the Azure vault #{LeoVar1}#:
My test web.config file:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<connectionStrings>
<add name="DefaultConnection"
connectionString="Data Source=(LocalDb)\\MSDB;DbFilename=aspcore-local.mdf;" />
</connectionStrings>
<appSettings>
<add key="webpages:Version" value="#{LeoVar1}#" />
<add key="webpages:Enabled" value="false" />
</appSettings>
<system.web>
<authentication mode="None" />
<compilation targetFramework="4.5" debug="true" />
</system.web>
</configuration>
You will see the replace are finished successfully after this task ended:
You could check this document Store the app secrets in Azure Key Vault and use during Azure Pipelines for some details.
Update:
I just wandering why FileTransform#2 does not work with secrets
To make it work, you should make sure following things:
The path of appsettings.json is correct. Makre sure it under the folder $(workingDirectory).
The important is that make sure the name of the vault secret is the combination of the attribute name and its parent node.
For example, if we want change the DebugMode from enabled to disable in following json file:
{
"Data": {
"DefaultConnection": {
"ConnectionString": "Data Source=(LocalDb)\\MSDB;AttachDbFilename=aspcore-local.mdf;"
},
"DebugMode": "enabled",
"DBAccess": {
"Administrators": ["Admin-1", "Admin-2"],
"Users": ["Vendor-1", "vendor-3"]
},
"FeatureFlags": {
"Preview": [
{
"newUI": "AllAccounts"
},
{
"NewWelcomeMessage": "Newusers"
}
]
}
}
}
We need defined the name of the vault secret to Data.DebugMode instead of DebugMode.
However, the point . is not supported by vault secret, it will give the error:
Please provide a valid secret name. Secret names can only contain
alphanumeric characters and dashes.
If we test it with Variables in the pipeline, it works without any error:
Since the point . is not supported by vault secret, we could NOT replace the properties with parent node. That the reason why I suggest you use replace token task at first.

provide credentials in nuget command line parameters

We have an nuget server hosted in azure that only allow certain people to create packages.
If I run nuget command line, I get asked to provide username and password. Is there a possibility to provide username and password in command line?
I tried something such as -User or -UserName, but I get Unknown option: '-User'
nuget push -Source "XXX" -ApiKey VSTS -noninteractive mypackage.*.nupkg -User myUsername
Is there a way to provide credentials from command line?
You have to change the NuGet.config you use. That is possible from the command line:
nuget sources add -Name "MyExample" -Source "XXX" -username "Y" -password "Z"
The password will be stored encrypted in the config file and can only be decrypted in the same user context as it was encrypted.
Use PAT (Personal Access Token) as the password. Make sure that PAT has packaging read and write permission.
If you're using an Azure Pipelines build file and trying to push to a private feed, you can use these snippets, just make sure to update $VARIABLES and indentation as-needed:
- task: NuGetAuthenticate#0
displayName: "Auth NuGet"
name: Nuget_Authenticate
inputs:
nuGetServiceConnections: '$PROJECT_SETTINGS_SERVICE_CONNECTION_NAME'
forceReinstallCredentialProvider: true
- task: NuGetCommand#2
name: pushNuGetPkg
displayName: "Push NuGet"
inputs:
command: 'push'
packagesToPush: '$(Build.Repository.Localpath)\Packages\*.nupkg'
nuGetFeedType: 'external'
publishFeedCredentials: '$ARTIFACTS_FEED_NAME'
publishVstsFeed: '$AZURE_ORGANIZATION_NAME/$ARTIFACTS_FEED_NAME'
You may also need to set these env-vars up in your ADO Project's Library -> Defaults section. Make sure the second one is securely masked with the Lock icon.
NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED: true
VSS_NUGET_EXTERNAL_FEED_ENDPOINTS: {"endpointCredentials": [{"endpoint":"https://$AZURE_ORG_NAME.pkgs.visualstudio.com/_packaging/$ARTIFACTS_FEED_NAME/nuget/v3/index.json", "username":"$USERNAME", "password":"$PAT"}]}
For Using PAT(Personal Access Token) As Credential, you can use this code below:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageSources>
<clear />
<add key="ChequeNuget" value="http://192.168.11.23:6700/DefaultCollection/Framework/_packaging/ChequeNuget/nuget/v3/index.json" />
</packageSources>
<packageSourceCredentials>
<ChequeNuget>
<add key="Username" value="ChequeNuget" />
<add key="Password" value="AQAAANCMnd8BFdERjHoAwE/Cl+sBAAAAFyfr+GavvkSgtlD+6FbrwQAAAAACAAAAAAAQZgAAAAEAACAAAABHbg6o2T6HHgjyud+purjkcwLoK1OH2tQI3TZc3T6wfgAAAAAOgAAAAAIAACAAAADnEEVK46WCJAwou/6Gb2ZudmbEJoPoFUqqyK32yf1OYUAAAABlnx7sfP37M/Kblf7VTN2lD3kTiKjRdr0GPCtrLBxGE6pXCbnBY/E1vIrP8S2Ci8HkMW6SgnSkvSACfliZWzoDQAAAAI0xvaQX2XsnjOuDExXf+tLzcdsDK9ZJNen/f4oMbWejc/zytb7D+oczJmrmRbvKBLBC9nGlK2/JfVZii39euMU=" />
<add key="ValidAuthenticationTypes" value="basic" />
</ChequeNuget>
</packageSourceCredentials>
</configuration>
For updating password use the command below:
nuget sources update -ValidAuthenticationTypes basic -Name ChequeNuget -UserName "ChequeNuget" -Password [Personal Access Token]