I have an environment.yml shown as follow, I would like to read out the content of the name variable (core-force) and set it as a value of the global variable in my azure-pipeline.yamal file how can I do it?
name: core-force
channels:
- conda-forge
dependencies:
- click
- Sphinx
- sphinx_rtd_theme
- numpy
- pylint
- azure-cosmos
- python=3
- flask
- pytest
- shapely
in my azure-pipeline.yml file I would like to have something like
variables:
tag: get the value of the name from the environment.yml aka 'core-force'
Please check this example:
File: vars.yml
variables:
favoriteVeggie: 'brussels sprouts'
File: azure-pipelines.yml
variables:
- template: vars.yml # Template reference
steps:
- script: echo My favorite vegetable is ${{ variables.favoriteVeggie }}.
Please note, that variables are simple string and if you want to use list you may need do some workaraund in powershell in place where you want to use value from that list.
If you don't want to use template functionality as it is shown above you need to do these:
create a separate job/stage
define step there to read environment.yml file and set variables using REST API or Azure CLI
create another job/stage and move you current build defitnion into there
I found this topic on developer community where you can read:
Yaml variables have always been string: string mappings. The doc appears to be currently correct, though we may have had a bug when last you visited.
We are preparing to release a feature in the near future to allow you to pass more complex structures. Stay tuned!
But I don't have more info bout this.
Global variables should be stored in a separate template file. This file ideally would be in a separate repo where other repos can refer to this.
Here is another answer for this
Related
I am actually implementing CI/CD for my application. I want to start the application automatically using pm2. So I am getting the syntax error on line 22.
This is my yml file
This is the error I am getting on github
The problem in the syntax here is related to how you used the - symbol.
With Github actions, you need at least a run or uses field inform for each step inside your job, at the same level of the name field (which is not mandarory), otherwise the github interpreter will return an error.
Here, from line 22, you used something like this:
- name: ...
- run: ...
- run: ...
- run: ...
So there are two problems:
First, the name and the run field aren't at the same yaml level.
Second, your step with the name field doesn't have a run or uses field associated with it (you need at least one of them).
The correct syntax should be:
- name: ...
run: ...
- run: ...
- run: ...
Reference about workflow syntax
I am having trouble getting a deployment job in a template to expand a variable it is given via a parameter. Ive used some short hand stuff below.
If you want to see the code, there is a prototype that shows the problem at https://github.com/ausfestivus/azureDevOpsPrototypes
The pipeline looks like this:
stage00
buildjob00
task produces output vars (name: taskName.VAR_NAME)
buildjob01
task is able to reference the variable and retrieve/display the variable value via
dependency notation. [dep.buildjob00.taskName.VAR_NAME]
template:
parameters:
bunchOfVarsAsSequenceFormat:
var1: [dep.buildjob00.taskName.VAR_NAME]
var2: [dep.buildjob00.taskName.VAR_NAME]
template contains:
buildjob02
this build job will see the variables values fine
deplomentjob00
this deploy job will see the variable names but contain empty values
Apologies if this is not well explained, hopefully the above prototype helps illustrate it better than the pseudo code above.
What a super help you shared your YAML scripts here! Otherwise, it's too difficult to understand your structure:-)
To display the variable in tmpl: deploy, you need change its corresponding dependsOn as job00, rather than templateJob.
- deployment: templateDeploy
displayName: 'tmpl: deploy'
continueOnError: false
dependsOn: job00
Then you would see the value could display successfully:
We are currently using cloud formation to create a glue job (via codebuild and codepipeline). The one thing we are stuck on is how to automate the code that goes into the glue job.
Our current relevant piece of the cloudformation template looks like this:
MyJob:
Type: AWS::Glue::Job
Properties:
Command:
Name: glueetl
ScriptLocation: "s3://aws-glue-scripts//your-script-file.py"
DefaultArguments:
"--job-bookmark-option": "job-bookmark-enable"
ExecutionProperty:
MaxConcurrentRuns: 2
MaxRetries: 0
Name: cf-job1
Role: !Ref MyJobRole
The problem is is the "ScriptLocation". Looks like it is required to be an S3 location. How can we automate the upload of this. The code is in a .py file in our Git repository and I assume is uploaded to the artifact repository as are of the codebuild process, but how to access it?
Would like to hear how others are doing this. Thanks!
EDIT: I was able to find a similar stack overflow post:AWS Glue automatic job creation but it the answers really don't give a solution or understand the question posed.
I've written a tool to handle the upload of stack dependencies, including CloudFormation nested templates and non-inline Lambda functions.
Currently AWS Glue was not handled since I haven't try it in any project yet. But it should be easy to expand to support Glue.
The dependencies were defined in separate config file, and a piece of code within the tool is responsible for the config. Here's the sample config:
Nested CloudFormation templates:
# DEPENDS=( <ParameterName>=<NestedTemplate> )
#
# Required: Yes if has nested template, otherwise No
# Default: None
# Syntax:
# <ParameterName>: The name of template parameter that is referred at the
# value of nested template property `TemplateURL`.
# <NestedTemplate>: A local path or a S3 URL starting with `s3://` or
# `https://` pointing to the nested template.
# The nested templates at local is going to be uploaded
# to S3 Bucket automatically during the deployment.
# Description:
# Double quote the pairs which contain whitespaces or special characters.
# Use `#` to comment out.
# ---
# Example:
# DEPENDS=(
# NestedTemplateFooURL=/path/to/nested/foo/stack.json
# NestedTemplateBarURL=/path/to/nested/bar/stack.json
# )
Lambda functions:
# LAMBDA=( <S3BucketParameterName>:<S3KeyParameterName>=<LambdaFunction> )
#
# Required: Yes if has None-inline Lambda Function, otherwise No
# Default: None
# Syntax:
# <S3BucketParameterName>: The name of template parameter that is referred
# at the value of Lambda property `Code.S3Bucket`.
# <S3KeyParameterName>: The name of template parameter that is referred
# at the value of Lambda property `Code.S3Key`.
# <LambdaFunction>: A local path or a S3 URL starting with `s3://` pointing
# to the Lambda Function.
# The Lambda Functions at local is going to be zipped and
# uploaded to S3 Bucket automatically during the deployment.
# Description:
# Double quote the pairs which contain whitespaces or special characters.
# Use `#` to comment out.
# ---
# Example:
# DEPENDS=(
# S3BucketForLambdaFoo:S3KeyForLambdaFoo=/path/to/LambdaFoo.py
# S3BucketForLambdaBar:S3KeyForLambdaBar=s3://mybucket/LambdaBar.py
# )
The tools were written in bash and come with 2 parts:
xsh: It works as a bash library framework.
xsh-lib/aws: It's a library of xsh.
The code you may need to expand is located in xsh-lib/aws/functions/cfn/deploy.sh.
The example deploy command looks like:
$ xsh aws/cfn/deploy -C /path/to/your/template-and-config-dir -t stack.json -c sample.conf
I'm considering to abstract the dependencies such as CloudFormation template, Lambda functions and Glue, into a single interface for both configs and handlers.
This will make it easier to add new dependency handlers to the deployer.
I created a new build with Azure Pipelines (Azure DevOps) and it worked really well.
Usually, you use $(Rev:.r) to get the revision in the build. Unfortunately, it seems the variable isn't replaced/set in the build steps. The only place where you can use it is the name: property in the YAML document.
Now I set it in the name and extract it in some PowerShell, which isn't necessary if you can get it via an environment variable.
How do I get the Revision (like $(Rev)) in the new builds (outside of the name: property in the YAML document)?
(The Build Agents running on-premise, inside Docker - but this shouldn't affect the things above)
You can't get the revision number without parsing, it is not stored as a separate field somewhere or in an environment variable.
The $(Rev:.r) portion instructs Azure DevOps to come up with the first number that makes the build number unique (and, in that specific example, put a dot in front of it).
Like you said, the only way is to use PowerShell script to get the value:
$buildNumber = $Env:BUILD_BUILDNUMBER
$revision= $buildNumber.Substring($buildNumber.LastIndexOf('.') + 1)
Edit:
You can install the Get Revision Number extension that does it.
Another possible solution to the above problem could be to use counter expression for ex: we difine the variable and use it in a task to build nuget package.
variables:
counterVar: $[counter($(versionVariable),0)]
.......
- task: CmdLine#2
inputs:
script: >
nuget pack ClassLibrary1/ClassLibrary1.csproj
-OutputDirectory $(Build.ArtifactStagingDirectory)
-NonInteractive
-Properties Configuration=release
-Version $(versionVariable).$(counterVar)
-Verbosity Detailed
-IncludeReferencedProjects
Here versionVariable is a custome variable defined in pipelines->variables.And the seed value is 0(2nd param to counter).
It works as below
Let's assume the versionVariable is 1.19
Build Run 1 counterVar will be 0.
Build Run 2 counterVar will be 1.
Now say we change the versionVariable to 1.20
Build Run 3 counterVar will be 0.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops
Check the counter expression in above link it reset its value for diff prefix.
P.S. Benefit of using counter over $(Rev:r) is that it can start from 0 unlike $(Rev:r)
I have Ansible role, for example
---
- name: Deploy app1
include: deploy-app1.yml
when: 'deploy_project == "{{app1}}"'
- name: Deploy app2
include: deploy-app2.yml
when: 'deploy_project == "{{app2}}"'
But I deploy only one app in one role call. When I deploy several apps, I call role several times. But every time there is a lot of skipped tasks output (from tasks which do not pass condition), which I do not want to see. How can I avoid it?
I'm assuming you don't want to see the skipped tasks in the output while running Ansible.
Set this to false in the ansible.cfg file.
display_skipped_hosts = false
Note. It will still output the name of the task although it will not display "skipped" anymore.
UPDATE: by the way you need to make sure ansible.cfg is in the current working directory.
Taken from the ansible.cfg file.
ansible will read ANSIBLE_CONFIG,
ansible.cfg in the current working directory, .ansible.cfg in
the home directory or /etc/ansible/ansible.cfg, whichever it
finds first.
So ensure you are setting display_skipped_hosts = false in the right ansible.cfg file.
Let me know how you go
Since ansible 2.4, a callback plugin name full_skip was added to suppress the skipping of task names and skipping keyword in the ansible output. You can try the below ansible configuration:
[defaults]
stdout_callback = full_skip
Ansible allows you to control its output by using custom callbacks.
In this case you can simply use the skippy callback which will not output anything on a skipped task.
That said, skippy is now deprecated and will be removed in ansible v2.11.
If you don't mind losing colours you can elide the skipped tasks by piping the output through sed:
ansible-playbook whatever.yml | sed -nr '/^TASK/{h;n;/^skipping:/{n;b};H;x};p'
If you are using roles, you can use when to cancel the include in main.yml
# roles/myrole/tasks/main.yml
- include: somefile.yml
when: somevar is defined
# roles/myrole/tasks/somefile.yml
- name: this task will only run (and be seen in the output) if somevar is defined
debug:
msg: "Hello World"