What is the difference between linked task parameters (process parameters) and variables in classic Azure DevOps build pipeline? Don't they all allow having a single place where to change values?
What I mean by "linked" task parameters are what you get by clicking the link icon when configuring tasks like below
which leads to adding a textbox for the linked value in settings page for the pipeline as you see below
Regarding parameters in the classic pipeline, we generally use Process parameters. You can link all important arguments for tasks used across the build definition as process parameters, which are then shown at one place-the Pipeline view. This means you can quickly edit these arguments without needing to click through all the tasks. Templates come with a set of predefined process parameters.
Variables give you a convenient way to get key bits of data into various parts of the pipeline. The most common use of variables is to define a value that you can then use in your pipeline. All variables are stored as strings and are mutable. The value of a variable can change from run to run or job to job of your pipeline.
The difference between them is：
Variables can be a convenient way to collect information from the
user up front. You can also use variables to pass data from step to
step within a pipeline.Unlike variables, pipeline parameters can't be
changed by a pipeline while it's running.
Parameters have data types such as number and string, and they can be
restricted to a subset of values. Restricting the parameters is
useful when a user-configurable part of the pipeline should take a
value only from a constrained list. The setup ensures that the
pipeline won't take arbitrary data.
Process parameters differ from variables in the kind of input supported by them. Variables only take in string inputs while process parameters in addition to string inputs support additional data types like check boxes and drop-down list boxes.
For detailed information, please refer to the following documents:
Define variables
Process parameters
Variables and parameters
Related
I'm working on an integration between Azure Pipelines and ServiceNow's change management module. To achieve that the ServiceNow Change Management extension has been installed and configured according to this documentation page. In Azure DevOps we are using multistage yaml pipelines, which should create standard preapproved changes in ServiceNow.
The connection itself between the two applications works fine, I managed to put together a pipeline that creates change requests, waits until their status changes and then closes them. However, I'd like to pass some values set in the pipeline runs to the created change requests and I couldn't find a way to do it.
First I added a service connection to our Azure DevOps project, and created the ServiceNow check for it. I experimented a little with adding different expressions to it, like setting the short description to ${{ parameters.shortDescription }}, or defining a variable in the pipeline as ShortDescription: ${{ parameters.shortDescription }} and using that variable in the check as $(ShortDescription) or $[ variables.ShortDescription ]. Unfortunately none of these expressions got resolved. I also realized it is possible to use the predefined variables, but the values I'd like to set are not possible to describe by predefined variables. For example, selecting an assignment group would be pretty straightforward from a parameter defined as a list, but impossible to select from predefined variables.
So as a next idea, I tried to link a variable group to the check and update the variables through logging commands. Even though the variables from the group got resolved, they only showed the values I set them through the UI as a static default value. The dynamic values set via the logging commands were not visible. I played around some time and verified that I can update the definition of the variable groups through Azure CLI or REST API, so I can add new variables or update existing ones. Thus I tried to add a new variable to the linked group during the pipeline run named as ShortDescription_$(Build.BuildId). Even though it got added properly, I could not use it within the check, because it required double variable resolution, like $(ShortDescription_$(Build.BuildId)) and this expression was not resolved, not even partly. It remained $(ShortDescription_$(Build.BuildId)).
Then I started thinking about using only one variable from the group with a static name (e.g. ShortDescription) for all pipeline runs. However, I feel it would create a race condition and could cause some inconsistencies.
So as a last resort, I tried to put together an extension with an Agent and a ServerGate task, which are capable of storing the values I want to pass to change request and reading the stored values in an agentless environment. The problem here is, that the second task is not visible as a check for service connections. It's there as a release pipeline gate and looks good there, but I can't utilize it that way. Based on a question I found, this does not seem to be the problem with my task. To verify it, I copied the content of the same ServiceNow check I used before, and added it to my extension as a contribution with a different task id. And it did not show up as the question stated.
Which means now I can either
create a change request through my custom server task (as the ServerGate task can be used properly in yaml if it is changed to a Server task), but that way I can't wait for the state change of the ServiceNow ticket, or
create the change request in a separate stage where I want to use it, update it first in the same stage where I created it via the first-party check and wait for the state change in the stage where I would normally create it.
The second can work, but it has its own problems, like having misleading values stored in the changed request for the stage id field, or not having multiple change requests created for multiple run attempts of the deployment stage. Also I feel like it's not how the extension's task and check should be used.
Unfortunately, I'm out of ideas how this dynamic value passing can be achieved, if it's possible to do so in the first place. Could you please help me by sharing ideas, or pointing out errors in my attempts?
I have a switch statement that looks at a variable value and based on that determines which data flow to execute. The problem with this is, I need to update the switch statement every time I add a new ID/dataflow.
Is there an alternative design to this? What if my dataflows had the same name as the variable value - would it be possible to parameterize the name of the dataflow to execute?
e.g. variable value = "1" execute data flow with name "1_dataflow", 2 execute "2_dataflow" etc. How would I accomplish this?
Yes, you can parameterize the values in any activity in Azure Data Factory and make the pipeline dynamic instead of giving hard-coded values.
You can use parameters to pass external values into pipelines,
datasets, linked services, and data flows. Once the parameter has been
passed into the resource, it cannot be changed. By parameterizing
resources, you can reuse them with different values each time.
You can also use Set Variable activity in to define and variable and set value to it which you can use in Switch activity and also can change it later.
Refer: How to use parameters, expressions and functions in Azure Data Factory, Set Variable Activity in Azure Data Factory and Azure Synapse Analytics
I have 2 build pipelines in my azure devops project, one for building source code and the other one is for
making the setup.
I want the build number generated by the first pipeline that compiles code to be passed to the next pipeline which creates the setup file because i want the setup file to take the same version, so I added a variable group with a variable called sharedBuildCounter.
But when I set sharedBuildCounter the build number in the first pipeline using logging command like this(used inside PowerShell task):
Write-Host "##vso[task.setvariable variable=variable_name;]new_value"
The variable indeed takes the new value and I am able to output the new value using another PowerShell task with one line:
Write-Host $(SharedBuildCounter)
And when I run the next pipeline that builds the setup, I find that sharedBuildCounter is being re-set to the default empty value.
Notice: I found threads that suggests using API rest calls to change variable values, but it don't seem to include a specific pipeline name in case of using pipeline variables(not variable groups).
Variable groups will help to share static values across builds and releases pipeline.
What you need is a way to pass variables from one pipeline to another. I'm afraid to say the is no official way to do this.
As a workaround you could update the value of your variables inside your variable group. There are multiple ways to handle this, Rest API, powershell, 3rd-party extension. Detail ways please refer answers in this question: How to Increase/Update Variable Group value using Azure Devops Build Definition?
If you want to get the value of variable in the pipeline. Since you have used logging command to update that variable.
You need to use Rest API to get that particular build log to fetch related info.
You can use Azure Artifacts to pass information between pipelines. In one pipeline, you write the values to a file and publish the file to an artifact. In the other pipeline, you download the artifact and read the file.
There may be other ways to do it. Azure DevOps allows for free and infinite use of Azure Artifacts in this fashion.
See How to get variable values from pipeline resources in azure pipelines.
I am using the release pipeline classic editor and have a requirement of passing an output variable generated in a task to multiple jobs in the same stage or to outside stages. Currently, this output variable is available only inside the same job and I have to write the same task in multiple jobs and stages and I feel it is a redundancy. Is there any way to implement it?.
In the Classic editor, I am afraid that output variables in a different job is not feasible. Please refer to this document .
As a workaround , you can use variables across jobs and stages via variable groups.
First define the variable in the variable group, then update the variable group through rest api or azure cli, and replace the defined variable with the value of the variable generated by the task.
PUT https://dev.azure.com/{organization}/{project}/_apis/distributedtask/variablegroups/{groupId}?api-version=5.1-preview.1
Here is a case about update variable group with powershell script.
Another workaround : You can share values across all of the stages by using release pipeline variables. The solution is updating the Release Definition for the Release Pipeline variable in the Stage where the variable is set.
Define a variable in the release definition Variable.
Use REST API Definitions - Update to update the value of the
release definition variable in the agent job.
Use the updated value of the release definition variable in the next
agent job.
The details info about using REST API to update the value of the release definition variable, you can follow this ticket.
For detailed steps and guide, please refer to this blog .
I am looking for help for below 2 Dev Ops related topics
1) I am frequently getting warning on my Log "Environment variable 'VSTS_PUBLIC_VARIABLES' exceeds the maximum supported length"
I wanted to see which line of code/configuration/ parameters length is exceeding overall. is there any way to pin point code/ parameter etc to fix
2)I have heaps of variables under variable groups under different projects and many variables seems not in use being part of poor process. I want to trace each variable incase linked with a pipeline. or any other way to cleaning up unwanted variables
About the first question: This is windows environment limitation. We can’t deal with it in Azure DevOps. You could try to use shorter folder name and fewer levels. You can refer to this case about this .
About the second question: As far as I know, there is currently no out-of-the-box feature to achieve it. We should only exclude unused variables by searching one by one or use scripts to retrieve variables that match the variable names in the variable group.
The variable group works on multiple pipelines, and the variables may not be used in the current pipeline, but may be used in other pipelines. At present, there should be no such function to check which variables are used in the corresponding pipeline.