I am trying to create a secret runtime parameter as described in GitHub. But getting the below error.
Encountered error(s) while parsing pipeline YAML:
/azure-pipelines.yml (Line: 12, Col: 3): Unexpected value 'secret'
Code:
parameters:
- name: app
type: string
values:
- App1
- App2
- name: mySecret
type: string
secret: true
I have raised the same in
Developer Community portal as well. Please let me know if there is any way possible.
Unable to define runtime parameters with secret in azure YAML
Indeed, I could also reproduced this issue on my side. That because that document is a Design Docs.
You could check the state of that doc:
The design docs within this repo are created at different times during
the development of Azure Pipelines, to support collaborative
contributions to the design process. Designs documents are for,
features considered for implementation but never implemented
already implemented features
future ideas for features
The design docs in this repo may not represent the current state of an Azure Pipelines feature.
Obviously, this is a function that has not yet been implemented. That the reason why you get that error. You can expect it to come, I believe it will meet us soon.
Hope this helps.
Related
We are working on creating a custom azure policy with DiNE effect to deploy certain resources based on the tags existence.
Looking at the Built-in policies for tagging, it seems there is a way to determine the tag existence and run deployments using azure policies.
Due to our internal deployments and SDPs, I need to convert the custom policy definition to Bicep and then deploy it.
I have installed Bicep tools/ VS Code and hitting below errors during conversion.
Can you share inputs on how to handle these? Or there any extended production documentation on how to use Scope and resource functions in Biceps as the current product documentation doesn’t have these details?
Issue: Unable to check the subscription tag existence in Bicep azure policy definition file.
[enter image description here] (https://i.stack.imgur.com/MKcGj.jpg)
Policy definitions for tagging resources - Azure Resource Manager | Microsoft Learn
https://portal.azure.com/#blade/Microsoft_Azure_Policy/PolicyDetailBlade/definitionId/%2Fproviders%2FMicrosoft.Authorization%2FpolicyDefinitions%2Fb27a0cbd-a167-4dfa-ae64-4337be671140
Tried to escape the subscription function check but it is not working in bicep file.
I have created a data factory with a pipeline moving data from storage account to azure sql.
Company advised me to use a managed private endpoint to create connection with azure sql.
Scenario:
I have a Dev resource group where my storage account, data factory and sql sit and a Sit resource group where Sit resources sit. I have created managed private endpoint in both data factories with same name, but pointing to different sql servers.
sql_mpe: /subscriptions/123456789/resourceGroups/rg-dev/providers/Microsoft.Sql/servers/dev-sql-server
sql_mpe: /subscriptions/123456789/resourceGroups/rg-sit/providers/Microsoft.Sql/servers/sit-sql-server
As you can see managed private endpoint created has the same name but pointing to different sql servers based on the environment.
Now when I publish the dev adf to azure git, it takes the dev managed private endpoint keys as parameters as follows:
-sql_mpe_properties_privateLinkResourceId "/subscriptions/123456789/resourceGroups/rg-sit/providers/Microsoft.Sql/servers/sit-sql-server"
-sql_mpe_properties_groupId "sqlServer"
-sql_mpe_properties_ipAddress {}
-sql_mpe_properties_resourceId "/subscriptions/987654321/resourceGroups/vnet-45645632-UKSouth-567-rg/providers/Microsoft.Network/privateEndpoints/sit-sql-server.sql_mpe"
For some weird reason, in privateLinkResourceId, resource group and subscription are correct but in resourceId, they are weird values. I don't where they come from hence can't comment on it.
Now when I run my release pipeline, I get the following error:
2022-03-14T15:33:41.5334804Z ##[error]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.
2022-03-14T15:33:41.5366078Z ##[debug]Processed: ##vso[task.issue type=error;]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.
2022-03-14T15:33:41.5373551Z ##[error]Details:
2022-03-14T15:33:41.5374630Z ##[debug]Processed: ##vso[task.issue type=error;]Details:
2022-03-14T15:33:41.5376732Z ##[error]ManagedPrivateEndpointInvalidPayload: Managed private endpoint 'sql_mpe' is invalid.
Error is very generic, hence I went through the docs to understand it. I found the below reason from azure doc Best practices for CI CD:
If a private endpoint already exists in a factory and you try to
deploy an ARM template that contains a private endpoint with the same
name but with modified properties, the deployment will fail.
So I got to know that if you deploy managed private endpoint with same name but different modifies properties (like my sit endpoint is pointing to sit), it will fail.
So now I know why pipeline is failing.
I have to fix this issue for a successful release.
Below are my possible options that I can go with , but don't know how to ? This is where I require some help/ assistance:
resourceId value needs to be understood and changed for SIT (I mentioned some weird values are getting there, and in template, I am just overriding the 'dev' part to 'sit'. I am not changing the vnet resource group and other values.
Remove managed private endpoint parameters from template before publishing to azure git or remove them before creating a release. If I release them in pipeline, error is caused.
Need some insight and help here.
I hope you are all well!
I need to ask a question about azure devops, I already read the documentation, but I did not find a way to resolve these doubts
I have the X, Y and Z projects and in the X project create in the Pipeline >> Libray a group of variables called general that I would like to be shared with the Y and Z pipelines, when configuring this group I enabled the option "Allow Access to all pipelines" .
In the YAML of the Y and Z pipelines I made the following configuration:
**variables:
group: general**
When running the pipeline he returns an authorization request and even clicking authorize it, according to the print below:
Print Authorization error
There was a resource authorization issue:
"An error occurred while loading the YAML build pipeline.
Variable group was not found or is not authorized for use.
For authorization details, refer to https://aka.ms/yamlauthz."
My question is, is there any way to share variables between pipelines of different purposes, if there is, can you please send me some documentation that can help me to configure this?
Testing in my side and I can reproduce this issue, setting the Allow access to all pipelines option will enable the variable group to be accessible for any pipelines in current project not across projects. It should be a known limitation and this is the feedback ticket. You can vote and follow this ticket. You can also create a new suggestion ticket here. The product group will review these tickets regularly, and consider to take it as roadmap.
In addition, as a workaround, you could try to add these shared variables to Azure Key Vault, and then use Azure Key Vault task to fetch the latest values of all or a subset of secrets from the vault, and set them as variables that can be used in subsequent tasks of a pipeline. See: Use Azure Key Vault secrets in Azure Pipelines for details.
This link gives a good overview of using variables in Azure DevOps pipelines:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables
In the most common case, you set the variables and use them within the
YAML file.
In the YAML file, you can set a variable at various scopes:
At the root level, to make it available to all jobs in the pipeline.
At the stage level, to make it available only to a specific stage.
At the job level, to make it available only to a specific job.
Q: It sounds like you would like to share variables between YAML pipeline roots, correct? You're currently trying to use variable groups to accomplish this, correct?
Another solution would be to have your pipeline read from a "shared file". For example:
Is there a way to read file from Azure DevOps YAML?
... you can use any scripting language you like to parse the file and
"spit out" whatever you need as a build variable and consume it later
on. here's what I've been doing:
- script: echo "##vso[task.setvariable variable=dp]$(cat $(Build.Repository.LocalPath)/deployment/dp)"
- script: az group delete -n $(dp)-k8s -y --no-wait
In other words:
Create a file with the variables you wish to "export". It can be any script format: Powershell, bash, etc., etc.
Modify your pipeline(s) to read the file and "import" the variable definitions at runtime.
There are many ways to do this. The SO link above is just an example.
Question based on this document: https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#set-up-an-azure-pipelines-release
When I want to deploy to a data factory from an arm template built from my dev data factory environment, I have an issue with my IR that uses "AAD authentication with the managed identity for your ADF": indeed, in this case the catalogAdminUserName is set to null. Therefore, I get this error at deployment from my release pipeline:
The provided value for the template parameter
'XXXX_properties_typeProperties_ssisProperties_catalogInfo_catalogAdminUserName'
at line '1' and column '4053' is not valid.'
Any idea about how to fix this issue ?
Thank you!
I'm trying to create an sqs queue with cloudformation but I keep getting this error in the console.
API: sqs:CreateQueue Access to the resource https://sqs.us-east-1.amazonaws.com/ is denied.
Obviously I'm missing some sort of permission. This guide didn't really specify how I could resolve this.
Here's the code I made:
AWSTemplateFormatVersion: "2010-09-09"
Resources:
MyQueue:
Type: AWS::SQS::Queue
Properties:
FifoQueue: false
QueueName: sqs-test
ReceiveMessageWaitTimeSeconds: 20
RedrivePolicy:
deadLetterTargetArn:
Fn::GetAtt:
- "MyDLQ"
- "Arn"
maxReceiveCount: 4
Tags:
-
Key: "ProjectName"
Value: "project-x"
MyDLQ:
Type: AWS::SQS::Queue
Properties:
FifoQueue: false
QueueName: sqs-dlq-test
I'm trying to understand this doc. But I'm not sure how I could attach a policy to allow creation of queues. Someone please give me a full example.
tyron's comment on your question is spot on. Check permissions of the user executing the CloudFormation. If you're running commands directly, this is usually pretty easy to check. In some cases, you may be working with a more complicated environment with automation.
I find the best way to troubleshoot permissions in an automated world is via CloudTrail. After any API call has failed, whether from the CLI, CloudFormation, or another source, you can look up the call in CloudTrail.
In this case, searching for "Event Name" = "CreateQueue" in the time range of the failure will turn up a result with details like the following:
Source IP Address; this field may say something like cloudformation.amazonaws.com, or the IP of your machine/office. Helpful when you need to filter events based on the source.
User name; In my case, this was the EC2 instance ID of the agent running the CFN template.
Access Key ID; For EC2 instances, this is likely a set of temporary access credentials, but for a real user, it will show you what key was used.
Actual event data; Especially helpful for non-permissions errors, the actual event may show you errors in the request itself.
In my case, the specific EC2 instance that ran automation was out of date and needed to be updated to use the correct IAM Role/Instance Profile. CloudTrail helped me track that down.
If you are using AWS CodePipeline (where you may be using AWS CodeBuild to run & deploy your CloudFormation stack), remember your CodeBuild role (created under IAM Roles) must have the correct permissions.
You can identify which role is being used & attach required policies -
Open CodeBuild Project
Go to Build Details > Environment > Service Role
Open Service Role (hyperlinked)
Add SQS to role policies