I am pushing an ADF factory to another environment via a CICD Pipeline and YAML Config file in Azure Devops. I can successfully deploy but one of my linked services becomes a "bad resource" although it works in the master branch when I published it.
Furthermore I cannot delete this in the target data factory nor can I edit it. Getting the bad resource error. I suspect I need to edit something in the ARM file but I don't really understand this error nor can I find much information on similar.
{"stack":"Error: Error: Unable to save [SERVICENAME]. Bad resource\n at Rl.<anonymous> (https://adf.azure.com/app.06b0e174dd8e6fa8.js:1:11274843)\n at Generator.next (<anonymous>)\n at https://adf.azure.com/main.d1fe4ec6f69aa72f.js:1:66326\n at new c
That when I deploy my ADF to a new environment it succeeds with connections intact or at least that I can fix/edit.
EDIT: Even when I recreate the Linked Service I get the same error.
The answer to this is to store all of your connection credentials as secrets in Azure Keyvault then reference that. I am unclear why using the parameters in a linked service do not transfer into the ARM template and this cause it to be a "bad resource" but the Keyvault method translates into ARM correctly and the problem doesn't persist.
I have azure key vault service in which we are maintaining secrets.
I have to deploy APIM service using ARM JOB in Azure devops release pipeline so I have added this job and added configured template.json and parameter.json and how to pass key vault as over ride parameter to ARM job in over ride parameters?. I tried with below option
I have added keyvault job/varaible group in azure pipelines then in over ride params i called $(keyvaultname/secretname) then saved it and ran the pipeline but i am geeting below issue
enter image description here
Please go to Pipelines -> Library -> create a variable group which contain the keyvalut.
Link the Variable group in your pipeline, make sure the variable of secret is listed.
In the ARM task, overwrite the parameters with "$(var)" name.
PFA .
I have created Variable group and then came back to release pipeline arm job then in the override parameter .
Arm job over ride parameter
Variable group
I have created a data factory with a pipeline moving data from storage account to azure sql.
Company advised me to use a managed private endpoint to create connection with azure sql.
Scenario:
I have a Dev resource group where my storage account, data factory and sql sit and a Sit resource group where Sit resources sit. I have created managed private endpoint in both data factories with same name, but pointing to different sql servers.
sql_mpe: /subscriptions/123456789/resourceGroups/rg-dev/providers/Microsoft.Sql/servers/dev-sql-server
sql_mpe: /subscriptions/123456789/resourceGroups/rg-sit/providers/Microsoft.Sql/servers/sit-sql-server
As you can see managed private endpoint created has the same name but pointing to different sql servers based on the environment.
Now when I publish the dev adf to azure git, it takes the dev managed private endpoint keys as parameters as follows:
-sql_mpe_properties_privateLinkResourceId "/subscriptions/123456789/resourceGroups/rg-sit/providers/Microsoft.Sql/servers/sit-sql-server"
-sql_mpe_properties_groupId "sqlServer"
-sql_mpe_properties_ipAddress {}
-sql_mpe_properties_resourceId "/subscriptions/987654321/resourceGroups/vnet-45645632-UKSouth-567-rg/providers/Microsoft.Network/privateEndpoints/sit-sql-server.sql_mpe"
For some weird reason, in privateLinkResourceId, resource group and subscription are correct but in resourceId, they are weird values. I don't where they come from hence can't comment on it.
Now when I run my release pipeline, I get the following error:
2022-03-14T15:33:41.5334804Z ##[error]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.
2022-03-14T15:33:41.5366078Z ##[debug]Processed: ##vso[task.issue type=error;]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.
2022-03-14T15:33:41.5373551Z ##[error]Details:
2022-03-14T15:33:41.5374630Z ##[debug]Processed: ##vso[task.issue type=error;]Details:
2022-03-14T15:33:41.5376732Z ##[error]ManagedPrivateEndpointInvalidPayload: Managed private endpoint 'sql_mpe' is invalid.
Error is very generic, hence I went through the docs to understand it. I found the below reason from azure doc Best practices for CI CD:
If a private endpoint already exists in a factory and you try to
deploy an ARM template that contains a private endpoint with the same
name but with modified properties, the deployment will fail.
So I got to know that if you deploy managed private endpoint with same name but different modifies properties (like my sit endpoint is pointing to sit), it will fail.
So now I know why pipeline is failing.
I have to fix this issue for a successful release.
Below are my possible options that I can go with , but don't know how to ? This is where I require some help/ assistance:
resourceId value needs to be understood and changed for SIT (I mentioned some weird values are getting there, and in template, I am just overriding the 'dev' part to 'sit'. I am not changing the vnet resource group and other values.
Remove managed private endpoint parameters from template before publishing to azure git or remove them before creating a release. If I release them in pipeline, error is caused.
Need some insight and help here.
I have two yaml templates defined one for creating a docker registry service connection and second for deploying some stuff via container job. The second template uses the docker registry connection which is being deployed in first template. When I am running both the templates separately then both the stages are successful but when I run them in one azure-pipelines.yaml, it fails :
There was a resource authorization issue: "The pipeline is not valid. A service connection with name shared-stratus-acr-endpoint could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."
Is there any way like dependsOn or condition that we can provide in this situation?
It's likely that you only authorized the service connection for the individual template\pipelines when you created them. The workflow is not super friendly.
What if you try and authorize the pipeline that is failing for that service connection explicitly. See docs here
You could also just authorize the service connection for all pipelines depending on your security needs.
I'm trying to create my first release pipeline, however I keep getting this error:
Exception Message: The pipeline is not valid. Job Phase_1: Step AzureResourceGroupDeployment input ConnectedServiceName references service connection
which could not be found. The service connection does not exist or has not been authorized for use. For authorization details,
refer to https://aka.ms/yamlauthz. (type PipelineValidationException)
I've tried to follow the instructions in the link, however the "Authorize Resources" button does not exist.
"Allow all pipelines to use this service connection" is already enabled and I have recreated the deployment task after enabling this.
How do I authorise the resource?
I had the same issue, and I initally missed the fact that you need to click the 'Authorize resources' button that appears, as shown below
Also in my case, my pipeline was missing variables that included the correct service connection name. These were set up in a variable group that was already being used by another pipeline. I needed to link them in my new pipeline:
Edit pipeline > select elipsis at top right > triggers > Variables > Variable groups > Link variable group
You can either use existing Service principal or create a new one. All you need is in documentation already.
Create an Azure Resource Manager service connection using automated security
From Azure DevOps -> Project setting -> Service connection: Then click on "New Service Connection".Choose "Azure resource Manager" as type of service connection. Select Service Principal (Automatic). Run your pipeline
My "Service connection" which defined the service principal connection had been created separately to the task in my release pipeline.
In order for "Authorize Resources" to occur, you must create a new connection from the task itself (you may need to use the advanced options to add an existing service principal).
under "Azure subscription" click the name of the subscription you wish to use
Click the drop down next to "Authorize" and open advanced options
Click " use the full version of the service connection dialog."
Enter all your credentials and hit save
The admittely silly solution for me was to avoid declaring and using the Service Connection as a variable, i.e. in the case of connecting to an Azure Container Registry:
Failed
pool:
vmImage: 'ubuntu-20.04'
variables:
dockerRegistryServiceConnection: 'my-service-connection'
baseContainerUrl: 'myregistry.azurecr.io/my_image:latest'
container:
image: $(baseContainerUrl)
endpoint: $(dockerRegistryServiceConnection)
Worked
pool:
vmImage: 'ubuntu-20.04'
variables:
baseContainerUrl: 'myregistry.azurecr.io/my_image:latest'
container:
image: $(baseContainerUrl)
endpoint: my-service-connection
In "classic release pipelines", a release is a snapshot of a release pipeline, with all the settings of the pipeline materialized, and a deployment is the execution of a pipeline stage in the release.
In our case, a release generating this error during deployment predated a service principal renewal. We could edit the release to use the new service principal and successfully deploy the modified release. The release pipeline did not need modifications.
Navigate to the release overview of the relevant release.
https://dev.azure.com/<user>/<project>/_releaseProgress?...
Select Edit > Edit release.
Click Edit tasks for the relevant stage.
Select the Azure App Service Deploy step.
Choose the correct Azure subscription and App Service name values.
These values will likely be the same as in the release pipeline
In our case these fields were empty and "needed attention".
Click Save.
Click Deploy.
The changes persist, so the corrected release can be deployed a second time without incident.
It is unclear whether this applies to YAML pipelines but I would guess no.
I was having this error because I was declaring the variable group containing the service connection name at stage level.
The error was fixed once I declared the variable group at pipeline level.
In my case, I was trying to use [AzureAppServiceSettings][1] task, and was using
azureSubscription with subscription id in the YAML file.
The YAML file
- task: AzureAppServiceSettings#1
displayName: Update App Settings of the Logic App
inputs:
azureSubscription: '$(azureSubscriptionId)'
resourceGroupName: ...
appName: ...
appSettings: ...
I got the same error and after clicking Authorise resources, I got error of no plan found for identifier xxxx.
Needed to change my YAML file to use connection name in Azure DevOps.
- task: AzureAppServiceSettings#1
displayName: Update App Settings of the Logic App
inputs:
azureSubscription: 'Nonprod Connection'
resourceGroupName: ...
appName: ...
appSettings: ...
So Nonprod Connection is one of my service connections in Azure Devops.
After fixing the value of azureSubscription, the pipeline doesn't show up the error anymore.
I had a similar problem and noticed that adding a task to run an Azure PowerShell script was also getting a similar error.
Turned out that the problem was solved when I verified the service connection twice within the project that was having the problem: