I am trying to deploy Azure Data Factory from development to Test and Production instances using Azure DevOps. The pipeline steps include:
Copying the Linked Templates to a storage account (Azure PowerShell Task).
Disabling the triggers (Azure PowerShell Task).
ARM Template Deployment (Azure Resource Group Deployment Task).
Enabling the Triggers (Azure PowerShell Task).
I have set the override parameters for Test and Production Tasks.
The deployment to the Test Data Factory completed successfully. However, when I tried deploying it to the Production Data Factory, it failed giving the following error :
There were errors in your deployment. Error code: DeploymentFailed.
2022-11-24T14:20:51.2337688Z ##[error]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.
2022-11-24T14:20:51.2339819Z ##[debug]Processed: ##vso[task.issue type=error;]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.
2022-11-24T14:20:51.2340752Z ##[error]Details:
2022-11-24T14:20:51.2341700Z ##[debug]Processed: ##vso[task.issue type=error;]Details:
2022-11-24T14:20:51.2343511Z ##[error]DeploymentFailed: At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.
2022-11-24T14:20:51.2351532Z ##[error]Task failed while creating or updating the template deployment.
2022-11-24T14:20:51.2352528Z ##[debug]Processed: ##vso[task.issue type=error;]Task failed while creating or updating the template deployment.
2022-11-24T14:20:51.2361768Z ##[debug]Processed: ##vso[task.complete result=Failed;]Task failed while creating or updating the template deployment.
I checked the steps for Test and Production tasks, and both seems correct to me. The Resource Group, Data Factory name, template parameters all have been set up.
How do I resolve this, since the error doesn't point to anything specific?
The issue was with my connection to the Shared Integration Runtime for Data Factory.
So, the steps I followed were as follows :
In the Azure Portal, go to your Resource Group -> Activity Log
Here, I found the exact cause of my Data Factory deployment failing.
Access denied. Unable to access shared integration runtime 'integrationRuntimeSelfHosted'. Please check whether this resource has been granted permission by the shared integration runtime.
Then I went to the Data Factory, where I had created the integration runtime and granted access to the new Data Factory which I was trying to deploy.
Go to Azure DevOps -> Deploy the Pipeline again.
Following the above steps helped me resolve my issue.
Related
I am pushing an ADF factory to another environment via a CICD Pipeline and YAML Config file in Azure Devops. I can successfully deploy but one of my linked services becomes a "bad resource" although it works in the master branch when I published it.
Furthermore I cannot delete this in the target data factory nor can I edit it. Getting the bad resource error. I suspect I need to edit something in the ARM file but I don't really understand this error nor can I find much information on similar.
{"stack":"Error: Error: Unable to save [SERVICENAME]. Bad resource\n at Rl.<anonymous> (https://adf.azure.com/app.06b0e174dd8e6fa8.js:1:11274843)\n at Generator.next (<anonymous>)\n at https://adf.azure.com/main.d1fe4ec6f69aa72f.js:1:66326\n at new c
That when I deploy my ADF to a new environment it succeeds with connections intact or at least that I can fix/edit.
EDIT: Even when I recreate the Linked Service I get the same error.
The answer to this is to store all of your connection credentials as secrets in Azure Keyvault then reference that. I am unclear why using the parameters in a linked service do not transfer into the ARM template and this cause it to be a "bad resource" but the Keyvault method translates into ARM correctly and the problem doesn't persist.
I have created a data factory with a pipeline moving data from storage account to azure sql.
Company advised me to use a managed private endpoint to create connection with azure sql.
Scenario:
I have a Dev resource group where my storage account, data factory and sql sit and a Sit resource group where Sit resources sit. I have created managed private endpoint in both data factories with same name, but pointing to different sql servers.
sql_mpe: /subscriptions/123456789/resourceGroups/rg-dev/providers/Microsoft.Sql/servers/dev-sql-server
sql_mpe: /subscriptions/123456789/resourceGroups/rg-sit/providers/Microsoft.Sql/servers/sit-sql-server
As you can see managed private endpoint created has the same name but pointing to different sql servers based on the environment.
Now when I publish the dev adf to azure git, it takes the dev managed private endpoint keys as parameters as follows:
-sql_mpe_properties_privateLinkResourceId "/subscriptions/123456789/resourceGroups/rg-sit/providers/Microsoft.Sql/servers/sit-sql-server"
-sql_mpe_properties_groupId "sqlServer"
-sql_mpe_properties_ipAddress {}
-sql_mpe_properties_resourceId "/subscriptions/987654321/resourceGroups/vnet-45645632-UKSouth-567-rg/providers/Microsoft.Network/privateEndpoints/sit-sql-server.sql_mpe"
For some weird reason, in privateLinkResourceId, resource group and subscription are correct but in resourceId, they are weird values. I don't where they come from hence can't comment on it.
Now when I run my release pipeline, I get the following error:
2022-03-14T15:33:41.5334804Z ##[error]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.
2022-03-14T15:33:41.5366078Z ##[debug]Processed: ##vso[task.issue type=error;]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.
2022-03-14T15:33:41.5373551Z ##[error]Details:
2022-03-14T15:33:41.5374630Z ##[debug]Processed: ##vso[task.issue type=error;]Details:
2022-03-14T15:33:41.5376732Z ##[error]ManagedPrivateEndpointInvalidPayload: Managed private endpoint 'sql_mpe' is invalid.
Error is very generic, hence I went through the docs to understand it. I found the below reason from azure doc Best practices for CI CD:
If a private endpoint already exists in a factory and you try to
deploy an ARM template that contains a private endpoint with the same
name but with modified properties, the deployment will fail.
So I got to know that if you deploy managed private endpoint with same name but different modifies properties (like my sit endpoint is pointing to sit), it will fail.
So now I know why pipeline is failing.
I have to fix this issue for a successful release.
Below are my possible options that I can go with , but don't know how to ? This is where I require some help/ assistance:
resourceId value needs to be understood and changed for SIT (I mentioned some weird values are getting there, and in template, I am just overriding the 'dev' part to 'sit'. I am not changing the vnet resource group and other values.
Remove managed private endpoint parameters from template before publishing to azure git or remove them before creating a release. If I release them in pipeline, error is caused.
Need some insight and help here.
I'm trying to build a Azure Devops pipeline and ran into this error during the run stage. Can you please help me solve this issue? Thank you
There was a resource authorization issue:
The pipeline is not valid. Job Build_Scoring_image: Step buildscoringimage input connectedServiceNameARM references service connection aml-workspace-connection which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz. Job Deploy_to_Staging: Step input kubernetesServiceConnection references service connection mlops-aks which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.
I clicked 'authorize resources' next to the error and it still failed.
We suppose that your issue is could be resolved by re-configure your service connection in Project Setting.
Service Connection in Project Setting
Service Connection Management
And your service connection would be available again in pipelines.
I have created a Docker Compose in my pipeline and Azure created the code. The azureSubscription and the azureContainerRegistry connection are very clear.
I tried to replace them with variable from the Library but when the pipeline starts I immediately get an error.
There was a resource authorization issue: "The pipeline is not valid. Job Build: Step DockerCompose1 input azureSubscriptionEndpoint references service connection $(AzureSubscription) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz. Job Build: Step DockerCompose2 input azureSubscriptionEndpoint references service connection $(AzureSubscription) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."
Basically, Azure DevOps can't replace the variable with the value for those particular parameters. I don't want to send around those configurations for obviuos reasons.
I saw some old posts where Microsoft said this was an issue in DevOps. Is this issue still there? Is there any way to move those values in the Libray or a variables?
This is still an issue. It have to be an literal or variables defined in YAML. It cannot be variable provied via variable group for instance. Please check these topics:
How to parametrize azureSubscription in azure devops template task
Azure subscription endpoint ID cannot be provided through a variable in build definition YAML file
Azure subscription endpoint ID cannot be provided through a variable in build definition YAML file
I am just new to Azure Cloud and Devops, so forgive me if I may forget some critical info here.
So during creation of tasks for the release and selecting subscriptions, I get an error when trying to authorize the subscription (which I suspect is because of insufficient permissions associated to my account), so I go to advanced options to select the managed identity authentication.
After which no error shows now. So I set all remaining items and assign Deploy Azure App Service task. However during the running of the agent I get an error during Deploy Azure App Service step.
Error: Failed to get resource ID for resource type 'Microsoft.Web/Sites' and resource name 'sample-vue'. Error: Could not fetch access token for Managed Service Principal. Please configure Managed Service Identity (MSI) for virtual machine 'https://aka.ms/azure-msi-docs'. Status code: 400, status message: Bad Request
I have already set my azure app service to have a system assigned managed identity, but still this error occurs. I can't find any answer, online, with regards to the error above so hoping that someone could help explain to me the problem and how to possibly fix it. My hunch now is that I may have some insufficient permissions, but I don't know what it may be.
Please try the following items:
Remove and re-add the service connection in DevOps.
Check the rights of the account on Azure subscription. Please verify if the account has at least contributor access on Azure subscriptions. Check https://learn.microsoft.com/en-us/azure/billing/billing-add-change-azure-subscription-administrator