I would like to copy files with Azure File Copy with Azure Pipeline.
I'm following instruction of https://praveenkumarsreeram.com/2021/04/14/azure-devops-copy-files-from-git-repository-to-azure-storage-account/
I'm using automatically created Service Connection named "My Sandbox (a1111e1-d30e-4e02-b047-ef6a5e901111)"
I'm getting error with AzureBlob File Copy:
INFO: Authentication failed, it is either not correct, or
expired, or does not have the correct permission ->
github.com/Azure/azure-storage-blob-go/azblob.newStorageError,
/home/vsts/go/pkg/mod/github.com/!azure/azure-storage-blob-
go#v0.10.1-0.20201022074806-
8d8fc11be726/azblob/zc_storage_error.go:42
RESPONSE Status: 403 This request is not authorized to perform
this operation using this permission.
I'm assuming that Azure Pipeline have no access to Azure Storage.
I wonder how do find service principal which should get access to Azure Storage.
I can also reproduce your issue on my side, as different Azure file copy task versions use different versions of AzCopy in behind, then they use different auth ways to call the API to do the operations.
There are two ways to fix the issue.
If you use the automatically created service connection, it should have Contributor role in your storage account, you could use Azure file copy task version 3.* instead of 4.*, then it will work.
If you want to use Azure file copy task version 4.*, navigate to your storage account -> Access Control (IAM) -> add your service principal used in the service connection as a Storage Blob Data Contributor role, see detailed steps here. It will also work.
Related
Was working on az ml cli v2 to deploy real-time endpoint with command az ml online-deployment through Azure pipeline. had double confirmed that the service connection used in this pipeline task had added the permissions below in Azure Portal but still showing the same error.
ERROR: Error with code: You don't have permission to alter this storage account. Ensure that you have been assigned both Storage Blob Data Reader and Storage Blob Data Contributor roles.
Using the same service connection, we are able to perform the creation of online endpoint with az ml online-endpoint create in the same and other workspaces.
Issue was resolved. I did not change anything in the service principal and running it on second day using same yml got through the issue. I guess there might be some propagation issue, but longer than usual.
we have automated the following steps using azure devops
A Release pipeline which creates a website in azure and the next step which deploys the code, all is well and working so far
The next step that we need to do is create a Azure SQLDB and and an Azure Storage Account in the release pipeline and then configure these values in the appsettings.json file
Questions
Creating the Storage account is the easy part, but how do we get the storage account key back in the pipeline and associate that value in the appsetting.json file ?
Similarly for the SQLDB how do we get the IPAddress and add it in the exclusions list.
and also it you could point us to any documentation on this
how do we get the storage account key back in the pipeline.
You can use the Azure CLI az storage account keys to get storage account keys.
As the Pre-Requisite For migration , we have AADDS(Azure Active directory Domain Services) used instead of on premise Active directory
So got error during validation with identitymapLog.csv file
""No Match Found (Check Azure AD Sync)""
AD synch tool cannot be used for AADDS .So any suggestions on solving this issues?
After opening the log directory noted in the data migration tool's output we could see the file IdentityMapLog.csv, it contains the generated mapping of AD to Azure AD identities. It's recommended that you review the identity map log file, IdentityMapLog.csv, for completeness before kicking off an import.
According to the doc: You and your Azure AD admin will need to investigate those users marked as No Match Found (Check Azure AD Sync) to understand why they aren't part of your Azure AD Connect sync.
We could refer to this doc for more details.
Yes followed that, but found that SID of Azure ADDS and SID of Azure active directory is not same .. so on that note is there any way to customized the identityLogMap.csv with UPN attribute instead of SID –
I am using an open source tool for deployment of schema for my warehouse snowflake. I have successfully done it for tables, views and procedures. Currently I'm facing an issue, I have to deploy snowflake stages same way. But stages required url and azure saas token when you define it in your sql file like this:
CREATE or replace STAGE myStage
URL = 'azure://xxxxxxxxx.blob.core.windows.net/'
CREDENTIALS = ( AZURE_SAS_TOKEN = 'xxxxxxxxxxxxxxxxxxxx' )
file_format = myFileFormat;
As it is not encouraged to use your credentials in file that will be published on version control and access by others. Is there a way/task in azure devOps so I can just pass a template SQL file in repo and change it before compilation and execution(may be via azure key vault) and change back to template? So these credentials and token always remain secure.
Have you considered using a STORAGE INTEGRATION, instead? If you use the storage integration credentials and grant that to your Blob storage, then you'd be able to create STAGE objects without passing any credentials at all.
https://docs.snowflake.net/manuals/sql-reference/sql/create-storage-integration.html
For this issue ,you can use credential-less stages to secure your cloud storage without sharing secrets.
Here agree with Mike, storage integrations, a new object type, allow a Snowflake administrator to create a trust policy between Snowflake and the cloud provider. When Snowflake connects to the organization’s cloud storage, the cloud provider authenticates and authorizes access through this trust policy.
Storage integrations and credential-less external stages put into the administrator’s hands the power of connecting to storage in a secure and manageable way. This functionality is now generally available in Snowflake.
For details ,please refer to this document. In addition, you can also via azure key vault, key vault provides a secure place for accessing and storing secrets.
We currently have one DevOps repository, with a functional CI/CD pipeline. We have another website hosted on a different instance (and different region) on Azure. We are trying to use our existing repo to deploy to the other Azure instance, but it is giving is the following message:
Failed to query service connection API: 'https://management.azure.com/subscriptions/c50b0601-a951-446c-b637-afa8d6bb1a1d?api-version=2016-06-01'. Status Code: 'Forbidden', Response from server: '{"error":{"code":"AuthorizationFailed","message":"The client '2317de35-b2c2-4e32-a922-e0d076a429f5' with object id '2317de35-b2c2-4e32-a922-e0d076a429f5' does not have authorization to perform action 'Microsoft.Resources/subscriptions/read' over scope '/subscriptions/c50b0601-a951-446c-b637-afa8d6bb1a1d'."}}'
I have tried all of the recommended trouble-shooting, making sure that the user is in a Global Administrator role and what-not, but still not luck. The secondary Azure subscription that we are hoping to push our builds to is a trial account. I'm not sure if it being a trial account matters.
I came across the same error. It turns out that, as the error message states, the service principal didn't have Read permission over the subscription. So the solution was to go to Azure Portal, select the subscription, select IAM and assign the role Reader to my service principal. Full explanation on here:
https://clydedz.medium.com/connecting-azure-devops-with-azure-46a908e3048f
I have the same problem. There are one repository and two instances of the application on the Azure portal. For the first instance, the subscription Pay-As-You-Go is used, and there were no problems for it when creating the service connection and CI/CD settings. For the second instance, a free subscription is used and when trying to create a new service connection (Azure Resource Manager) I get the same error.
I tried to do it with the permissions of Owner and Contributor
UPD: I was helped by the re-creation of the application in the azure portal
https://learn.microsoft.com/en-ca/azure/active-directory/develop/howto-create-service-principal-portal
Another option would be to save without verification if the Service Principle will not require permissions at the Subscription level. Like for example providing access to a Keyvault.
Check if the service connection for the second instance is correctly added in project settings: