I've been getting this "useful" error message when deploying an experimental data factory pipeline that has a DotNetActivity within it.
Entity provisioning failed: The blob supplied is not valid. Blob:
customactivity/package.zip. Error code: . Error message: .
I've been reading trouble shooting tips and double checking connection strings, but to no avail.
After double checking everything in particular blob storage connection strings, I was eventually able to resolve it by copying the connection string from the azure portal -> storage account -> access keys into the azure storage linked service settings.
I can only theorize that there is a subtlety in the connection string format which results in a poor error message from the pipeline provisioning.
Interestingly, I can now put a completely wrong access key into the storage account linked service, redeploy it along with the the pipeline and it doesn't complain about provisioning. So perhaps there is additional checks done on the first provision.
Related
I am creating a self hosted integration runtime. There is error in registration after installed the IR on Window.The authentication key is just copied from portal and paste on it. But, the warning says the key is invalid or empty. How can I solve it?
Both Azure Data Factory and on-prem data server are on same time zone so no time delay.
Any suggestion would be great as struggled most of the day.
I have tried by using express route exe and integration runtime msi both but same result no luck :(
It was networking issue as same key working on public endpoint network setting. Now it will fix private endpoint.
I have a blob zip file in my storage account, I have a linked service and binary dataset to get the file as the source in a copy activity. There is an outside service I call in a web activity that returns a writable SAS URL to a different storage account in this format.
https://foo.blob.core.windows.net/dmf/43de9fb6-3b96-4f47-b730-eb8de040859dblah.zip?sv=2014-02-14&sr=b&sig=0mgvh25htg45b5u4ty5E%2Bf0ahMwFkHVy3iTC2nh%2FIKw%3D&st=2022-08-13T02%3A19%3A33Z&se=2022-08-13T02%3A54%3A33Z&sp=rw
I tried adding a SAS azure blob linked service, I added a parameter for the uri on the LS, then added a dataset bound to the LS and also added a parameter for the uri, I pass the SAS uri dynamically all the way down to the linked service. The copy fails each time with The remote server returned an error: (403). I have to be doing something wrong but not sure what it is. I'd appreciate any input, thanks.
I tried to reproduce the same in my environment and got same error:
To resolve the above 403 error, you need to enable it from all network option and also check whether the Storage blob data contributor was added or not. If not , Go to Azure Storage Account -> Access control (IAM) -> +Add role assignment as Storage blob data contributor.
Now, its working.
Before I deploy a function app and functions through PowerShell > ARM template I dynamically get a service bus connection string and update the value in the connection binding in the function.json file. Then I deploy the function app and function.
Deployment works, but the function app returns an error message:
Microsoft.Azure.WebJobs.ServiceBus: Microsoft Azure WebJobs SDK ServiceBus connection string 'Endpoint=sb://sbname.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;[Hidden Credential]' is missing or empty.
The question is: Do we really have to create an application setting on the function app with this connection string in it? Is there no other option than using the app setting?
It feels like storing the connection string in another location than on the service bus would just create another vulnerability.
I am aware that I could define "connection": "RootManageSharedAccessKey_SERVICEBUS" in the function.json file and then create an app setting with that same name, but that's not the way forward I want to go.
Please check if my findings help to:
Instead of storing the Service Bus Connection String directly, you can utilize the Azure Key Vault Service or Azure App Configuration Service.
You can store the confidential values like Connection Strings, Key-value pairs, Client Secrets, Certificate Passwords, etc. in Azure Key Vault.
Assume you have stored the ServiceBus connection string in the Key Vault. That you want to retrieve in the Function App through Application Settings but here in the app setting you can define as:
"connection": #Microsoft.KeyVault(SecretUri=https://myvault.vault.azure.net/secrets/RootManageSharedAccessKey_SERVICEBUS/)
This KeyVault URI cannot be accessed by any other user until they are assigned with System or User Assigned Managed Identity Permission.
Azure App Configuration is a central place where you can manage application settings and secure their accesses in one place. Refer here for more information.
I have created a data factory with a pipeline moving data from storage account to azure sql.
Company advised me to use a managed private endpoint to create connection with azure sql.
Scenario:
I have a Dev resource group where my storage account, data factory and sql sit and a Sit resource group where Sit resources sit. I have created managed private endpoint in both data factories with same name, but pointing to different sql servers.
sql_mpe: /subscriptions/123456789/resourceGroups/rg-dev/providers/Microsoft.Sql/servers/dev-sql-server
sql_mpe: /subscriptions/123456789/resourceGroups/rg-sit/providers/Microsoft.Sql/servers/sit-sql-server
As you can see managed private endpoint created has the same name but pointing to different sql servers based on the environment.
Now when I publish the dev adf to azure git, it takes the dev managed private endpoint keys as parameters as follows:
-sql_mpe_properties_privateLinkResourceId "/subscriptions/123456789/resourceGroups/rg-sit/providers/Microsoft.Sql/servers/sit-sql-server"
-sql_mpe_properties_groupId "sqlServer"
-sql_mpe_properties_ipAddress {}
-sql_mpe_properties_resourceId "/subscriptions/987654321/resourceGroups/vnet-45645632-UKSouth-567-rg/providers/Microsoft.Network/privateEndpoints/sit-sql-server.sql_mpe"
For some weird reason, in privateLinkResourceId, resource group and subscription are correct but in resourceId, they are weird values. I don't where they come from hence can't comment on it.
Now when I run my release pipeline, I get the following error:
2022-03-14T15:33:41.5334804Z ##[error]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.
2022-03-14T15:33:41.5366078Z ##[debug]Processed: ##vso[task.issue type=error;]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.
2022-03-14T15:33:41.5373551Z ##[error]Details:
2022-03-14T15:33:41.5374630Z ##[debug]Processed: ##vso[task.issue type=error;]Details:
2022-03-14T15:33:41.5376732Z ##[error]ManagedPrivateEndpointInvalidPayload: Managed private endpoint 'sql_mpe' is invalid.
Error is very generic, hence I went through the docs to understand it. I found the below reason from azure doc Best practices for CI CD:
If a private endpoint already exists in a factory and you try to
deploy an ARM template that contains a private endpoint with the same
name but with modified properties, the deployment will fail.
So I got to know that if you deploy managed private endpoint with same name but different modifies properties (like my sit endpoint is pointing to sit), it will fail.
So now I know why pipeline is failing.
I have to fix this issue for a successful release.
Below are my possible options that I can go with , but don't know how to ? This is where I require some help/ assistance:
resourceId value needs to be understood and changed for SIT (I mentioned some weird values are getting there, and in template, I am just overriding the 'dev' part to 'sit'. I am not changing the vnet resource group and other values.
Remove managed private endpoint parameters from template before publishing to azure git or remove them before creating a release. If I release them in pipeline, error is caused.
Need some insight and help here.
I have an internal document (Excel) that has an Azure information Protection / O365 Unified Sensitivity Labelling applied to it.
Im trying to extract that data, but I'm getting an Encryption Error because and rightly so the information is encrypted.
The process:
The document is pulled from Sharepoint into a blob storage container and then Azure Data factory picks up the file using the Copy activity and reads the contents into an Azure SQL Database
Error message:
ErrorCode=EncryptedExcelIsNotSupported,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Encrypted excel file 'dummy file.xlsx' is not supported, please remove its password.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=ICSharpCode.SharpZipLib.Zip.ZipException,Message=Wrong Local header signature: 0xE011CFD0,Source=ICSharpCode.SharpZipLib,'
I have a Linked Service using a Service principal that can connect to the file, but previewing the data results in a message saying the file is encrypted.
I presume I would need to give permissions to the Service Principal, but im still stuck what those would be.
I tried adding Azure Rights Management read/create in the API permissions but that still hasn't worked
Data Factory can't read the file data which is protected by other service.
If you want to copy data from the encrypted files, you must have the permission to access it.