Azure data factory ci/cd release errors - azure-data-factory

I am trying to deploy to the UAT environment and followed the needed steps shown in a blog post and youtube video.
However, I keep getting failures.
If I run it in 'validation only' mode it passes fine. But to actually deploy it under 'incremental' I receive the following errors
2022-03-01T17:24:27.6268601Z ##[error]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.
2022-03-01T17:24:27.6301466Z ##[error]Details:
2022-03-01T17:24:27.6306699Z ##[error]BadRequest: Failed to encrypt sub-resource payload {
"Id": "/subscriptions/a834838a-11d5-4657-a9c3-bc8b2ebdaa59/resourceGroups/adf-rg-uat-uks/providers/Microsoft.DataFactory/factories/adf-df-uat-uks/linkedservices/adfSQLDB_Dev",
"Name": "adfSQLDB_Dev",
"Properties": {
"annotations": [],
"type": "AzureSqlDatabase",
"typeProperties": {
"connectionString": "********************"
}
}
} and error is: Message for the errorCode not found..
2022-03-01T17:24:27.6314634Z ##[error]BadRequest: Failed to encrypt sub-resource payload {
"Id": "/subscriptions/a834838a-11d5-4657-a9c3-bc8b2ebdaa59/resourceGroups/adf-rg-uat-uks/providers/Microsoft.DataFactory/factories/adf-df-uat-uks/linkedservices/adfSQLDB_Prod",
"Name": "adfSQLDB_Prod",
"Properties": {
"annotations": [],
"type": "AzureSqlDatabase",
"typeProperties": {
"connectionString": "********************"
}
}
} and error is: Message for the errorCode not found..
2022-03-01T17:24:27.6322501Z ##[error]BadRequest: Failed to encrypt sub-resource payload {
"Id": "/subscriptions/a834838a-11d5-4657-a9c3-bc8b2ebdaa59/resourceGroups/adf-rg-uat-uks/providers/Microsoft.DataFactory/factories/adf-df-uat-uks/linkedservices/adfBlobStorage",
"Name": "adfBlobStorage",
"Properties": {
"annotations": [],
"type": "AzureBlobStorage",
"typeProperties": {
"connectionString": "********************"
}
}
} and error is: Expecting connection string of format "key1=value1; key2=value2"..
2022-03-01T17:24:27.6328134Z ##[error]BadRequest: Failed to encrypt sub-resource payload {
"Id": "/subscriptions/a834838a-11d5-4657-a9c3-bc8b2ebdaa59/resourceGroups/adf-rg-uat-uks/providers/Microsoft.DataFactory/factories/adf-df-uat-uks/linkedservices/adfSQLDB",
"Name": "adfSQLDB",
"Properties": {
"annotations": [],
"type": "AzureSqlDatabase",
"typeProperties": {
"connectionString": "********************"
}
}
} and error is: Message for the errorCode not found..
2022-03-01T17:24:27.6336169Z ##[error]BadRequest: Failed to encrypt sub-resource payload {
"Id": "/subscriptions/a834838a-11d5-4657-a9c3-bc8b2ebdaa59/resourceGroups/adf-rg-uat-uks/providers/Microsoft.DataFactory/factories/adf-df-uat-uks/linkedservices/adfSQLDB_UAT",
"Name": "adfSQLDB_UAT",
"Properties": {
"annotations": [],
"type": "AzureSqlDatabase",
"typeProperties": {
"connectionString": "********************"
}
}
} and error is: Message for the errorCode not found..
2022-03-01T17:24:27.6339122Z ##[error]Check out the troubleshooting guide to see if your issue is addressed: https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/azure-resource-group-deployment?view=azure-devops#troubleshooting
2022-03-01T17:24:27.6342204Z ##[error]Task failed while creating or updating the template deployment.
Regards
Mark

Are you using any Self Hosted Integration Runtime? If yes, then please check it is in available state during the deployment or your connection string may not have the Secure String property:
"typeProperties": {
"connectionString": {
"type": "SecureString",
"value": .............
}
}

Related

Azure ARM Template parameters for parametrized linked service

Please, forgive the confusing tittle, if it is, but it does describe the problem I am having
So, I have a linked service in my Azure Datafactory. It is used for Azure SQL Database connect.
The Database name and user name are being taken from the parameters set in linked service itself. Here is a snippet of json config
"typeProperties": {
"connectionString": "Integrated Security=False;Encrypt=True;Connection Timeout=30;Data Source=myserver.database.windows.net;Initial Catalog=#{linkedService().dbName};User ID=#{linkedService().dbUserName}",
"password": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "KeyVaultLink",
"type": "LinkedServiceReference"
},
"secretName": "DBPassword"
},
"alwaysEncryptedSettings": {
"alwaysEncryptedAkvAuthType": "ManagedIdentity"
}
}
This works fine when in debug in the Azure portal. However, when I get the ARM Template for the whole thing, during ARM Template deployment it asks for input Connection string for the linked service. If I go to the linked service definition, and look up its connection string it will come this way
"connectionString": "Integrated Security=False;Encrypt=True;Connection Timeout=30;Data Source=dmsql.database.windows.net;Initial Catalog=#{linkedService().dbName};User ID=#{linkedService().dbUserName}"
Then when I input it in the ARM Template deployment should I be replacing "#{linkedService().dbName}" and "#{linkedService().dbUserName}" with actual values at the spot when I am entring it ? I am confused because during the ARM Template deployment there are no separate fields for these parameters, and these (parameters specific to linked service itself) are not present as separate parameters in the ARM Template definition.
I created database in my azure portal
and enabled system assigned managed Identity for sql db.
Image for reference:
I created azure keywault and created secret.
Image for reference:
I have created new access policy for Azure data factory.
Image for reference:
I created Azure data factory and enabled system managed identity.
Image for reference:
I have created new parametrized linked service to connect with database with below parameters dbName and userName. I am taking database name and User name dynamically by using above parameters.
Image for reference:
Linked service is created successfully.
json format of my lined service:
{
"name": "SqlServer1",
"properties": {
"parameters": {
"dbName": {
"type": "String"
},
"userName": {
"type": "String"
}
},
"annotations": [],
"type": "SqlServer",
"typeProperties": {
"connectionString": "Integrated Security=False;Data Source=dbservere;Initial Catalog=#{linkedService().dbName};User ID=#{linkedService().userName}",
"password": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "AzureKeyVault1",
"type": "LinkedServiceReference"
},
"secretName": "DBPASSWORD"
},
"alwaysEncryptedSettings": {
"alwaysEncryptedAkvAuthType": "ManagedIdentity"
}
}
}
}
I exported the arm template of data factory.
This is my linked service in my ARM template:
"SqlServer1_connectionString": {
"type": "secureString",
"metadata": "Secure string for 'connectionString' of 'SqlServer1'",
"defaultValue": "Integrated Security=False;Data Source=dbservere;Initial Catalog=#{linkedService().dbName};User ID=#{linkedService().userName}"
},
"AzureKeyVault1_properties_typeProperties_baseUrl": {
"type": "string",
"defaultValue": "https://keysqlad.vault.azure.net/"
}
Image for reference:
I have got parameters dbName and userName in my ARM template description.
{
"name": "[concat(parameters('factoryName'), '/SqlServer1')]",
"type": "Microsoft.DataFactory/factories/linkedServices",
"apiVersion": "2018-06-01",
"properties": {
"parameters": {
"dbName": {
"type": "String"
},
"userName": {
"type": "String"
}
},
"annotations": [],
"type": "SqlServer",
"typeProperties": {
"connectionString": "[parameters('SqlServer1_connectionString')]",
"password": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "AzureKeyVault1",
"type": "LinkedServiceReference"
},
"secretName": "DBPASSWORD"
},
"alwaysEncryptedSettings": {
"alwaysEncryptedAkvAuthType": "ManagedIdentity"
}
}
},
"dependsOn": [
"[concat(variables('factoryId'), '/linkedServices/AzureKeyVault1')]"
]
}
Image for reference:
If you didn't get parameters in ARM template description copy the value of "connectionString" and modified what you needed to and left the parameters in place and added it to the "connectionString" override parameter in my Azure Release Pipeline, and it will work.

Permission denied for a root to lookup a token in Vault

I logged to vault with a root token.
I try to
$ vault token lookup
but I keep getting
Error looking up token: Error making API request.
URL: GET https://106.120.137.192:8200/v1/auth/token/lookup-self
Code: 403. Errors:
* permission denied
I have vault logs on Trace level, but there is no related event.
I enabled audit logs to see what's going on but they give me no hint.
[
{
"time": "2021-10-21T15:34:17.647568529Z",
"type": "request",
"auth": {
"token_type": "default"
},
"request": {
"id": "1d5d7f5f-94ca-e281-c0b2-5ffbceccb0dc",t
"operation": "read",
"mount_type": "token",
"client_token": "hmac-sha256:75f6fc0b19c105af0f2c27fd180742eef282c38d346fc732771bfaa2d1ce2ea6",
"namespace": {
"id": "root"
},
"path": "auth/token/lookup-self",
"remote_address": "172.18.0.1"
},
"error": "permission denied"
},
{
"time": "2021-10-21T15:34:17.647692649Z",
"type": "response",
"auth": {
"token_type": "default"
},
"request": {
"id": "1d5d7f5f-94ca-e281-c0b2-5ffbceccb0dc",
"operation": "read",
"mount_type": "token",
"client_token": "hmac-sha256:75f6fc0b19c105af0f2c27fd180742eef282c38d346fc732771bfaa2d1ce2ea6",
"namespace": {
"id": "root"
},
"path": "auth/token/lookup-self",
"remote_address": "172.18.0.1"
},
"response": {
"mount_type": "token",
"data": {
"error": "hmac-sha256:9493ed1bac12e9a7fae0e03c488dd1d5f46bcc33ea36ee2c1e5ca92acd683c81"
}
},
"error": "1 error occurred:\n\t* permission denied\n\n"
}
]
What else can I do?
I am running Vault 1.7.0
Ok. I found the problem does not happen when I am running same command on localhost, i.e., against local instance of Vault.

Google Storage AuditLogs - finding who is trying to access

I have a google storage bucket with Audit Logs enabled. Every one\two days I getting logs about PERMISSION DENIED. The log is specifying what kind of access the requestor is asking for. But, not give me enough information to answer the question - who is requesting?
This is the log message:
{
"insertId": "rr6wsd...",
"logName": "projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fdata_access",
"protoPayload": {
"#type": "type.googleapis.com/google.cloud.audit.AuditLog",
"authenticationInfo": {},
"authorizationInfo": [
{
"permission": "storage.buckets.get",
"resource": "projects//buckets/BUCKET_NAME",
"resourceAttributes": {}
}
],
"methodName": "storage.buckets.get",
"requestMetadata": {
"callerSuppliedUserAgent": "Blob/1 (cr/340918833)",
"destinationAttributes": {},
"requestAttributes": {
"auth": {},
"reason": "8uSywAZKWkhOZWVkZWQg...",
"time": "2021-01-20T03:43:38.405230045Z"
}
},
"resourceLocation": {
"currentLocations": [
"us-central1"
]
},
"resourceName": "projects//buckets/BUCKET_NAME",
"serviceName": "storage.googleapis.com",
"status": {
"code": 7,
"message": "PERMISSION_DENIED"
}
},
"receiveTimestamp": "2021-01-20T03:43:38.488787956Z",
"resource": {
"labels": {
"bucket_name": "BUCKET_NAME",
"location": "us-central1",
"project_id": "PROJECT_ID"
},
"type": "gcs_bucket"
},
"severity": "ERROR",
"timestamp": "2021-01-20T03:43:38.399417759Z"
}
As you can see, the only information who talking about "who is trying to access" is
"callerSuppliedUserAgent": "Blob/1 (cr/340918833)",
But what that means? mean nothing to me.
How I can understand who is trying to access this permission?
The callerSuppliedUserAgent can be anything the client application puts in their request headers - Ignore it as this header can be faked. Only legitimate applications put anything meaningful in the header.
This is an unauthenticated request. There is no identity to record. Most likely a troll scanning the Internet looking for open buckets.
Notice that the auth key is empty. No authorization was provided in the request.
"requestAttributes": {
"auth": {},
"reason": "8uSywAZKWkhOZWVkZWQg...",
"time": "2021-01-20T03:43:38.405230045Z"
}

Linked Service parameterization not working for Linked Service of type Azure Data Explorer (Kusto)

I initially successfully created the following linked service in ADFv2 of type AzureDataExplorer for accessing my database in ADX called CustomerDB:-
{
"name": "ls_AzureDataExplorer",
"properties": {
"type": "AzureDataExplorer",
"annotations": [],
"typeProperties": {
"endpoint": "https://mycluster.xxxxmaskingregionxxxx.kusto.windows.net",
"tenant": "xxxxmaskingtenantidxxxx",
"servicePrincipalId": "xxxxmaskingspxxxx",
"servicePrincipalKey": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "ls_AzureKeyVault_MyKeyVault",
"type": "LinkedServiceReference"
},
"secretName": "MySecret"
},
"database": "CustomerDB"
}
},
"type": "Microsoft.DataFactory/factories/linkedservices"
}
This worked smoothly. Some values I had to mask for obvious reasons but just wanted to say that there is no issue with this connection. Now inspired from this Microsoft documentation I am trying to create a generic version of this linked service, which makes sense because otherwise if I have 10 databases in the cluster, I will have to create 10 different linked services.
So I tried to create the parameterized version in the following manner:-
{
"name": "ls_AzureDataExplorer_Generic",
"properties": {
"type": "AzureDataExplorer",
"annotations": [],
"typeProperties": {
"endpoint": "https://mycluster.xxxxmaskingregionxxxx.kusto.windows.net",
"tenant": "xxxxmaskingtenantidxxxx",
"servicePrincipalId": "xxxxmaskingspxxxx",
"servicePrincipalKey": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "ls_AzureKeyVault_MyKeyVault",
"type": "LinkedServiceReference"
},
"secretName": "MySecret"
},
"database": "#{linkedService().DBName}"
}
},
"type": "Microsoft.DataFactory/factories/linkedservices"
}
But while publishing the changes I keep getting the following error:-
Is there any solution to this?
The article clearly says that:-
For all other data stores, you can parameterize the linked service by selecting the Code icon on the Connections tab and using the JSON editor
So as per that my changes should have been published successfully. But I keep getting the error.
It appears I need to specify the parameter elsewhere in the same JSON. The followed worked:-
{
"name": "ls_AzureDataExplorer_Generic",
"properties": {
"parameters": {
"DBName": {
"type": "string"
}
},
"type": "AzureDataExplorer",
"annotations": [],
"typeProperties": {
"endpoint": "https://mycluster.xxxxmaskingregionxxxx.kusto.windows.net",
"tenant": "xxxxmaskingtenantidxxxx",
"servicePrincipalId": "xxxxmaskingspxxxx",
"servicePrincipalKey": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "ls_AzureKeyVault_MyKeyVault",
"type": "LinkedServiceReference"
},
"secretName": "MySecret"
},
"database": "#{linkedService().DBName}"
}
},
"type": "Microsoft.DataFactory/factories/linkedservices"
}

"Error getting chaincode package bytes" when deploying chaincode on hyperledger via REST

I'm trying to deploy chaincode on hyperledger (Bluemix service) via POST/REST to
/chaincode
QuerySpec
{ "jsonrpc": "2.0", "method": "deploy", "params": { "type": 1,
"chaincodeID": { "path":
"https://github.com/romeokienzler/learn-chaincode/tree/master/finished"
}, "ctorMsg": { "function": "init", "args": [ "hi there" ] },
"secureContext": "user_type1_0" }, "id": 1 }
I've also tried those links
https://github.com/romeokienzler/learn-chaincode/blob/master/finished/chaincode_finished?raw=true
https://raw.githubusercontent.com/romeokienzler/learn-chaincode/master/finished/chaincode_finished.go
I always get
{ "jsonrpc": "2.0", "error": {
"code": -32001,
"message": "Deployment failure",
"data": "Error when deploying chaincode: Error getting chaincode package bytes: Error getting code 'go get' failed with error: 'exit
status 1'\npackage
github.com/romeokienzler/learn-chaincode/tree/master/finished: cannot
find package
'github.com/romeokienzler/learn-chaincode/tree/master/finished' in any
of:\n\t/usr/local/go/src/github.com/romeokienzler/learn-chaincode/tree/master/finished
(from
$GOROOT)\n\t/go/usercode/552962906/src/github.com/romeokienzler/learn-chaincode/tree/master/finished
(from
$GOPATH)\n\t/go/src/github.com/romeokienzler/learn-chaincode/tree/master/finished\n"
}, "id": 1 }
Any idea?
Considering that you are playing with Bluemix service, I assume you are following "Implementing your first chain code tutorial"
If your forked repository you will see instructions to use branch v1.0 for Bluemix Blockchain Services (link) IBM BMX Service is (still) using Fabric v0.5.
Once you have Registered with one of the available Enroll ID you should be able to deploy your chaincode using DeploySpec (note the path: "https://github.com/romeokienzler/learn-chaincode/tree/v1.0/finished")
{
"jsonrpc": "2.0",
"method": "deploy",
"params": {
"type": 1,
"chaincodeID": {
"path": "https://github.com/romeokienzler/learn-chaincode/tree/v1.0/finished"
},
"ctorMsg": {
"function": "init",
"args": [
"hi there"
]
},
"secureContext": "user_type1_0"
},
"id": 1
}
First of all deploy command should be changed to ( the value for path variable was changed):
{
"jsonrpc": "2.0",
"method": "deploy",
"params": {
"type": 1,
"chaincodeID": {
"path": "https://github.com/romeokienzler/learn-chaincode/finished"
},
"ctorMsg": {
"function": "init",
"args": ["hi there"]
},
"secureContext": "user_type1_0"
},
"id": 1
}
P.S. As #Mil4n correctly mentioned, IBM Bluemix still works with Fabric v0.5. Chaincode romeokienzler/learn-chaincode/finished should be adopted to this version.
For example shim.ChaincodeStubInterface is not available yet and should be replaced with *shim.ChaincodeStub.