I am trying to load multiple csv files from blob to Azure MySQL. I am using getMetadata activity and foreach activity, within forEach activity one copy activity to copy data from blob to sink. Pipeline is failing with error
{
"errorCode": "2200",
"message": "'Type=Microsoft.Azure.Data.Governance.Plugins.Core.TypeConversionException,Message=Exception occurred when converting value '' for column name 'DATASTORE_ID' from type 'String' (precision:, scale:) to type 'Int32' (precision:0, scale:0). Additional info: Input string was not in a correct format.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.FormatException,Message=Input string was not in a correct format.,Source=mscorlib,'",
"failureType": "UserError",
"target": "Copy data1",
"details": []
}
I am not importing any schema in Mapping tab.
please suggest the solution.
Thanks in Advance.
Geetha.
Replace 'Copy Data' with 'Data Flow' with straight field to field mapping and the copy from blob to sink works fine.
Related
Im my pipeline, there is only one lookup activity and a stored procedure activity following it when the lookup fails.
The lookup sends a query like
select 1/count(*) as result from sometable
The stored procedure activity calls a stored precedure with a parameter named 'error'.
Basically I want this lookup to fail when count(*) is 0, and then I want to capture the divide by 0 error message from the activity output and use that as the input parameter for the stored procedure.
The output from the lookup when count(*) is 0 is:
{
"errorCode": "2100",
"message": "Failure happened on 'Sink' side. ErrorCode=UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Upload file failed at path 37b1ec96-be95-4010-8547-10387fc407a3\\result.,Source=Microsoft.DataTransfer.Common,''Type=System.Data.SqlClient.SqlException,Message=Divide by zero error encountered.,Source=.Net SqlClient Data Provider,SqlErrorNumber=8134,Class=16,ErrorCode=-2146232060,State=1,Errors=[{Class=16,Number=8134,State=1,Message=Divide by zero error encountered.,},],'",
"failureType": "UserError",
"target": "Lookup source count",
"details": []
}
So in the stored procedure activity, I want to put the message from above to the parameter of the stored procedure. I have tried #activity('Lookup source count').output.message for the input parameter. But get this error:
{
"errorCode": "InvalidTemplate",
"message": "The expression 'activity('Lookup source count').output.message' cannot be evaluated because property 'message' doesn't exist, available properties are ''.",
"failureType": "UserError",
"target": "log fail Lookup source count",
"details": ""
}
I have also tried many other options but none of them works. How can I get the error message from the lookup when it fails?
Using, "Add dynamic content", use this as your SP parameter value
#activity('<name of your lookup>').Error.Message
i am using Copy Data activity(Can't use Web activity due to security reasons) to connect to Rest API(source) (via REST dataset)and I get a response in json format as shown below which is what i need, only that this is the first page and there are many other pages that i need to retrieve. The retrieved information is then saved in SQL database table (destination). In process to retrieve the next pages i need to extract the token from "continuationToken" and send it back to my REST dataset as an dynamic value. Not sure if this is the correct way but gave it a try.
[
{
"pagedResponse":[
{
"docType":"current",
"orgNumber":"98789765",
"persons":[
],
"clientId":43533,
"name":null,
"clientResponsible":{
"id":null,
"name":null
}
}
],
"continuationToken":{
"token":"-RID:Om1+ANeDbNMWASEAAAAAAAAA==#RT:1#TRC:10#ISrV:2#IEO:6554436",
"range":{
"min":"",
"max":"05C1DFFFFFFFFFFC"
}
}
}
]
I tried to run some test by saving the json response as a file. Created Lookup activity(Lookup1) to read in the json file and in the then created a for-each activity and in the items i added a dynamic content
#activity('Lookup1').output.value[0].continuationToken
but got this error message.:
{
"errorCode": "InvalidTemplate",
"message": "The function 'length' expects its parameter to be an
array or a string. The provided value is of type 'Object'.",
"failureType": "UserError",
"target": "ForEach1"
}
expected result was:
"token":"-RID:Om1+ANeDbNMWASEAAAAAAAAA==#RT:1#TRC:10#ISrV:2#IEO:6554436",
"range":{"min":"","max":"05C1DFFFFFFFFFFC"}
I think your solution mentioned in the question should work.I suppose that you want to know whether the continuationToken is null so that the loop should be ended,so you use length function.However,according to the error message:The function 'length' expects its parameter to be an array or a string. The provided value is of type 'Object'. It is stated in the document clearly:
Then continuationToken is an object,should be used with empty function:
I am using a Copy Data activity to copy data from a REST service to an Azure SQL Server. The REST service is returning a hierarchical JSON response and I am using schema mapping to push it into the SQL table. This is working as expected except for a few attributes that have whitespace in their name. Here is an example schema mapping with the last attribute defined the one in question:
"translator": {
"type": "TabularTranslator",
"schemaMapping": {
"id": "id",
"type": "type",
"lead_verb": "lead_verb",
"lead_action_performed": "lead_action_performed",
"created_at": "created_at",
"lead_id": "lead_id",
"selected_action": "selected_action",
"rate_type": "rate_type",
"channel_rate_id": "channel_rate_id",
"tenant_id": "tenant_id",
"unit_id": "unit_id",
"created_by_id": "created_by_id",
"source_id": "source_id",
"lead_changes_humanized.Quoted Rate[1]": "lead_changes_humanized"
},
"collectionReference": "$.lead_events"
}
This results in the following error:
{
"errorCode": "2200",
"message": "ErrorCode=UserErrorInvalidJsonArrayPathDefinition,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error occurred when deserializing source JSON data. Please check whether the JsonPath in JsonNodeReference and JsonPathDefintion is valid.,Source=Microsoft.DataTransfer.Common,''Type=Newtonsoft.Json.JsonException,Message=Unexpected character while parsing path: ,Source=Newtonsoft.Json,'",
"failureType": "UserError",
"target": "cpyCompaniesLeadEvents"
}
I have tried quoting path as follows:
"lead_changes_humanized.\"Quoted Rate\"[1]": "lead_changes_humanized"
And, with single quotes:
"lead_changes_humanized.'Quoted Rate'[1]": "lead_changes_humanized"
These all result in the same error. I have validated the JSON path using path validator.
Question: how can I deal with this whitespace?
Tried but failed to skip the copy activity validator. Please consider a workaround as below.
1.If you could totally control the rest api (named A)output,try to return lead_changes_humanized.Quoted Rate[1] as lead_changes_humanized.Quoted_Rate[1].If you can't,create another your own rest api(named B) and invoke the A rest api inside B rest api,then process the output as lead_changes_humanized.Quoted_Rate[1].
2.Then you could skip the validator with this trick.If you do concern the original format of the json key,please do the next step.
3.On the sql db side,you could copy the data from source data to the temporary table.And restore the original format in the stored procedure so that you could store the original format into the exact destination table.There are very detailed steps i did in my previous case: Azure Data Factory mapping 2 columns in one column refer to it.
I am trying to rename object using patch and i am getting this error: https://cloud.google.com/storage/docs/json_api/v1/objects/patch
"errors": [{
"domain": "global",
"reason": "invalidParameter",
"message": "Value 'test rename.qgm' in content does not agree with value 'arson accelerant headspace lab.qgm'. This can happen when a value set through a parameter is inconsistent with a value set in the request."
}]
Otherwise, to rename using gstuil it performs a copy from source to destination followed by removing the source for each object.
Can we rename object in single operation through the Google API instead two operations?
Sorry, there is no rename operation. The closest thing is copying the object to its new name, and then deleting the original.
I have below json data in S3:
{ "message": "{\"a\":\"b\",\"c\":\"d\"}"}
and jsonpath:
{
"jsonpaths": [
"$.message.a"
]
}
I am using below copy command to load redshift table.
copy test.table1 from 's3://bucket1/test.gz' credentials 'aws_access_key_id=<>;aws_secret_access_key=<>' json 's3://bucket1/jsonpath.json' GZIP ;
but it fails with error.
Invalid JSONPath format: Member is not an object.
The Redshift copy error means it is not seeing an "a" key in the "message" object in your datafile.
The quotes around your "message" object value in your json data is causing your problem. Your json should look like:
{ "message": {"a":"b","c":"d"}}
Escaping the quotes should not be necessary, so I took out the backslashes. Leaving them in may work, but I did not test it.