I have below json data in S3:
{ "message": "{\"a\":\"b\",\"c\":\"d\"}"}
and jsonpath:
{
"jsonpaths": [
"$.message.a"
]
}
I am using below copy command to load redshift table.
copy test.table1 from 's3://bucket1/test.gz' credentials 'aws_access_key_id=<>;aws_secret_access_key=<>' json 's3://bucket1/jsonpath.json' GZIP ;
but it fails with error.
Invalid JSONPath format: Member is not an object.
The Redshift copy error means it is not seeing an "a" key in the "message" object in your datafile.
The quotes around your "message" object value in your json data is causing your problem. Your json should look like:
{ "message": {"a":"b","c":"d"}}
Escaping the quotes should not be necessary, so I took out the backslashes. Leaving them in may work, but I did not test it.
Related
The JSON file is local, in public where it is accessible. The error references the text found in the JSON file where the error occurs.
Check for commas after the last item in arrays within the JSON file. Fetch doesn't like these, even if online JSON checkers give it a pass.
{"documents": [ {"important":"something"}, ]
^NO
I am using API Gateway to receive data from a webhook and pushing it to Kinesis. Using mapping template, I create a JSON using data from the webhook payload.
The problem is, when I encode it to base64 and Kinesis decodes it, it ends up becoming an invalid json. The ":" get replaced by "=".
Here's my mapping template:
#set ($account = $util.parseJson($input.params().header.get("some-field")).account) //fetching a field from a header
{
"StreamName": "$input.params('stream-name')",
"Data": "$util.base64Encode(
{"Payload": "$input.json('$.payload')",
"Account": "$account",
"Event": "$input.json('$.event')"
})",
"PartitionKey": "$account"
}
What's happening is, the Data JSON object is getting corrupted when it gets decoded by Firehose for inline parsing (for dynamic partitioning). I get
{
Payload={"a":1,"b":2},
Account=abcdehdds,
Event="abc"
}
Notice the = instead of :, making this an invalid JSON.
What's strange is when the data is pushed from Firehose to s3 without any dynamic partitioning, it's a perfect JSON.
I'm at my wits end trying to solve this. Would appreciate any help.
I have a web task passing data to an api in data factory.
The api only accepts a malformed json fragment:
[
{
"RowNumber":1,"Tag":"ddddd",
"LastUpdateDateTime":"2022-07-26T13:14:28Z"
}
]
Datafactory wont allow this to be converted to json with the json function as its not valid json.
But if its sent in the body as a string, all the double quotes are escaped:
[
{
\"RowNumber\":1,
\"Tag\":\"ddddd\",
\"LastModifiedDate\":\"2022-07-26T13:14:28Z\"
}
]
is there any way I can remove the backslashes in data factory without to do all this in an azure function?
seems replace does remove the back slashes when sending the data between tasks, it just puts it in again if i try to view the output of the task!
using replace fixed the issue.
I am trying to load multiple csv files from blob to Azure MySQL. I am using getMetadata activity and foreach activity, within forEach activity one copy activity to copy data from blob to sink. Pipeline is failing with error
{
"errorCode": "2200",
"message": "'Type=Microsoft.Azure.Data.Governance.Plugins.Core.TypeConversionException,Message=Exception occurred when converting value '' for column name 'DATASTORE_ID' from type 'String' (precision:, scale:) to type 'Int32' (precision:0, scale:0). Additional info: Input string was not in a correct format.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.FormatException,Message=Input string was not in a correct format.,Source=mscorlib,'",
"failureType": "UserError",
"target": "Copy data1",
"details": []
}
I am not importing any schema in Mapping tab.
please suggest the solution.
Thanks in Advance.
Geetha.
Replace 'Copy Data' with 'Data Flow' with straight field to field mapping and the copy from blob to sink works fine.
I am using a Copy Data activity to copy data from a REST service to an Azure SQL Server. The REST service is returning a hierarchical JSON response and I am using schema mapping to push it into the SQL table. This is working as expected except for a few attributes that have whitespace in their name. Here is an example schema mapping with the last attribute defined the one in question:
"translator": {
"type": "TabularTranslator",
"schemaMapping": {
"id": "id",
"type": "type",
"lead_verb": "lead_verb",
"lead_action_performed": "lead_action_performed",
"created_at": "created_at",
"lead_id": "lead_id",
"selected_action": "selected_action",
"rate_type": "rate_type",
"channel_rate_id": "channel_rate_id",
"tenant_id": "tenant_id",
"unit_id": "unit_id",
"created_by_id": "created_by_id",
"source_id": "source_id",
"lead_changes_humanized.Quoted Rate[1]": "lead_changes_humanized"
},
"collectionReference": "$.lead_events"
}
This results in the following error:
{
"errorCode": "2200",
"message": "ErrorCode=UserErrorInvalidJsonArrayPathDefinition,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error occurred when deserializing source JSON data. Please check whether the JsonPath in JsonNodeReference and JsonPathDefintion is valid.,Source=Microsoft.DataTransfer.Common,''Type=Newtonsoft.Json.JsonException,Message=Unexpected character while parsing path: ,Source=Newtonsoft.Json,'",
"failureType": "UserError",
"target": "cpyCompaniesLeadEvents"
}
I have tried quoting path as follows:
"lead_changes_humanized.\"Quoted Rate\"[1]": "lead_changes_humanized"
And, with single quotes:
"lead_changes_humanized.'Quoted Rate'[1]": "lead_changes_humanized"
These all result in the same error. I have validated the JSON path using path validator.
Question: how can I deal with this whitespace?
Tried but failed to skip the copy activity validator. Please consider a workaround as below.
1.If you could totally control the rest api (named A)output,try to return lead_changes_humanized.Quoted Rate[1] as lead_changes_humanized.Quoted_Rate[1].If you can't,create another your own rest api(named B) and invoke the A rest api inside B rest api,then process the output as lead_changes_humanized.Quoted_Rate[1].
2.Then you could skip the validator with this trick.If you do concern the original format of the json key,please do the next step.
3.On the sql db side,you could copy the data from source data to the temporary table.And restore the original format in the stored procedure so that you could store the original format into the exact destination table.There are very detailed steps i did in my previous case: Azure Data Factory mapping 2 columns in one column refer to it.