How to log the mismatch elements while validating a json file against a json-schema?
I used json-schema-validator, but it just showing whether the json matches with the schema or not.
It is not listing the mismatch element
Related
I have a CSV file with the below format:
where column c3 is of JSON format. I need to parse that Json value to different columns and generate a CSV file.
I am using parse functionlaity :
But I am getting the below error:
Can someone help me identify whether I am doing something wrong
Looks like you have selected Document per line under JSON settings in Parse transformation. As your file contains single JSON, change the settings to select Single document. (By default, document form is selected as Document per line).
I tried the sample by selecting Document per line and got same error as you. After changing the settings to Single document, I can preview data without errors.
Changed the JSON settings:
Make sure the format is a valid JSON.
The format of c3 column in your file is not exactly in JSON format. When you preview data, the value is not parsed and shown as NULL/blank data in parsed columns.
I tried parsing with valid JSON format and it’s working fine.
Example of JSON data format looks like:
{"Name": "None", "status": "None", "processdby": 2}
Reference: Parse transformation in mapping data flow
I have the following avro definition for my Nifi flow, where i'm reading a from a BLOB database column. I'm mapping the 'xxPZPVSTREAM' column as a 'bytes' type in my avro definition
{
"namespace":"a.b.c",
"name":"pc_history",
"type":"record",
"fields": [
{"name":"COMMITDATETIME","type":["null",{"type":"long","logicalType":"timestamp-millis"}]},
....
{"name":"xxPZPVSTREAM","type":["bytes","null"]},
{"name":"xxx","type":["string","null"]}
]
}
When i attempt to write the mapped data to a Postgres database i get this error
org.postgresql.util.PSQLException: Can’t infer the SQL type to use for an instance of [Ljava.lang.Byte;. Use setObject() woth an explicit Types values to specify the type to use.
Can i add extra meta information to the avro definition to allow the Nifi processor to correctly map this binary column?
You didn't say which processor you're using, but this should be supported by PutDatabaseRecord. That processor is what you'd want to use for this as it should map byte array fields to a blob. If it doesn't, then please join the nifi-dev mailing list and let us know.
I want to read data from s3 and applymapping to it and then write it to another s3.
I want to check by datatype in field wise whether the data match the mapping datatype or not.
Like in mapping I made username is string.
Now when I write it to a S3 I need to check whether username field has all string or it has some thing odd value.
How can I achieve it ?
any help would really appreciable to me.
I have created http get service that is returning collection data as json. But I don't want to show column data type
This is my function code.
const result = doc.find({},{id:1,desc:1,label:1,LocationTree:1,path:1,coordinates:1,_id: 0}).toArray();
return result;
I contacted MongoDB support about this issue when I first started with Stitch. They indicated there is no way to suppress the type returned with numeric and datetime value types.
Your client application must handle the type when processing the JSON response. This is done with dot notation.
Assume your id field is of type Int32.
A typical API call might return an integer or string at result.id. The Stitch HTTP service, however, appends the type to the value, returning it at result.id.$numberInt.
As noted above, type data is included with numeric and datetime values. Booleans, Strings, Objects, and Arrays do not include type data. They can be accessed at result.someBoolean, result.someString, result.someObject, or result.someArray.
I haven't found a full list of types returned by the Stitch HTTP service. For some reason, they do NOT match the BSON Type aliases.
I'm trying to get a simple JSON output value in PDI from a field that was defined in an earlier step.
The field is id_trans, and I want the result to look like {"id_trans":"1A"} when id_trans value is 1A.
However, when using the JSON Output step and setting the json bloc name to empty, I get this: {"":[{"id_trans":"1A"}]}, which is normal given that the JSON Ouptut step generates json blocks, as specified in the doc.
How can I get rid of the bloc ( i.e. []) structure in a simple manner? I thought of using an external python script, but I would rather use steps in PDI.
You can easily do that with another JSON Input step. Just specify your output value from JSON Output step as Select field and under the tab fields, specify a fieldname and data[0] as Path.