how to use json_extract_path_text? - amazon-redshift

I am facing an issue with JSON extract using JSON_EXTRACT_PATH_TEXT in redshift
I have two separate JSON columns
One containing the modems the customer is using and the other one containing the recharge details
{"Mnfcr": "Technicolor","Model_Name":"Technicolor ABC1243","Smart_Modem":"Y"}
For the above, I have no issue extracting the Model_name using JSON_EXTRACT_PATH_TEXT(COLUMN_NAME, 'Model_Name') as model_name
[{"Date":"2021-12-24 21:42:01","Amt":50.00}]
This one is causing me trouble. I used the same method above and it did not work. it gave me the below error
ERROR: JSON parsing error Detail: ----------------------------------------------- error: JSON parsing error code: 8001 context: invalid json object [{"Date":"2021-07-03 17:12:16","Amt":50.00
Can I please get assistance on how to extract this using the json_extract_path_text?
One other method I have found and it worked was to use regexp_substring.

This second string is a json array (square braces), not an object (curly brackets). The array contains a single element which is an object. So you need to extract the object from the array before using JSON_EXTRACT_PATH_TEXT().
The junction for this is JSON_EXTRACT_ARRAY_ELEMENT_TEXT().
Putting this all together we get:
JSON_EXTRACT_PATH_TEXT(
JSON_EXTRACT_ARRAY_ELEMENT_TEXT( <column>, 0)
, 'Amt')

you can use json_extract_path_text like below example
json_extract_path_text(json_columnName, json_keyName) = compareValue
for more you can refer this article
https://docs.aws.amazon.com/redshift/latest/dg/JSON_EXTRACT_PATH_TEXT.html

Related

Unable to pass dynamic and unique date values in JMeter

I have a request payload(JSON format) which has an array with 1000 objects and each object has 6 key value pairs out of which 5 I’m reading from the csv file using parameterization and the 6th key has to be a unique date value of a future date for each of the object in the array.
I tried this with time-shift function which works for 1 iteration but I want to execute it for n- number of iterations.
I checked for groovy code for this but I have no knowledge of groovy and have started learning it.
How can I achieve this in JMeter.
Also, on reading time-shift function from HTTP Request Defaults-Parameters or from the Test Plan-User Defined Variables it does not read different date for each object, it duplicates same date of the first variable in each object.
{
“deviceNumber": “XX”,
“array: [
{
“keyValue1: “${value1_ReadFromCSV}”,
"keyValue2”: “${value2_ReadFromCSV}”,
"keyValue3”: “${value3_ReadFromCSV}”,
"keyValue4”: “${value4_ReadFromCSV}”,
"keyValue5”: “${value5_ReadFromCSV}”,
"keyValue6”: "2020-05-23” (Should be dynamically generated)
},
{
“keyValue7: “value7_ReadFromCSV”,
"keyValue8”: "value8_ReadFromCSV",
"keyValue9”: "value9_ReadFromCSV",
"keyValue10”: "value10_ReadFromCSV",
"keyValue11”: "value11_ReadFromCSV",
"keyValue12”: "2020-05-24” (Should be dynamically generated)
},
.
.
.
.
{
“keyValue995: “value995_ReadFromCSV”,
"keyValue996”: "value996_ReadFromCSV",
"keyValue997”: "value997_ReadFromCSV",
"keyValue998”: "value998_ReadFromCSV",
"keyValue999”: "value999_ReadFromCSV",
"keyValue1000”: "2025–12-31” (Should be dynamically generated)
}
]
}
I have got the partial solution to this, by reading the csv file line by line and storing each line into a variable using groovy. However, I don't want to store directly the line into the variable but to create a JSON object like above from each line of csv file with a unique future date for each object which is in the array.
The csv file is : (Note: I have removed column for date column in csv as I no longer need it.)
deviceNumber,keyValue1,keyValue2,keyValue3,keyValue4,keyValue5,keyValue7,keyValue8,keyValue9,keyValue10,keyValue11,keyValue12,keyValue13,keyValue15,keyValue15,keyValue16
01,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring
02,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring
03,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring
.
.
.
1000,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring,somestring
Kindly suggest any reference/example to do this.
I provide only generic instructions:
You can dynamically construct request body using JSR223 PreProcessor
You can read CSV file into memory using File.readLines() function
You can build JSON out of the values from the CSV file using JsonBuilder class
More information:
Apache Groovy - Parsing and producing JSON
Apache Groovy - Why and How You Should Use It

Data factory lookup (dot) in the item() name

I am having lookup wherein salesforce query is there. I am using elements (item()) in subsequent activities. Till now i had item().name or item().email but now i have item().NVMStatsSF__Related_Lead__r.FirstName which has (dot) in the field name.
How should i parse it through body tag so that it reads it correctly?
So I have the following data in item()
{
"NVMStatsSF__Related_Lead__c": "00QE000egrtgrAK",
"NVMStatsSF__Agent__r.Name": "ABC",
"NVMStatsSF__Related_Lead__r.Email": "geggegg#gmail.com",
"NVMStatsSF__Related_Lead__r.FirstName": "ABC",
"NVMStatsSF__Related_Lead__r.OwnerId": "0025434535IIAW"
}
now when i use item().NVMStatsSF__Agent__r.Name it will not parse because of (dot) after NVMStatsSF__Agent__r. And it is giving me the following error.
'item().NVMStatsSF__Related_Lead__r.Email' cannot be evaluated because property 'NVMStatsSF__Related_Lead__r' doesn't exist, available properties are 'NVMStatsSF__Related_Lead__c, NVMStatsSF__Agent__r.Name, NVMStatsSF__Related_Lead__r.Email, NVMStatsSF__Related_Lead__r.FirstName, NVMStatsSF__Related_Lead__r.OwnerId'.",
"failureType": "UserError",
"target": "WebActivityToAddPerson"
this is because ADF uses '.' for object reading.
Could you find a way to rename the field name which contains '.'?
Seems like you need a built-in function to get the value of an object according to the key. Like getValue(item(), 'key.nestkey'). But unfortunately, seems there isn't such a function. You may need handle your key first.
Finally, it worked. I was being silly.
Instead of taking the value from the child table with the help of (dot) operator I just used subquery. Silly see.
And it worked.

Is PapaParse adding an empty string to the end of its data array?

Papa Parse seems wise, but I think he might be giving me null. I'm just:
Papa.parse(countries);
Where countries is a string containing the XMLHttpRequest of the countries csv file from a timezone database here:
https://timezonedb.com/download
But Papa Parse seems to have added an empty array to the end of it's data array. So when I'm searching and sorting through the array, that one empty guy at the end is giving me troubles. I can write around it but it's not ideal, and I thought Papa Parse was supposed to make those kind of csv parsing problems go away. Am I Parsing wrong?
Here is the end of the PapaParsed Array in console:
You need to use skipEmptyLines: true in parse config. For example:
Papa.parse(this.csvData, {skipEmptyLines: true,})
it was adding empty line to my iteration as well. i decided to skip it by doing loop:
for(let i=0;i<data.length -1;i++){
We can also use below syntax to remove empty lines from the record.
For example, in order to remove empty values from header, we can use the below code snippet.
headers.filter(Boolean);

Encoding issue when retrieving text from QTableWidgetItem

I'm working on a GUI using PyQt5 and a QTableWidget containing CheckBoxes items. I populate the table with data from json file and the user can check/uncheck items.
When I want to retrieve data from my QTableWidget items, I get unexpected error due to some characters : \x19. The character in question is ’ which is well displayed elsewhere.
I don't understand the issue as my input json data is utf-8, I use python3 and when I check my data, I see no encoding issue. This error occurs only when calling text() method on QTableWidgetItem.
As I don't see any other way to retrieve data from my items, I'm quite stuck on it.
with open(filepath) as file:
self.data = json.load(file)
for key, value in self.data.items():
print(key, self.data[key])
keyword_box = QTableWidgetItem(key)
I get no error if I use key to compare with other string for example but when I want to retrieve data (no editing was performed by the user, strings didn't change)
items = self.table.selectedItems()
for item in items:
print(item.text())
I get unknown character in my console and if I try to use the value, I get a \x19 character which fails when I use it to compare with keys in my dictionary.
Could this come from a bug in QTableWidgetItem text() method ?

Pyspark: ValueError

I have a dictionary of PySpark RDDs and am trying to convert them to data frames, save them as variable and then join them. When I attempt to convert one of my RDDs to a data frame I get the following error:
File "./spark-1.3.1/python/pyspark/sql/types.py",
line 986, in _verify_type
"length of fields (%d)" % (len(obj), len(dataType.fields)))
ValueError: Length of object (52) does not match with length of fields (7)
Does anyone know what this exactly means or can help me with a work around?
I agree - we need to see more code - obfuscated data is fine.
You are using SparkQL it seems (sql types) - mapped onto what ? HDFS/Text
From the error it would appear that your create schema is incorrect - leading to an error - when to create a Data Frame.
This was due to the passing of an incorrect RDD, sorry everyone. I was passing the incorrect RDD which caused didn't fit the code I was using.