Jolt transformation chunk flat array into pairs of two - jolt

I am trying to group pairs of two in a flat list, note that the amount of pairs can be variable (e.g. 0, 2, 4, 6 and so on).
See intended input/output.
Input:
{
"coordinates": [
1.1,
5.1,
1.2,
5.3,
1.3,
5.5
]
}
Output:
{
"coordinates": [
[1.1, 5.1],
[1.2, 5.3],
[1.3, 5.5],
]
}
Is this something that can be easily achieved using a Jolt transformation?

Yes, you can achieve it by using successive transformations such as
[
//index each values seperately
{
"operation": "shift",
"spec": {
"*": {
"*": "&1.&"
}
}
},
// convert values to string type so as to prevent the issue of truncation of decimal parts of those values
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"*": "=toString(#(1,&))"
}
}
},
// exchange key-value pairs
{
"operation": "shift",
"spec": {
"*": {
"*": {
"$": "&2.#(0)"
}
}
}
},
{
// increment the values by 1 in order to prepare them for modular arithetic logic held in the following steps
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"*": "=intSum(#(1,&),1)"
}
}
},
{
// pairs means to have two components, then need to divide the values by 2 along with rounding to the nearest grater integer
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"*": "=divideAndRound(0,#(1,&),2)"
}
}
},
{
// exchange key-value pairs back while keeping the name of the object("coordinates")
"operation": "shift",
"spec": {
"*": {
"*": {
"$": "&2.#(0)"
}
}
}
},
{
// dissipate each component of the list to their proper place
"operation": "shift",
"spec": {
"*": {
"*": "&1[]"
}
}
}
]

Related

Convert Multiple List Of Strings to different objects in Jolt

I have multiple list of strings in my request array. I have to iterate each list and take its first elements and form an object as bucketList similarly second elements of each list and form another object. How can be this achieved in jolt?
Request Packet:
{
"recordSet": [
{
"id": "123",
"bucketIdArray": [ "M03", "M02" ],
"bucketNameArray": [ "EmployeeBucket200", "EmployeeBucket100" ],
"bucketUsageArray": [ "500000000", "5000" ],
"postBucketValueArray": [ "5004000", "5000" ]
},
{
"id": "456",
"bucketIdArray": [ "M04" ],
"bucketNameArray": [ "EmployeeBucket300" ],
"bucketUsageArray": [ "500000000" ],
"postBucketValueArray": [ "5004000" ]
}
]
}
Response Expected:
{
"datas": {
"historyDetails": [
{
"historyId": "123",
"bucketlist": [
{
"bucketId": "M03",
"bucketName": "EmployeeBucket200",
"bucketUsage": "500000000",
"postBucketValue": "5004000"
},
{
"bucketId": "M02",
"bucketName": "EmployeeBucket300",
"bucketUsage": "50000",
"postBucketValue": "5004000"
}
]
},
{
"historyId": "456",
"bucketlist": [
{
"bucketId": "M04",
"bucketName": "EmployeeBucket300",
"bucketUsage": "500000000",
"postBucketValue": "5004000"
}
]
}
]
}
}
You can use this shift spec
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"id": "datas.historyDetails[&1].&",
"*": { "*": "datas.historyDetails[&2].bucketlist[&].&1" }
}
}
}
}
]
If the Array strings needed to be removed from the key names, then each key-value pair should explicitly be specified for them such as
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"*": "datas.historyDetails[&1].&",
"bucketIdArray": { "*": "datas.historyDetails[&2].bucketlist[&].bucketId" },
"bucketNameArray": { "*": "datas.historyDetails[&2].bucketlist[&].bucketName" },
"bucketUsageArray": { "*": "datas.historyDetails[&2].bucketlist[&].bucketUsage" },
"postBucketValueArray": { "*": "datas.historyDetails[&2].bucketlist[&].postBucketValue" }
}
}
}
}
]
Another option would be adding three more steps before our original shift spec in order to prevent hardcoding of each keys individually with Array suffix such as
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"*": {
"$": "&2.&.key",
"#": "&2.&.val"
}
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"*": {
"key": "=split('Array',#(1,&))"
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": {
"val": "&.[&2].#(1,key[0])"
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": {
"id": "datas.historyDetails[&1].&",
"*": { "*": "datas.historyDetails[&2].bucketlist[&].&1" }
}
}
}
}
]

JOLT to join different properties

Source JSON:
[
{
"metrics": {
"cost_per_app_install": "0.08",
},
"dimensions": {
"stat_time_day": "2021-06-04 00:00:00",
"campaign_id": 170011516
}
},
{
"metrics": {
"cost_per_app_install": "12.3",
},
"dimensions": {
"stat_time_day": "2021-06-04 00:00:00",
"campaign_id": 17013
}
}
]
Expected:
[
{
"cost_per_app_install": "0.08",
"stat_time_day": "2021-06-04 00:00:00",
"campaign_id": 170011516
},
{
"cost_per_app_install": "12.3",
"stat_time_day": "2021-06-04 00:00:00",
"campaign_id": 17013
}
]
JOLT spec:
[
{
"operation": "shift",
"spec": {
"*": {
"metrics": "[&1]",
"dimensions": "[&1]"
}
}
}
]
I want to add data from dimensions and metrics properties as one combined record. But unable with my JOLT config. This config just removes metrics and dimensions namings. Can I do it only with shift operation?
You need one more step to roam the indexes such as
[
{
"operation": "shift",
"spec": {
"*": {
"*": { "*": "[&2].&" }
}
}
}
]
Btw, no need to explicitly list the key-value pairs for the current case, just "*": "[&1]" should be preferred rather than "metrics": "[&1]","dimensions": "[&1]" for the current pairs if it were the case.

Map Jolt Transformation

I am trying to transform the following input:
{
"test_types": {
"3": "Type3",
"21": "type21",
"16": "Type16",
"15": "type15"
},
"name": "BlackSquid",
"obj_name": "malware",
"value": "2341743",
"type": "16"
}
Into this output:
{
"test_types": {
"3": "Type3",
"21": "type21",
"16": "Type16",
"15": "type15"
},
"name": "BlackSquid",
"obj_name": "malware",
"value": "2341743",
"type": "16",
"type_name": "Type16"
}
I have tried to use a "modify-overwrite-beta" spec by hardcoding the key value which gives me the result I need:
[
{
"operation": "modify-overwrite-beta",
"spec": {
"type_name": "#(1,test_types.16)"
}
}
]
but I would like to use the "type" value dynamically. The following throws an exception but maybe something like this:
[
{
"operation": "modify-overwrite-beta",
"spec": {
"type_name": "#(1,test_types.#(1,type))"
}
}
]
No need to use modify-overwrite-beta spec, but condition based shift spec for the key type would suffice such as
[{
"operation": "shift",
"spec": {
"*": "&",
"type": {
"*": {
"#(2,type)": "type",
"#(2,test_types.&)": "type_name"
}
}
}
}]

transform from one list format to key-value format in JOLT

I have one scenario where I have to transform below json
{ "Authenticators": [
"ALPHA",
"BETA",
"GAMA"
] }
to
{ "availableAuth":"ALPHA,BETA,GAMA" }
Check this spec
[
{
"operation": "modify-overwrite-beta",
"spec": {
"Authenticators": "=join(',',#(1,Authenticators))"
}
}, {
"operation": "shift",
"spec": {
"Authenticators": "availableAuth"
}
}
]

Jolt grouping together and including new tags

I have the following input JSON and want to group the sub jsons according to device name.(Device name has to be extracted from address). Also, i want to add new tags to the new arrays formed(Each array contains jsons having that type of device name only).
I've already tried below spec but the array Name name is appearing as deviceName instead of custom Parameter(I don't know how to provide custom name. for example i want the array name as "Parameters"). I'm not able to extract the device name from address field.(Device Name should come as "bee"/"honey"). Also, I want to add a new field which should be there once for the new array (and not for each element).
Input Json:
{
"CID": "AND",
"parameters": [{
"address": "abc:api:honey",
"name": "CH1"
},
{
"address": "abc:api:honey",
"name": "CH2"
},
{
"address": "abc:api:bee",
"name": "lat"
},
{
"address": "abc:api:bee",
"name": "long"
}
],
"rNo": 1232
}
Expected Output:
[{
"ID": "AND_1232",
"parameters": [{
"deviceName": "honey",
"name": "CH1",
"locoId": 1232,
"CID": "AND"
},
{
"deviceName": "honey",
"name": "CH2",
"locoId": 1232,
"CID": "AND"
}
],
"SpData": {
}
},
{
"ID": "AND_1232",
"parameters": [{
"deviceName": "bee",
"name": "lat",
"locoId": 1232,
"CID": "AND"
},
{
"deviceName": "bee",
"name": "long",
"locoId": 1232,
"CID": "AND"
}
],
"SpData": {
}
}]
Spec I tried:
[
{
"operation": "shift",
"spec": {
"parameters": {
"*": {
"#(2,CID)": "&2.[&1].CID",
"*": "&2.[&1].&",
"#(2,rNo)": "&2.[&1].locoId"
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"ID": "=concat(#(0,CID),'_',#(0,rNo))",
"parameters": {
"*": {
"deviceName": "=substring(#(1,address),8,11)",
"SpData": {}
}
}
}
}, {
"operation": "shift",
"spec": {
"parameters": {
"*": {
"deviceName": {
"*": {
"#2": "&[]"
}
}
}
}
}
}
]
The following will output desired results, each operation does the following respectively:
Add ID by concatenating CID and rNo
Add rNo and CID to all parameters and add deviceName (see here for details on device name splitting).
Group records by deviceName using object
Add ID to each group
Convert temporary grouping to array
Add default SpData
[
{
"operation": "modify-default-beta",
"spec": {
"ID": "=concat(#(1,CID),'_',#(1,rNo))"
}
},
{
"operation": "shift",
"spec": {
"ID": "&",
"parameters": {
"*": {
"address": {
"*:*:*": {
"$(0,3)": "parameters.[&3].deviceName"
}
},
"#(0,name)": "parameters.[&].name",
"#(2,rNo)": "parameters.[&].locoId",
"#(2,CID)": "parameters.[&].CID"
}
}
}
},
{
"operation": "shift",
"spec": {
"ID": "&",
"parameters": {
"*": {
"deviceName": {
"*": {
"#(3,[&2])": "tmp.&.parameters.[]"
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"tmp": {
"*": {
"#(2,ID)": "&1.ID",
"parameters": {
"#": "&2.parameters"
}
}
}
}
},
{
"operation": "shift",
"spec": {
"*": "[]"
}
},
{
"operation": "default",
"spec": {
"*": {
"SpData": {}
}
}
}
]