I'm new to JOLT, I'm facing some issue in getting the proper JSON after the jolt sepc transformation. Below are the given json, used Jolt Spec and expected output. But when I process the Jolt Spec I'm not getting ReportedQtyUom and ReportedQty data in the JSON output. Please anyone let me know where I'm going wrong. Thank You
Given JSON
[
{
"MaterialId": "na-103437",
"LocationId": "0TB9",
"OHReportingDate": "2020-08-18T20:57:16Z",
"TankNumber": "KTLA",
"stocks": [
{
"stockStatus": "OnHand",
"quantity": [
{
"uom": "KG",
"amount": "0.000",
"ISOUom": "KGM"
}
]
},
{
"stockStatus": "Damaged",
"quantity": [
{
"uom": "KG",
"amount": "0.000",
"ISOUom": "KGM"
}
]
},
{
"stockStatus": "QualityInspection",
"quantity": [
{
"uom": "KG",
"amount": "0.000",
"ISOUom": "KGM"
}
]
}
]
},
{
"MaterialId": "na-103437",
"LocationId": "0TB9",
"OHReportingDate": "2020-08-18T20:57:16Z",
"TankNumber": "KTLJ",
"stocks": [
{
"stockStatus": "OnHand",
"quantity": [
{
"uom": "KG",
"amount": "0.000",
"ISOUom": "KGM"
}
]
},
{
"stockStatus": "Damaged",
"quantity": [
{
"uom": "KG",
"amount": "0.000",
"ISOUom": "KGM"
}
]
},
{
"stockStatus": "QualityInspection",
"quantity": [
{
"uom": "KG",
"amount": "0.000",
"ISOUom": "KGM"
}
]
}
]
}
]
JOLT Spec Used
[
{
// Keeping the same three nested arrays structure,
// build all the output "elements" into a parallel
// structure, creating a "header" object and an
// array of attachment objects.
"operation": "shift",
"spec": {
"*": {
"MaterialId": "[&1].MaterialId",
"LocationId": "[&1].LocationId",
"OHReportingDate": "[&1].OHReportingDate",
"TankNumber": "[&1].TankNumber",
"stocks": {
"*": {
"stockStatus": {
"OnHand": {
"quantity": {
"*": {
"#(0,uom)": "[&7].POI",
"#(0,amount)": "[&7].XYZ"
}
}
}
}
}
}
}
}
}
]
Expected
[ {
"MaterialId" : "na-103437",
"LocationId" : "0TB9",
"OHReportingDate" : "2020-08-18T20:57:16Z",
"TankNumber" : "KTLA",
"ReportedQtyUom" : "KG",
"ReportedQty": "0.000"
}, {
"MaterialId" : "na-103437",
"LocationId" : "0TB9",
"OHReportingDate" : "2020-08-18T20:57:16Z",
"TankNumber" : "KTLJ",
"ReportedQtyUom" : "KG",
"ReportedQty" : "0.000"
}]
Your spec will check for stockStatus as onHand and tries to move one level, since OnHand is String null gets returned for the shifts written inside the quantity.
Once you check for stockStatus as onHand, traverse 2 level backwards and try shifting the values.
[
{
"operation": "shift",
"spec": {
"*": {
"MaterialId": "[&1].MaterialId",
"LocationId": "[&1].LocationId",
"OHReportingDate": "[&1].OHReportingDate",
"TankNumber": "[&1].TankNumber",
"stocks": {
"*": {
"stockStatus": {
"OnHand": {
"#(2,quantity.[0].uom)": "[&5].ReportedQtyUom",
"#(2,quantity.[0].amount)": "[&5].ReportedQty"
}
}
}
}
}
}
}
]
Related
I have a lot of complex JSON objects which are placed in the collection. For example:
{
"name": "Mike",
"price": "444",
"distance": 881,
"someFiend": 123,
"lots": [
{
"aa": "111",
"bb": "222"
},
{
"xx": "000"
}
],
"apps": [
{
"app": 1
},
{
"app": 2
}
]
}
I only want to project only those fields which are present in the following reference document:
{
"name": "",
"price": "",
"lots": [
{
"aa": "",
"bb": ""
}
]
}
Expected output:
{
"name": "Mike",
"price": "444",
"lots": [
{
"aa": "111",
"bb": "222"
}
]
}
Is there any way to iterate all documents in the collection and then filter out fields that are not present in the reference doc?
Get key value pair result of activities. get all activities under first elements term as key.
INPUT
[
{
"_id": "diamond",
"activities": [
[
{
"term": "11",
"sport_name": "football"
}
]
]
},
{
"_id": "topaz",
"activities": [
[
{
"term": "12",
"sport_name": "football"
}
],
[
{
"term": "11",
"sport_name": "football"
},
{
"term": "11",
"sport_name": "hand ball"
}
]
]
}
]
OUTPUT
[
{
"_id": "diamond",
"activities": [
{
"11": [
{
"term": "11",
"sport_name": "football"
}
]
}
]
},
{
"_id": "topaz",
"activities": [
{
"12": [
{
"term": "12",
"sport_name": "football"
}
]
},
{
"11": [
{
"term": "11",
"sport_name": "football"
},
{
"term": "11",
"sport_name": "hand ball"
}
]
}
]
}
]
You can use below aggregation
db.collection.aggregate([
{ "$project": {
"activities": {
"$arrayToObject": {
"$map": {
"input": "$activities",
"in": {
"k": { "$arrayElemAt": ["$$this.term", 0] },
"v": "$$this"
}
}
}
}
}}
])
MongoPlayground
I am new to JOLT transformation. I am trying to create a transform spec.
I have an list of categories in the object where I need to only transform few details.
My sample code and spec re shown below.
In "0/SYS_CATALOG_DESCRIPTION" list, I need to convert it to a String based on the lang, i.e for en_US, I need to get AA Products
end result will be "_description" : "AA Products"
The "subCategories" should give me the following result:
"subCategories": [
{
"_id": "ce_155584",
"_parentIds": ["ce_128375"],
"_description": "Filters" //based on lang = en_US
}
]
Sample JSON:
{
"total": 16,
"max_score": 2.2809339,
"hits": [
{
"_index": "bosch-dms-frontend-service_en_us_1558584002",
"_type": "categories",
"_id": "ce_128375",
"_score": 2.2809339,
"_source": {
"_parentIds": [
"1234"
],
"0/SYS_CATALOG_DESCRIPTION": [
{
"lang": "de_DE",
"value": "AA Produkte"
},
{
"lang": "en_US",
"value": "AA Products"
}
],
"subCategories": [
{
"_index": "bosch-dms-frontend-service_en_us_1558584002",
"_type": "categories",
"_id": "ce_155584",
"_score": 2.2809339,
"_source": {
"_parentIds": [
"ce_128375"
],
"0/SYS_CATALOG_DESCRIPTION": [
{
"lang": "en_US",
"value": "Filters"
},
{
"lang": "zh_CN",
"value": "AA Filters (CN)"
}
],
"0/SYS_SYSTEMNAME": "AA_Filters"
}
}
]
}
}
]
}
SPEC:
[
{
"operation": "shift", // shift operation
"spec": {
"hits": {
"*": {
"_id": "_id",
"_source": {
"_parentIds": "_parentIds",
"0/SYS_CATALOG_DESCRIPTION": "_description",
}
}
}
}
}
]
The end result will be
{
"_id" : "ce_128375",
"_parentIds" : [ "1234" ],
"_description" : "AA Products (BR)",
"subCategories": [
{
"_id": "ce_155584",
"_score": 2.2809339,
"_parentIds": ["ce_128375"],
"_description" : "Filters"
}
]
}
I tried several ways but could not achieve the result.
Thank you.
Check if this spec is what you need:
[
{
"operation": "shift", // shift operation
"spec": {
"hits": {
"*": {
"_id": ["&",
"subCategories.[]._parentIds[]"],
"_source": {
"_parentIds": "&",
"0/SYS_CATALOG_DESCRIPTION": {
"*": {
"lang": {
"en_US": {
"#(2,value)": "_description"
}
}
}
},
"subCategories": {
"*": {
"_id": "subCategories.[&1].&",
"_score": "subCategories.[&1].&",
"_source": {
"0/SYS_CATALOG_DESCRIPTION": {
"*": {
"lang": {
"en_US": {
"#(2,value)": "subCategories.[&6]._description"
}
}
}
}
}
}
}
}
}
}
}
}
]
I have a jsonb data of following format, with nested arrays
{
"outerArray": [
{
"price": {
"amount": 108.95,
"currencyCode": "GBP"
},
"innerArray": [
{
"details": {
"field1": "val1",
"field2": "val2",
"field3": "val3"
},
"otherDetail": {
"date": "2016-07-23",
"time": "19:43:00"
},
"innerMostArray": [
{
"A1": "A1"
},
{
"B1": "B1"
}
]
}
],
"someField": "values"
},
{
"price": {
"amount": 108.95,
"currencyCode": "GBP"
},
"innerArray": [
{
"details": {
"field1": "val1",
"field2": "val2",
"field3": "val3"
},
"otherDetail": {
"date": "2016-07-23",
"time": "19:43:00"
},
"innerMostArray": [
{
"A1": "A1"
},
{
"B1": "B1"
}
]
}
],
"someField": "values"
}
]
}
I want to write a retrieve query on this, to maintain same json structure but hide fields "price", "details" , "otherDetail"and "someField"
The retrieved result should look like this
{
"outerArray": [
{
"innerArray": [
{
"innerMostArray": [
{
"A1": "A1"
},
{
"B1": "B1"
}
]
}
]
},
{
"innerArray": [
{
"innerMostArray": [
{
"A1": "A1"
},
{
"B1": "B1"
}
]
}
]
}
]
}
Can this be done?
Please always specify a version of PostgreSQL you are using. An example below should work fine for versions v9.5+.
I would approach this by building a JSONB object you need with jsonb_build_object() and jsonb_build_array() functions:
Sample query:
WITH test(data) AS ( VALUES
('{
"outerArray": [
{
"price": {
"amount": 108.95,
"currencyCode": "GBP"
},
"innerArray": [
{
"details": {
"field1": "val1",
"field2": "val2",
"field3": "val3"
},
"otherDetail": {
"date": "2016-07-23",
"time": "19:43:00"
},
"innerMostArray": [
{
"A1": "A1"
},
{
"B1": "B1"
}
]
}
],
"someField": "values"
},
{
"price": {
"amount": 108.95,
"currencyCode": "GBP"
},
"innerArray": [
{
"details": {
"field1": "val1",
"field2": "val2",
"field3": "val3"
},
"otherDetail": {
"date": "2016-07-23",
"time": "19:43:00"
},
"innerMostArray": [
{
"A1": "A1"
},
{
"B1": "B1"
}
]
}
],
"someField": "values"
}
]}'::JSONB)
)
SELECT
jsonb_build_object(
'outerArray',
array_agg(
jsonb_build_object(
'innerArray',
json_build_array(
json_build_object(
'innerMostArray',
innerArray->'innerMostArray')
)
)
)
) as result
FROM test t,
jsonb_array_elements(t.data->'outerArray') as outerElement,
jsonb_array_elements(outerElement->'innerArray') as innerArray;
Result:
result
----------------------------------------------------------------------------------------------------------------------------------------------------------
{"outerArray": [{"innerArray": [{"innerMostArray": [{"A1": "A1"}, {"B1": "B1"}]}]}, {"innerArray": [{"innerMostArray": [{"A1": "A1"}, {"B1": "B1"}]}]}]}
(1 row)
Google Analytics API documentation shows that, for fetching the lifetime values, the date ranges should not be specified. But when I make such a request (without date range), it returns empty dimension and metrics result. But when I use date range, it returns dimension and metrics values for that date range.
The following is an excerpt from the API documentation :
Date ranges should not be specified for cohorts or Lifetime value
requests.
For example, if I make the request without date range, as follows:
{
"reportRequests": [
{
"viewId": "XXXXXXXXX",
"dimensions": [
{
"name": "ga:date"
},
{
"name": "ga:eventLabel"
}
],
"metrics": [
{
"expression": "ga:totalEvents"
}
]
}
]
}
I get the following response:
{
"reports": [
{
"columnHeader": {
"dimensions": [
"ga:date",
"ga:eventLabel"
],
"metricHeader": {
"metricHeaderEntries": [
{
"name": "ga:totalEvents",
"type": "INTEGER"
}
]
}
},
"data": {
"totals": [
{
"values": [
"0"
]
}
]
}
}
]
}
However, if I include the date range,
{
"reportRequests": [
{
"viewId": "XXXXXXXX",
"dimensions": [
{
"name": "ga:date"
},
{
"name": "ga:eventLabel"
}
],
"metrics": [
{
"expression": "ga:totalEvents"
}
],
"dateRanges": [
{
"startDate": "2016-01-01",
"endDate": "2016-04-30"
}
]
}
]
}
I get the following response:
{
"reports": [
{
"columnHeader": {
"dimensions": [
"ga:date",
"ga:eventLabel"
],
"metricHeader": {
"metricHeaderEntries": [
{
"name": "ga:totalEvents",
"type": "INTEGER"
}
]
}
},
"data": {
"rows": [
{
"dimensions": [
"20160412",
"http://mytestblog.com/"
],
"metrics": [
{
"values": [
"1"
]
}
]
},
{
"dimensions": [
"20160412",
"http://mytestblog.com/2016/04/first-post.html"
],
"metrics": [
{
"values": [
"3"
]
}
]
},
{
"dimensions": [
"20160419",
"http://mytestblog.com/"
],
"metrics": [
{
"values": [
"4"
]
}
]
},
{
"dimensions": [
"20160419",
"http://mytestblog.com/2016/04/fourth.html"
],
"metrics": [
{
"values": [
"13"
]
}
]
}
],
"totals": [
{
"values": [
"21"
]
}
],
"rowCount": 4,
"minimums": [
{
"values": [
"1"
]
}
],
"maximums": [
{
"values": [
"13"
]
}
]
}
}
]
}
Why is it that, even though specified in the documentation, I have to specify date range in the ReportRequest to get the values? Am I misunderstanding the meaning of Lifetime values here?
The reportRequest object should have either a value for dateRanges or a definition value for cohortGroup. When you omit both the requests assumes the default values for a startDate of 7daysAgo and an endDate of yesterday.
The correct interpretation of the docs is that the reportRequest should not have a dateRange defined for cohort and LTV requests. But in order to make a cohort or lifetime value request you must add a cohort definition. For Lifetime value requests the cohort definition should have a specific dateRange in addition to the lifetimeValue field set to true:
POST https://analyticsreporting.googleapis.com/v4/reports:batchGet
{
"reportRequests": [
{
"viewId": "XXXX",
"dimensions": [
{"name": "ga:cohort" },
{"name": "ga:cohortNthWeek" }],
"metrics": [
{"expression": "ga:cohortTotalUsersWithLifetimeCriteria"},
{"expression": "ga:cohortRevenuePerUser"}
],
"cohortGroup": {
"cohorts": [{
"name": "cohort 1",
"type": "FIRST_VISIT_DATE",
"dateRange": {
"startDate": "2015-08-01",
"endDate": "2015-09-01"
}
},
{
"name": "cohort 2",
"type": "FIRST_VISIT_DATE",
"dateRange": {
"startDate": "2015-07-01",
"end_date": "2015-08-01"
}
}],
"lifetimeValue": True
}
}]
}