How to remove the duplicate elements in arrays of JSON file using Datawave transformations? - mule-studio

Question

Here I want to remove duplicate accounts with the help of account_id for all accounts in this JSON file. There are 12 accounts in this JSON file need to duplicate accounts to make them unique. someone, Please help me resolve this.
Here total 12 accounts in input JSON payload
Input JSON:
{
"Accounts" : [
{
"account_id" :"c93c9cc0-26b4-11ed-a261-0242ac120002",
"account_Number" : 458554,
"account_name": "Ramu"
},
{
"account_id" : "e4b0efdc-26b5-11ed-a261-0242ac120002",
"accountNumber" : 741852,
"account_name": "Rajesh"
},
{
"account_id" : "027aa0f8-26b6-11ed-a261-0242ac120002",
"accountNumber" : 963258,
"account_name": "Harsha"
},
{
"account_id" : "09ebdd0c-26b6-11ed-a261-0242ac120002",
"accountNumber" : 852456,
"account_name": "Vamsi"
}
],
"address" : [
{
"street" : "XXXX",
"state" : "XXXX",
"country": "XXXX"
}
],
"Accounts" : [
{
"account_id" : "f4974e1e-26b5-11ed-a261-0242ac120002",
"accountNumber" : 246598,
"account_name": "Indu"
},
{
"account_id" :"2fa15b30-26b6-11ed-a261-0242ac120002",
"accountNumber" : 789789,
"account_name": "Suresh"
},
{
"account_id" : "c93c9cc0-26b4-11ed-a261-0242ac120002",
"accountNumber" : 458554,
"account_name": "Ramu"
},
{
"account_id" : "e4b0efdc-26b5-11ed-a261-0242ac120002",
"accountNumber" : 741852,
"account_name": "Rajesh"
}
],
"PhoneNumbers" :[
{
"phcountry" : "XXXX",
"phno" : "XXXX"
}
],
"Accounts" :[
{
"account_id" : "09ebdd0c-26b6-11ed-a261-0242ac120002",
"accountNumber" : 852456,
"account_name": "Vamsi"
},
{
"account_id" : "c93c9cc0-26b4-11ed-a261-0242ac120002",
"account_Number" : 458554,
"account_name": "Ramu"
},
{
"account_id" : "2ad45a96-b907-4e2e-ae90-4f429c3fc0e4",
"accountNumber" : 741852,
"account_name": "Savitri"
},
{
"account_id" :"f4974e1e-26b5-11ed-a261-0242ac120002",
"accountNumber" : 246598,
"account_name": "Indu"
}
]
}
Finally, I NEED the output of unique accounts along with other data present in this JSON file. thank you
Here total of 7 unique accounts in output payload
Expected output:
{
"Accounts" : [
{
"account_id" :"c93c9cc0-26b4-11ed-a261-0242ac120002",
"account_Number" : 458554,
"account_name": "Ramu"
},
{
"account_id" : "e4b0efdc-26b5-11ed-a261-0242ac120002",
"accountNumber" : 741852,
"account_name": "Rajesh"
},
{
"account_id" : "027aa0f8-26b6-11ed-a261-0242ac120002",
"accountNumber" : 963258,
"account_name": "Harsha"
},
{
"account_id" : "09ebdd0c-26b6-11ed-a261-0242ac120002",
"accountNumber" : 852456,
"account_name": "Vamsi"
}
],
"address" : [
{
"street" : "XXXX",
"state" : "XXXX",
"country": "XXXX"
}
],
"Accounts" : [
{
"account_id" : "f4974e1e-26b5-11ed-a261-0242ac120002",
"accountNumber" : 246598,
"account_name": "Indu"
},
{
"account_id" :"2fa15b30-26b6-11ed-a261-0242ac120002",
"accountNumber" : 789789,
"account_name": "Suresh"
}
],
"PhoneNumbers" :[
{
"phcountry" : "XXXX",
"phno" : "XXXX"
}
],
"Accounts" :[
{
"account_id" : "2ad45a96-b907-4e2e-ae90-4f429c3fc0e4",
"accountNumber" : 741852,
"account_name": "Savitri"
},
]
}

Answer 1

I will assume that the logic to remove duplicates is to remove any element from Accounts keys where their account_id is already present in an element in a previous Accounts key in the same JSON document. I created a script that looks for each Accounts key, finds the unique accounts_ids in previous elements and uses them to filter the current array. It doesn't not remove duplicates inside the same array, however that should be easy to add.
%dw 2.0
output application/json
fun unique_accounts_until_index(o, i)=flatten((o filterObject ((value, key, index) -> key as String == "Accounts" and index < i )).*Accounts).account_id distinctBy $
---
payload mapObject ((value, key, index) ->
(key): if (key as String == "Accounts")
value filter ( !( unique_accounts_until_index(payload, index) contains log("id", $.account_id)))
else value
)
Output for the input in the question:
{
"Accounts": [
{
"account_id": "c93c9cc0-26b4-11ed-a261-0242ac120002",
"account_Number": 458554,
"account_name": "Ramu"
},
{
"account_id": "e4b0efdc-26b5-11ed-a261-0242ac120002",
"accountNumber": 741852,
"account_name": "Rajesh"
},
{
"account_id": "027aa0f8-26b6-11ed-a261-0242ac120002",
"accountNumber": 963258,
"account_name": "Harsha"
},
{
"account_id": "09ebdd0c-26b6-11ed-a261-0242ac120002",
"accountNumber": 852456,
"account_name": "Vamsi"
}
],
"address": [
{
"street": "XXXX",
"state": "XXXX",
"country": "XXXX"
}
],
"Accounts": [
{
"account_id": "f4974e1e-26b5-11ed-a261-0242ac120002",
"accountNumber": 246598,
"account_name": "Indu"
},
{
"account_id": "2fa15b30-26b6-11ed-a261-0242ac120002",
"accountNumber": 789789,
"account_name": "Suresh"
}
],
"PhoneNumbers": [
{
"phcountry": "XXXX",
"phno": "XXXX"
}
],
"Accounts": [
{
"account_id": "2ad45a96-b907-4e2e-ae90-4f429c3fc0e4",
"accountNumber": 741852,
"account_name": "Savitri"
}
]
}