Tabular Data to Nested JSON - mule-studio

I am trying to convert tabular query output to nested JSON using MuleSoft. My query output is like below:
----------------------------------
Customer | Order | Items
----------------------------------
C | Order1 | Itm1
C | Order1 | Itm2
C | Order2 | Itm1
C | Order2 | Itm4
C | Order3 | Itm3
C | Order3 | Itm4
and using Mule4 dataweave I am trying to convert it in flowing JSON output:
Customer: C
Orders: {
Order1:{
Items: {
Item: Item1
Item: Item2
}
}
Order2:{
Items: {
Item: Item1
Item: Item4
}
}
Order3:
}
So far I have tried code like below with no luck:
%dw 2.0
output application/json
---
payload map ((st, stindex) -> {
Customer: st.Customer,
Orders: payload filter(($.Customer == st.Customer) and ($.Order == st.Order)) map ((f, fIndex) ->{
Order: f.Order
Items : payload filter (($.ItemName == f.ItemName) and ($.Order == f.Order)) map ((i, iIndex) -> {
item: i.ItemName
})
})
})
it seems I am missing something important. Appreciate your help!!

I needed a multilevel groupBy(), mapObjects and a reduce() for the item.
%dw 2.0
output application/json
---
(payload map {
Customer: $.Customer,
Order: $.Order,
Items: $.Items
} groupBy (item) -> item.Customer)
mapObject ((value, key, index) -> {
Customer : key,
Orders: (value groupBy (orders)->orders.Order)
mapObject ((value1, key1, index1) -> {
"$(key1)": value1 reduce ((item, accumulator={}) -> accumulator ++ {Item: item.Items} )
}
)
}
)
Output:
{
"Customer": "C",
"Orders": {
"Order2": {
"Item": "Itm1",
"Item": "Itm4"
},
"Order1": {
"Item": "Itm1",
"Item": "Itm2"
},
"Order3": {
"Item": "Itm3",
"Item": "Itm4"
}
}
}
The indenting of the script is bad but it is too late for me to fix it.

Related

Convert JSON.RawMessage to JSON

I am using gqlgen, sqlx and pgx. And I'm trying to use a custom scalar for sqlx's types.JSONText.
I have this attributes jsonb field in items table.
-- migrations/001_up.sql
CREATE TABLE IF NOT EXISTS items (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
quantity INT NOT NULL,
attributes JSONB
);
I have these model structs:
// graph/model/item.go
type Item struct {
ID string `json:"id,omitempty" db:"id,omitempty"`
Quantity int `json:"quantity" db:"quantity"`
Attributes *Attributes `json:"attributes,omitempty" db:"attributes,omitempty"`
}
type Attributes types.JSONText
I have this graphql schema:
// graph/schema.graphql
type Item {
id: ID!
quantity: Int!
attributes: Attributes
}
scalar Attributes
I can successfully inserted into database, but got error at retrieving.
| id | quantity | attributes |
|---------------|----------|------------------------------------|
| 031e1489-... | 100 | {"size": "medium", "color": "red"} |
This is the log I got from db query:
>> items.Db: &{
031e1489-02c9-46d3-924d-6a2edf1ca3ba // id
100 // quantity
0xc000430600 // attributes
}
I tried to marshal attributes scalar:
// graph/model/item.go
...
func (a *Attributes) MarshalGQL(w io.Writer) {
b, _ := json.Marshal(a)
w.Write(b)
}
// Unmarshal here
...
Add custom scalar type in gqlgen.yml:
...
Attributes:
model:
- github.com/my-api/graph/model.Attributes
But I got the string instead of json:
{
"data": {
"item": {
"id": "031e1489-02c9-46d3-924d-6a2edf1ca3ba",
"quantity": 100,
"attributes": "eyJjb2xvciI6ICJyZWQifQ==",
}
}
}
The desired output is:
{
"data": {
"item": {
"id": "031e1489-02c9-46d3-924d-6a2edf1ca3ba",
"quantity": 100,
"attributes": {
"size": "medium",
"color": "red",
}
}
}
}
What I am doing wrong?
Here are my attempts:
If I removed pointer from Attributes in Item struct, gqlgen throws error:
go generate ./...
go: finding module for package github.com/my-api/graph/generated
generating core failed: type.gotpl: template: type.gotpl:49:28: executing "type.gotpl" at <$type.Elem.GO>: nil pointer evaluating *config.TypeReference.GOexit status 1
graph/resolver.go:3: running "go": exit status 1
make: *** [Makefile:2: gengql] Error 1
This returns the desired result, but I don't know how to use it with real data:
func (a *Attributes) MarshalGQL(w io.Writer) {
raw := json.RawMessage(`{"foo":"bar"}`)
j, _ := json.Marshal(&raw)
s := string(j)
w.Write([]byte(s))
}
Query result:
{
"data": {
"item": {
"id": "031e1489-02c9-46d3-924d-6a2edf1ca3ba",
"quantity": 100,
"attributes": {
"foo": "bar"
}
}
}
}
Attributes is defined using types.JSONText which is defined using json.RawMessage which is defined using []byte. This means that the underlying type of all 4 types, including []byte, is []byte, which in turn means that all of the 4 types can be converted to []byte.
Hence it should be enough to do this:
func (a *Attributes) MarshalGQL(w io.Writer) {
w.Write([]byte(*a))
}

How to query the additional column (other than siblings) added to pivot table?

Here is my RecipeIngredientPivot table schema to which I added an addition quantity column.
-----------------------------------------------------
id | ingredientID | recipeID | quantity
-----------------------------------------------------
| 1 | A001 | R001 | 100 |
| 2 | A002 | R001 | 50 |
| 3 | C004 | R001 | 23 |
| 4 | A001 | R002 | 75 |
I was able to create a quantity column to the Pivot table which has the ingredient_id and recipe_id as siblings by only using Pivot instead of ModifiablePivot.
extension RecipeIngredientPivot: Pivot {
init(_ recipe: Recipe, _ ingredient: Ingredient, _ quantity: Double) throws {
self.recipeID = try recipe.requireID()
self.ingredientID = try ingredient.requireID()
self.quantity = quantity
}
}
However, I am not sure how to query and get quantities of a recipe.
extension Recipe {
var ingredients: Siblings<Recipe, Ingredient, RecipeIngredientPivot> {
return siblings()
}
var quantities: [Double] {
// Not sure how to get this.
}
}
Any help is much appreciated.
I figured it out. I had to do query(on.req) & pivot(on: req) on the siblings() and wait for both to complete successfully before mapping on the results.
let ingredientFuture = try recipe.ingredients.query(on: req).all()
let pivotFuture = try recipe.ingredients.pivots(on: req).all()
return ingredientFuture.and(pivotFuture).map { ingredients, pivots in
for (ingredient,pivot) in zip(ingredients, pivots) {
print("\(ingredient) - \(pivot.quantity)")
}
}

Dictionary init(grouping:by:), 2 levels - simplifying code

Dictionary(init(grouping:by:)) works fine when I need to group array elements by some property.
What I get is:
{
key1: [
value1,
value2,
value3
],
key2: [
value4,
value5,
value6
]
}
However, I need to transform the array further, grouping every partition into smaller groups so, that the resulting data structure is having two layers:
{
key1: {
anotherKey1: [
value1,
],
anotherKey2: [
value2,
value3]
},
key1: {
anotherKey3: [
value4,
],
anotherKey4: [
value5,
value6]
},
}
What is the simplest way of achieving this result? Currently I have to iterate over the result of the 1st dictionary initializer call:
let grouped = Dictionary(grouping: Array(source), by: {$0.key1)
var grouped2 = [KeyType1 : [KeyType2? : [ValueType]]]()
for pair in grouped {
if let key = pair.key {
grouped2[key] = Dictionary(grouping: pair.value, by: {$0.key1})
}
}
print(grouped2)
And this gets me exactly the result I want: two-level dictionary of arrays.
But I suspect, there is a simpler way of achieving the same result, without manually interating over every key/value pair.
You can do this by doing a init(grouping:by:) call, followed by a mapValues call, which further maps every "group" into another dictionary. And this dictionary is going to be created by init(grouping:by:) again, using the second grouping key.
i.e.
let grouped = Dictionary(grouping: source, by: firstKey)
.mapValues { Dictionary(grouping: $0, by: secondKey) }
For example, to group a bunch of numbers first by their % 2 values, then by their % 4 values:
let source = [1,2,3,4,5,6,7,8,9,10,11,12,13]
let grouped = Dictionary(grouping: source, by: { $0 % 2 })
.mapValues { Dictionary.init(grouping: $0, by: { $0 % 4 }) }
print(grouped)

How to group Postgres data correctly

In PostgreSQL, I have such table:
| QUESTION_TEXT | CATEGORY | AGREE_PERCENT | DISAGREE_PERCENT |
|----------------------------------------|----------|---------------|------------------|
| Do you support the President's policy? | Policy | 50 | 50 |
| Do you support Democrats? | Policy | 32 | 68 |
| Do you support the Lannisters? | Cinema | 45 | 55 |
| Do you support Spielberg's work? | Cinema | 60 | 40 |
In my Go application with the help of gorm library I make SQL request to PostgreSQL database like that:
type Entry struct {
QuestionText string `json:"question_text"`
Category string `json:"category"`
AgreePercent float64 `json:"agree_percent"`
DisagreePercent float64 `json:"disagree_percent"`
}
rows, _ := database.DBGORM.Raw("SELECT * FROM SPECIFICATION").Rows()
for rows.Next() {
entry := &Entry{}
if err = rows.Scan(&entry.QuestionText, & entry.Category, &entry.AgreePercent, &entry.DisagreePercent); err != nil {
utils.Logger().Println(err)
}
}
How to get a similar result? As you can see each object inside the array is grouped by value in the category column:
[
{
category: "Policy",
questions: ["Do you support the President's policy?", "Do you support Democrats?"],
series: [
{
name: "Agree, %",
data: [50, 32]
},
{
name: "Disagree, %",
data: [50, 68]
},
]
},
{
category: "Cinema",
questions: ["Do you support the Lannisters?", "Do you support Spielberg's work?"],
series: [
{
name: "Agree, %",
data: [45, 60]
},
{
name: "Disagree, %",
data: [55, 40]
},
]
},
]
Well, I can't say that it's elegant way, but finally I solve my task:
// Create struct called "Series".
type Series struct {
Name string `json:"name"`
Data []float64 `json:"data"`
}
// Create struct called "Specification".
type Specification struct {
Category string `json:"category"`
Questions []string `json:"questions"`
Series []Series `json:"series"`
}
// Initialize an array of struct.
var specifications []Specification
// Initialize several variables.
var catogoryName string
var arrayQuestionText []string
var arrayAgreePercent []float64
var arrayDisagreePercent []float64
for rows.Next() {
// Check the change in the name of the category.
if entry.Category == catogoryName {
// Add new elements to arrays.
arrayQuestionText = append(arrayQuestionText, entry.QuestionText)
arrayDisagreePercent = append(arrayDisagreePercent, entry.DisagreePercent)
arrayAgreePercent = append(arrayAgreePercent, entry.AgreePercent)
} else {
if len(catogoryName) > 0 {
// Fill the struct with data.
specification := Specification{
Category: catogoryName,
Questions: arrayQuestionText,
Series: []Series{
{
Name: "Agree, %",
Data: arrayAgreePercent,
},
{
Name: "Disagree, %",
Data: arrayDisagreePercent,
},
},
}
// Add new struct to array.
specifications = append(specifications, specification)
}
}
// Update data in arrays.
catogoryName = entry.Category
arrayQuestionText = nil
arrayQuestionText = append(arrayQuestionText, entry.QuestionText)
arrayDisagreePercent = nil
arrayDisagreePercent = append(arrayDisagreePercent, entry.DisagreePercent)
arrayAgreePercent = nil
arrayAgreePercent = append(arrayAgreePercent, entry.AgreePercent)
}

How can I collapse a struct array?

I have a struct for my items:
struct item: Decodable {
var category_id: Int
var name: String
}
Which I'm receiving through JSONDecoder().decode into an array (items: [item]).
And I'd like to collapse it into a structure like this:
struct collapsed{
var category_id = Int()
var items = [String]()
}
So that I can loop through it more conveniently. How can I do that?
So I have this variable:
items: [item]
Which looks like this:
items
[0]
-category_id: 0
-name: "item1"
[1]
-category_id: 1
-name: "item2"
[2]
-category_id: 0
-name: "item3"
[3]
-category_id: 1
-name: "item4"
[4]
-category_id: 0
-name: "item5"
And I want to collapse it to this:
collapseditems: [collapsed]
Which should look something like this:
collapseditems
[0]
-category_id: 0
-items:
[0]: "item1"
[1]: "item3"
[2]: "item5"
[1]
-category_id: 1
-items:
[0]: "item2"
[1]: "item4"
First thing - struct names should start with uppercase letters.
There are a few ways you can translate your array of Item to an array of Collapsed. Here is one approach that first groups the array of Item into a dictionary keyed by category_id. Then that dictionary is mapped into an array of Collapsed. Finally that array is sorted by category_id.
struct Item: Decodable {
var category_id: Int
var name: String
}
struct Collapsed {
let category_id: Int
var items: [String]
}
// Your array of Item - here is some test data
let items: [Item] = [
Item(category_id: 0, name: "item1"),
Item(category_id: 1, name: "item2"),
Item(category_id: 0, name: "item3"),
Item(category_id: 1, name: "item4"),
Item(category_id: 0, name: "item5")
]
// Create a dictionary where the keys are category_id and the values are an array of Item with the same category_id
let mapping = Dictionary(grouping: items, by: { $0.category_id })
// Map that dictionary into an array of Sorted and then sort that array by category_id
let collapsedItems: [Collapsed] = mapping
.map { Collapsed(category_id: $0.key, items: $0.value.map { $0.name }) }
.sorted { $0.category_id < $1.category_id }
You can achieve this by using below code.
var sortedList: [sorted] = []
for i in item {
if let index = sortedList.index(where: { $0.category_id == i.category_id }) {
sortedList[index].items.append(i.name)
} else {
sortedList.append(sorted(category_id: i.category_id, items: [i.name]))
}
}
print(sortedList)
For first question, do you want to make a sorted list of item struct?
If yes, you can use sorted method of Array like this.
let sorted = items.sorted { $0.category_id < $1.category_id }
For Second question, first struct has default value for its properties, but second one hasn't. Second one, therefore, does NOT offer an empty constructor.