How to modify the execution plan in Spark?

How to modify the execution plan in Spark? - scala

I am getting some execution plans in json format.
val df: DataFrame = ???
val jsonPlan = df.queryExecution.optimizedPlan.toJSON
jsonPlan output:
You can see that the InMemoryRelation operator has the child key: "child": [..].
Is it possible to add user information to the plan tree node before calling the toJSON method? So that after calling toJSON, the added information is serialized along with the tree node.
For example I want to add labels for all child nodes (label - it a cute child).
Then the result will be like this:
"child":[
{
"class":"org.apache.spark.sql.execution.WholeStageCodegenExec",
"num-children":1,
"child":0,
"codegenStageId":2
// My Added Information
"label": "it a cute child"
},
...
]
Full json plan:
[
{
"class":"org.apache.spark.sql.execution.columnar.InMemoryRelation",
"num-children":0,
"output":[
[
{
"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference",
"num-children":0,
"name":"criminal_name",
"dataType":"string",
"nullable":true,
"metadata":{
},
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":73,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":[
]
}],
[
{
"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference",
"num-children":0,
"name":"punishment",
"dataType":"string",
"nullable":true,
"metadata":{
},
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":78,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":[
]
}]],
"cacheBuilder":null,
"outputOrdering":[
],
"child":[
{
"class":"org.apache.spark.sql.execution.WholeStageCodegenExec",
"num-children":1,
"child":0,
"codegenStageId":2
},
{
"class":"org.apache.spark.sql.execution.ProjectExec",
"num-children":1,
"projectList":[
[
{
"class":"org.apache.spark.sql.catalyst.expressions.Alias",
"num-children":1,
"child":0,
"name":"criminal_name",
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":73,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":[
],
"explicitMetadata":{
}
},
{
"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference",
"num-children":0,
"name":"name",
"dataType":"string",
"nullable":true,
"metadata":{
},
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":56,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":"[spark_catalog, murphy, staff_dossiers]"
}],
[
{
"class":"org.apache.spark.sql.catalyst.expressions.Alias",
"num-children":1,
"child":0,
"name":"punishment",
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":78,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":[
],
"explicitMetadata":{
}
},
{
"class":"org.apache.spark.sql.catalyst.expressions.ScalaUDF",
"num-children":1,
"function":null,
"dataType":"string",
"children":[
0],
"inputEncoders":null,
"nullable":true,
"udfDeterministic":true
},
{
"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference",
"num-children":0,
"name":"hobby",
"dataType":"string",
"nullable":true,
"metadata":{
},
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":64,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":"[spark_catalog, murphy, indicators]"
}]],
"child":0
},
{
"class":"org.apache.spark.sql.execution.joins.BroadcastHashJoinExec",
"num-children":2,
"leftKeys":[
[
{
"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference",
"num-children":0,
"name":"name",
"dataType":"string",
"nullable":true,
"metadata":{
},
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":56,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":"[spark_catalog, murphy, staff_dossiers]"
}]],
"rightKeys":[
[
{
"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference",
"num-children":0,
"name":"name",
"dataType":"string",
"nullable":true,
"metadata":{
},
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":62,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":"[spark_catalog, murphy, indicators]"
}]],
"joinType":{
"object":"org.apache.spark.sql.catalyst.plans.Inner$"
},
"buildSide":{
"object":"org.apache.spark.sql.execution.joins.package$BuildLeft$"
},
"left":0,
"right":1
},
{
"class":"org.apache.spark.sql.execution.InputAdapter",
"num-children":1,
"child":0
},
{
"class":"org.apache.spark.sql.execution.exchange.BroadcastExchangeExec",
"num-children":1,
"mode":{
"product-class":"org.apache.spark.sql.execution.joins.HashedRelationBroadcastMode",
"key":[
[
{
"class":"org.apache.spark.sql.catalyst.expressions.BoundReference",
"num-children":0,
"ordinal":0,
"dataType":"string",
"nullable":true
}]]
},
"child":0
},
{
"class":"org.apache.spark.sql.execution.WholeStageCodegenExec",
"num-children":1,
"child":0,
"codegenStageId":1
},
{
"class":"org.apache.spark.sql.execution.ProjectExec",
"num-children":1,
"projectList":[
[
{
"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference",
"num-children":0,
"name":"name",
"dataType":"string",
"nullable":true,
"metadata":{
},
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":56,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":[
]
}]],
"child":0
},
{
"class":"org.apache.spark.sql.execution.FilterExec",
"num-children":1,
"condition":[
{
"class":"org.apache.spark.sql.catalyst.expressions.And",
"num-children":2,
"left":0,
"right":1
},
{
"class":"org.apache.spark.sql.catalyst.expressions.IsNotNull",
"num-children":1,
"child":0
},
{
"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference",
"num-children":0,
"name":"name",
"dataType":"string",
"nullable":true,
"metadata":{
},
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":56,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":"[spark_catalog, murphy, staff_dossiers]"
},
{
"class":"org.apache.spark.sql.catalyst.expressions.EqualTo",
"num-children":2,
"left":0,
"right":1
},
{
"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference",
"num-children":0,
"name":"name",
"dataType":"string",
"nullable":true,
"metadata":{
},
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":56,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":"[spark_catalog, murphy, staff_dossiers]"
},
{
"class":"org.apache.spark.sql.catalyst.expressions.Literal",
"num-children":0,
"value":"Nikita",
"dataType":"string"
}],
"child":0
},
{
"class":"org.apache.spark.sql.execution.ColumnarToRowExec",
"num-children":1,
"child":0
},
{
"class":"org.apache.spark.sql.execution.InputAdapter",
"num-children":1,
"child":0
},
{
"class":"org.apache.spark.sql.execution.FileSourceScanExec",
"num-children":0,
"relation":null,
"output":[
[
{
"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference",
"num-children":0,
"name":"name",
"dataType":"string",
"nullable":true,
"metadata":{
},
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":56,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":[
]
}]],
"requiredSchema":{
"type":"struct",
"fields":[
{
"name":"name",
"type":"string",
"nullable":true,
"metadata":{
}
}]
},
"partitionFilters":[
],
"dataFilters":[
[
{
"class":"org.apache.spark.sql.catalyst.expressions.IsNotNull",
"num-children":1,
"child":0
},
{
"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference",
"num-children":0,
"name":"name",
"dataType":"string",
"nullable":true,
"metadata":{
},
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":56,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":"[spark_catalog, murphy, staff_dossiers]"
}],
[
{
"class":"org.apache.spark.sql.catalyst.expressions.EqualTo",
"num-children":2,
"left":0,
"right":1
},
{
"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference",
"num-children":0,
"name":"name",
"dataType":"string",
"nullable":true,
"metadata":{
},
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":56,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":"[spark_catalog, murphy, staff_dossiers]"
},
{
"class":"org.apache.spark.sql.catalyst.expressions.Literal",
"num-children":0,
"value":"Nikita",
"dataType":"string"
}]],
"tableIdentifier":{
"product-class":"org.apache.spark.sql.catalyst.TableIdentifier",
"table":"staff_dossiers",
"database":"murphy"
}
},
{
"class":"org.apache.spark.sql.execution.ProjectExec",
"num-children":1,
"projectList":[
[
{
"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference",
"num-children":0,
"name":"name",
"dataType":"string",
"nullable":true,
"metadata":{
},
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":62,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":[
]
}],
[
{
"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference",
"num-children":0,
"name":"hobby",
"dataType":"string",
"nullable":true,
"metadata":{
},
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":64,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":[
]
}]],
"child":0
},
{
"class":"org.apache.spark.sql.execution.FilterExec",
"num-children":1,
"condition":[
{
"class":"org.apache.spark.sql.catalyst.expressions.And",
"num-children":2,
"left":0,
"right":1
},
{
"class":"org.apache.spark.sql.catalyst.expressions.EqualTo",
"num-children":2,
"left":0,
"right":1
},
{
"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference",
"num-children":0,
"name":"name",
"dataType":"string",
"nullable":true,
"metadata":{
},
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":62,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":"[spark_catalog, murphy, indicators]"
},
{
"class":"org.apache.spark.sql.catalyst.expressions.Literal",
"num-children":0,
"value":"Nikita",
"dataType":"string"
},
{
"class":"org.apache.spark.sql.catalyst.expressions.IsNotNull",
"num-children":1,
"child":0
},
{
"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference",
"num-children":0,
"name":"name",
"dataType":"string",
"nullable":true,
"metadata":{
},
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":62,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":"[spark_catalog, murphy, indicators]"
}],
"child":0
},
{
"class":"org.apache.spark.sql.execution.ColumnarToRowExec",
"num-children":1,
"child":0
},
{
"class":"org.apache.spark.sql.execution.InputAdapter",
"num-children":1,
"child":0
},
{
"class":"org.apache.spark.sql.execution.FileSourceScanExec",
"num-children":0,
"relation":null,
"output":[
[
{
"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference",
"num-children":0,
"name":"name",
"dataType":"string",
"nullable":true,
"metadata":{
},
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":62,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":[
]
}],
[
{
"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference",
"num-children":0,
"name":"hobby",
"dataType":"string",
"nullable":true,
"metadata":{
},
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":64,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":[
]
}]],
"requiredSchema":{
"type":"struct",
"fields":[
{
"name":"name",
"type":"string",
"nullable":true,
"metadata":{
}
},
{
"name":"hobby",
"type":"string",
"nullable":true,
"metadata":{
}
}]
},
"partitionFilters":[
],
"dataFilters":[
[
{
"class":"org.apache.spark.sql.catalyst.expressions.EqualTo",
"num-children":2,
"left":0,
"right":1
},
{
"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference",
"num-children":0,
"name":"name",
"dataType":"string",
"nullable":true,
"metadata":{
},
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":62,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":"[spark_catalog, murphy, indicators]"
},
{
"class":"org.apache.spark.sql.catalyst.expressions.Literal",
"num-children":0,
"value":"Nikita",
"dataType":"string"
}],
[
{
"class":"org.apache.spark.sql.catalyst.expressions.IsNotNull",
"num-children":1,
"child":0
},
{
"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference",
"num-children":0,
"name":"name",
"dataType":"string",
"nullable":true,
"metadata":{
},
"exprId":{
"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId",
"id":62,
"jvmId":"2996f433-9e44-40c1-9aaf-d74c0768f68b"
},
"qualifier":"[spark_catalog, murphy, indicators]"
}]],
"tableIdentifier":{
"product-class":"org.apache.spark.sql.catalyst.TableIdentifier",
"table":"indicators",
"database":"murphy"
}
}
]
}
]

Sure, you can parse and modify any JSON object in memory, but that has nothing to do with Spark. Related: What JSON library to use in Scala?
Any modifications you make wouldn't be persisted within the execution plan itself.

Related

MongoDB aggregation re-group deep nested array of objects

Good day all,
I need some help figuring out this aggregation issue. I have a document with nested arrays where I want to perform lookups on the nested array's values after which I want to restore the document to its original structure. see below:
[
{
"_id": "63c7fec2fe9afea23afdbcef",
"primary_language": "en",
"image_link": "avatar_image_linl",
"description": ["Some description text here"],
"source": "community",
"standard": "imperial",
"gender": "male",
"base_a": 47,
"base_w": 220,
"base_h": 71.65,
"status": "draft",
"date_created": "2023-01-18T14:14:26.201Z",
"product_plan": {
"p_len": 4,
"p_fre": 1,
"p_qua": 4,
"avg_len": 3583,
"total_wor": 16,
"total_exe": 128,
"plan": {
"field_we": 1,
"field_da": 1,
"field_ti": 1,
"field_rest_we": false,
"field_rest_da": false,
"product": {
"name": "W111",
"comment": {
"en": ""
},
"pro_diff": [
{
"itemID": 2,
"description": "Intermediate"
}
],
"pro_int": [
{
"itemID": 2,
"description": "Moderate"
}
],
"pro_dur": 3736,
"nbr_exe": 8,
"target_mus": [
{
"itemID": 1,
"description": "Adominorus"
},
{
"itemID": 16,
"description": "Tricerus"
},
{
"itemID": 5,
"description": "Chiwawak"
}
],
"target_are": [
{
"itemID": 1,
"description": "Adominorus"
},
{
"itemID": 4,
"description": "Chiwawak"
},
{
"itemID": 2,
"description": "Amanus"
}
],
"items": {
"itemid": 44,
"target_are": [
{
"itemID": 4,
"description": "Chiwawak"
}
],
"target_mus": [
{
"itemID": 5,
"description": "Chiwawak"
}
],
"nbr_ref": 4,
"int_rec": "06~10",
"type_rec": [
{
"itemID": 14,
"description": "Straight Set"
}
],
"tempo": "1-1-1-1",
"rest_time": 90,
"note": "",
"order": null,
"pro_dur": 520,
"_id": "63ca9b798d47745ae5589906",
"status": 0,
"pct_complete": 0,
"set": []
},
"isCustomName": false,
"status": 0,
"pct_complete": 0
},
"_id": "63ca95368d47745ae558985e"
},
"_id": "63c7fec2fe9afea23afdbcf0"
},
"title": "Short Title text here",
"itemID": "bodyfyme882212-1674051266.214",
"__v": 32
},
{
"_id": "63c7fec2fe9afea23afdbcef",
"primary_language": "en",
"image_link": "avatar_image_linl",
"description": ["Some description text here"],
"source": "community",
"standard": "imperial",
"gender": "male",
"base_a": 47,
"base_w": 220,
"base_h": 71.65,
"status": "draft",
"date_created": "2023-01-18T14:14:26.201Z",
"product_plan": {
"p_len": 4,
"p_fre": 1,
"p_qua": 4,
"avg_len": 3583,
"total_wor": 16,
"total_exe": 128,
"plan": {
"field_we": 1,
"field_da": 1,
"field_ti": 1,
"field_rest_we": false,
"field_rest_da": false,
"product": {
"name": "W111",
"comment": {
"en": ""
},
"pro_diff": [
{
"itemID": 2,
"description": "Intermediate"
}
],
"pro_int": [
{
"itemID": 2,
"description": "Moderate"
}
],
"pro_dur": 3736,
"nbr_exe": 8,
"target_mus": [
{
"itemID": 1,
"description": "Adominorus"
},
{
"itemID": 16,
"description": "Tricerus"
},
{
"itemID": 5,
"description": "Chiwawak"
}
],
"target_are": [
{
"itemID": 1,
"description": "Adominorus"
},
{
"itemID": 4,
"description": "Chiwawak"
},
{
"itemID": 2,
"description": "Amanus"
}
],
"items": {
"itemid": 339,
"target_are": [
{
"itemID": 4,
"description": "Chiwawak"
}
],
"target_mus": [
{
"itemID": 5,
"description": "Chiwawak"
}
],
"nbr_ref": 3,
"int_rec": "06~10",
"type_rec": [
{
"itemID": 14,
"description": "Straight Set"
}
],
"tempo": "1-1-1-1",
"rest_time": 90,
"note": "",
"order": null,
"pro_dur": 390,
"_id": "63ca9b798d47745ae5589907",
"status": 0,
"pct_complete": 0,
"set": []
},
"isCustomName": false,
"status": 0,
"pct_complete": 0
},
"_id": "63ca95368d47745ae558985e"
},
"_id": "63c7fec2fe9afea23afdbcf0"
},
"title": "Short Title text here",
"itemID": "bodyfyme882212-1674051266.214",
"__v": 32
},
{
"_id": "63c7fec2fe9afea23afdbcef",
"primary_language": "en",
"image_link": "avatar_image_linl",
"description": ["Some description text here"],
"source": "community",
"standard": "imperial",
"gender": "male",
"base_a": 47,
"base_w": 220,
"base_h": 71.65,
"status": "draft",
"date_created": "2023-01-18T14:14:26.201Z",
"product_plan": {
"p_len": 4,
"p_fre": 1,
"p_qua": 4,
"avg_len": 3583,
"total_wor": 16,
"total_exe": 128,
"plan": {
"field_we": 1,
"field_da": 2,
"field_ti": 1,
"field_rest_we": false,
"field_rest_da": false,
"product": {
"name": "W121",
"comment": {
"en": ""
},
"pro_diff": [
{
"itemID": 2,
"description": "Intermediate"
}
],
"pro_int": [
{
"itemID": 2,
"description": "Moderate"
}
],
"pro_dur": 2674,
"nbr_exe": 6,
"target_mus": [
{
"itemID": 9,
"description": "Lats"
},
{
"itemID": 10,
"description": "Lower Back"
},
{
"itemID": 11,
"description": "Middle Back"
}
],
"target_are": [
{
"itemID": 3,
"description": "Back"
}
],
"items": {
"itemid": 47,
"target_are": [
{
"itemID": 3,
"description": "Back"
}
],
"target_mus": [
{
"itemID": 10,
"description": "Lower Back"
}
],
"nbr_ref": 4,
"int_rec": "06~10",
"type_rec": [
{
"itemID": 14,
"description": "Straight Set"
}
],
"tempo": "1-1-1-1",
"rest_time": 90,
"note": "",
"order": null,
"pro_dur": 520,
"_id": "63caa1568d47745ae5589b25",
"status": 0,
"pct_complete": 0,
"set": []
},
"isCustomName": false,
"status": 0,
"pct_complete": 0
},
"_id": "63ca9b8e8d47745ae558994b"
},
"_id": "63c7fec2fe9afea23afdbcf0"
},
"title": "Short Title text here",
"itemID": "bodyfyme882212-1674051266.214",
"__v": 32
},
{
"_id": "63c7fec2fe9afea23afdbcef",
"primary_language": "en",
"image_link": "avatar_image_linl",
"description": ["Some description text here"],
"source": "community",
"standard": "imperial",
"gender": "male",
"base_a": 47,
"base_w": 220,
"base_h": 71.65,
"status": "draft",
"date_created": "2023-01-18T14:14:26.201Z",
"product_plan": {
"p_len": 4,
"p_fre": 1,
"p_qua": 4,
"avg_len": 3583,
"total_wor": 16,
"total_exe": 128,
"plan": {
"field_we": 1,
"field_da": 2,
"field_ti": 1,
"field_rest_we": false,
"field_rest_da": false,
"product": {
"name": "W121",
"comment": {
"en": ""
},
"pro_diff": [
{
"itemID": 2,
"description": "Intermediate"
}
],
"pro_int": [
{
"itemID": 2,
"description": "Moderate"
}
],
"pro_dur": 2674,
"nbr_exe": 6,
"target_mus": [
{
"itemID": 9,
"description": "Lats"
},
{
"itemID": 10,
"description": "Lower Back"
},
{
"itemID": 11,
"description": "Middle Back"
}
],
"target_are": [
{
"itemID": 3,
"description": "Back"
}
],
"items": {
"itemid": 495,
"target_are": [
{
"itemID": 3,
"description": "Back"
}
],
"target_mus": [
{
"itemID": 11,
"description": "Middle Back"
}
],
"nbr_ref": 3,
"int_rec": "10~12",
"type_rec": [
{
"itemID": 14,
"description": "Straight Set"
}
],
"tempo": "1-1-1-1",
"rest_time": 90,
"note": "",
"order": null,
"pro_dur": 414,
"_id": "63caa1568d47745ae5589b26",
"status": 0,
"pct_complete": 0,
"set": []
},
"isCustomName": false,
"status": 0,
"pct_complete": 0
},
"_id": "63ca9b8e8d47745ae558994b"
},
"_id": "63c7fec2fe9afea23afdbcf0"
},
"title": "Short Title text here",
"itemID": "bodyfyme882212-1674051266.214",
"__v": 32
},
{
......
}
]
Now i want to get the document back to this structure using aggregation.
{
"_id": "63c7fec2fe9afea23afdbcef",
"primary_language": "en",
"image_link": "avatar_image_linl",
"description": ["Some description text here"],
"source": "community",
"standard": "imperial",
"gender": "male",
"base_a": 47,
"base_w": 220,
"base_h": 71.65,
"status": "draft",
"date_created": "2023-01-18T14:14:26.201Z",
"product_plan": {
"p_len": 4,
"p_fre": 1,
"p_qua": 4,
"avg_len": 3583,
"total_wor": 16,
"total_exe": 128,
// Group plan here by product_plan._id
"plan": [
{
"field_we": 1,
"field_da": 1,
"field_ti": 1,
"field_rest_we": false,
"field_rest_da": false,
"product": {
"name": "W111",
"comment": {
"en": ""
},
"pro_diff": [
{
"itemID": 2,
"description": "Intermediate"
}
],
"pro_int": [
{
"itemID": 2,
"description": "Moderate"
}
],
"pro_dur": 3736,
"nbr_exe": 8,
"target_mus": [
{
"itemID": 1,
"description": "Adominorus"
},
{
"itemID": 16,
"description": "Tricerus"
},
{
"itemID": 5,
"description": "Chiwawak"
}
],
"target_are": [
{
"itemID": 1,
"description": "Adominorus"
},
{
"itemID": 4,
"description": "Chiwawak"
},
{
"itemID": 2,
"description": "Amanus"
}
],
// group items by plan._id
"items": [
{
"itemid": 44,
"target_are": [
{
"itemID": 4,
"description": "Chiwawak"
}
],
"target_mus": [
{
"itemID": 5,
"description": "Chiwawak"
}
],
"nbr_ref": 4,
"int_rec": "06~10",
"type_rec": [
{
"itemID": 14,
"description": "Straight Set"
}
],
"tempo": "1-1-1-1",
"rest_time": 90,
"note": "",
"order": null,
"pro_dur": 520,
"_id": "63ca9b798d47745ae5589906",
"status": 0,
"pct_complete": 0,
"set": []
},
{
"itemid": 339,
"target_are": [
{
"itemID": 4,
"description": "Chiwawak"
}
],
"target_mus": [
{
"itemID": 5,
"description": "Chiwawak"
}
],
"nbr_ref": 3,
"int_rec": "06~10",
"type_rec": [
{
"itemID": 14,
"description": "Straight Set"
}
],
"tempo": "1-1-1-1",
"rest_time": 90,
"note": "",
"order": null,
"pro_dur": 390,
"_id": "63ca9b798d47745ae5589907",
"status": 0,
"pct_complete": 0,
"set": []
},
],
"isCustomName": false,
"status": 0,
"pct_complete": 0
},
"_id": "63ca95368d47745ae558985e"
},
{
"field_we": 1,
"field_da": 2,
"field_ti": 1,
"field_rest_we": false,
"field_rest_da": false,
"product": {
"name": "W121",
"comment": {
"en": ""
},
"pro_diff": [
{
"itemID": 2,
"description": "Intermediate"
}
],
"pro_int": [
{
"itemID": 2,
"description": "Moderate"
}
],
"pro_dur": 2674,
"nbr_exe": 6,
"target_mus": [
{
"itemID": 9,
"description": "Lats"
},
{
"itemID": 10,
"description": "Lower Back"
},
{
"itemID": 11,
"description": "Middle Back"
}
],
"target_are": [
{
"itemID": 3,
"description": "Back"
}
],
// group items by plan._id
"items": [
{
"itemid": 47,
"target_are": [
{
"itemID": 3,
"description": "Back"
}
],
"target_mus": [
{
"itemID": 10,
"description": "Lower Back"
}
],
"nbr_ref": 4,
"int_rec": "06~10",
"type_rec": [
{
"itemID": 14,
"description": "Straight Set"
}
],
"tempo": "1-1-1-1",
"rest_time": 90,
"note": "",
"order": null,
"pro_dur": 520,
"_id": "63caa1568d47745ae5589b25",
"status": 0,
"pct_complete": 0,
"set": []
},
{
"itemid": 495,
"target_are": [
{
"itemID": 3,
"description": "Back"
}
],
"target_mus": [
{
"itemID": 11,
"description": "Middle Back"
}
],
"nbr_ref": 3,
"int_rec": "10~12",
"type_rec": [
{
"itemID": 14,
"description": "Straight Set"
}
],
"tempo": "1-1-1-1",
"rest_time": 90,
"note": "",
"order": null,
"pro_dur": 414,
"_id": "63caa1568d47745ae5589b26",
"status": 0,
"pct_complete": 0,
"set": []
}
],
"isCustomName": false,
"status": 0,
"pct_complete": 0
},
"_id": "63ca9b8e8d47745ae558994b"
}
],
"_id": "63c7fec2fe9afea23afdbcf0"
},
"title": "Short Title text here",
"itemID": "bodyfyme882212-1674051266.214",
"__v": 32
}
Any help would be greatly appreciated

Merge object fields aggregation

I'm using MongoDB 3.6 and I have the following data:
[
{
"valoresVencimentos": [
1468.12,
85.9,
14.1,
1899.99,
241.92,
869.99,
696.11
],
"classesVencimentos": [
"lim_up_1ano",
"lim_1ano",
"a_venc_180d",
"a_venc_180d",
"outros",
"a_venc_180d",
"a_venc_180d"
],
"classesModalidades": [
"financiamento",
"financiamento",
"curto prazo",
"cartao",
"cartao",
"cartao",
"cartao"
]
},
{
"valoresVencimentos": [
627.29,
241.92,
413.47,
229.74,
1687.58,
100
],
"classesVencimentos": [
"a_venc_180d",
"outros",
"a_venc_180d",
"a_venc_180d",
"lim_up_1ano",
"lim_1ano"
],
"classesModalidades": [
"cartao",
"cartao",
"cartao",
"cartao",
"financiamento",
"financiamento"
]
},
{
"valoresVencimentos": [
268.59,
27.6,
428.51,
173.85,
2301.45,
100,
241.11
],
"classesVencimentos": [
"a_venc_180d",
"outros",
"a_venc_180d",
"a_venc_180d",
"lim_up_1ano",
"lim_1ano",
"a_venc_180d"
],
"classesModalidades": [
"cartao",
"cartao",
"cartao",
"cartao",
"financiamento",
"financiamento",
"curto prazo"
]
}
]
And I need merge fields with same name of each object in the input array, to be:
{
"valoresVencimentos": [
1468.12,
85.9,
14.1,
1899.99,
241.92,
869.99,
696.11,
627.29,
241.92,
413.47,
229.74,
1687.58,
100,
268.59,
27.6,
428.51,
173.85,
2301.45,
100,
241.11
],
"classesVencimentos": [
"lim_up_1ano",
"lim_1ano",
"a_venc_180d",
"a_venc_180d",
"outros",
"a_venc_180d",
"a_venc_180d",
"a_venc_180d",
"outros",
"a_venc_180d",
"a_venc_180d",
"lim_up_1ano",
"lim_1ano",
"a_venc_180d",
"outros",
"a_venc_180d",
"a_venc_180d",
"lim_up_1ano",
"lim_1ano",
"a_venc_180d"
],
"classesModalidades": [
"financiamento",
"financiamento",
"curto prazo",
"cartao",
"cartao",
"cartao",
"cartao",
"cartao",
"cartao",
"cartao",
"cartao",
"financiamento",
"financiamento",
"cartao",
"cartao",
"cartao",
"cartao",
"financiamento",
"financiamento",
"curto prazo"
]
}
Currently my aggregation is:
db.collection.aggregate([
{
$match: { code: '11122233344' }
},
{
$project: {
_id: 0,
valoresVencimentos: '$response.operations.expirations.value',
classesVencimentos: '$response.operations.expirations.class',
classesModalidades: '$response.operations.expirations.modality'
}
},
{
$addFields: {
valoresVencimentos: {
$reduce: {
input: '$valoresVencimentos',
initialValue: [],
in: { $concatArrays: ['$$this', '$$value'] }
}
},
classesVencimentos: {
$reduce: {
input: '$classesVencimentos',
initialValue: [],
in: { $concatArrays: ['$$this', '$$value'] }
}
},
classesModalidades: {
$reduce: {
input: '$classesModalidades',
initialValue: [],
in: { $concatArrays: ['$$this', '$$value'] }
}
},
}
},
])
It's imporant to say that ordering should be the same of the input data, so I think that using $group maybe a problem, I already tried a lot of alternatives but my knowledge with the aggregation framework is kind limited, appreciate any help.

Try this:
db.myCollection.aggregate([
{
$facet: {
valoresVencimentos: [
{ $unwind: "$valoresVencimentos" },
{
$group: {
_id: null,
valoresVencimentos: { $push: "$valoresVencimentos" }
}
}
],
classesVencimentos: [
{ $unwind: "$classesVencimentos" },
{
$group: {
_id: null,
classesVencimentos: { $push: "$classesVencimentos" }
}
}
],
classesModalidades: [
{ $unwind: "$classesModalidades" },
{
$group: {
_id: null,
classesModalidades: { $push: "$classesModalidades" }
}
}
]
}
},
{ $unwind: "$valoresVencimentos" },
{ $unwind: "$classesVencimentos" },
{ $unwind: "$classesModalidades" },
{
$project: {
valoresVencimentos: "$valoresVencimentos.valoresVencimentos",
classesVencimentos: "$classesVencimentos.classesVencimentos",
classesModalidades: "$classesModalidades.classesModalidades"
}
}
]);
or you can combine last three $unwinds and $project into single $project:
...,
{
$project: {
valoresVencimentos: { $arrayElemAt: ["$valoresVencimentos.valoresVencimentos", 0] },
classesVencimentos: { $arrayElemAt: ["$classesVencimentos.classesVencimentos", 0] },
classesModalidades: { $arrayElemAt: ["$classesModalidades.classesModalidades", 0] }
}
}
...

Expandable ListView / dynamic TreeView in Flutter

I need a nested TreeView where going through a list and build nodes based on parent child realationship. When tapping on a node it expands and shows its children with indentation. Also changing textstyle color of selectedt node..and 2d scrollable if the list is very long and has many nodes with children themselves having many children. How? please help.

You can copy paste run full code below
You can use package https://pub.dev/packages/flutter_treeview
It has some features you can customize, you can see working demo below
For parent child relation, you can see initState() in full code
#override
void initState() {
_nodes = [
Node(
label: 'documents',
key: 'docs',
expanded: docsOpen,
icon: NodeIcon(
codePoint:
docsOpen ? Icons.folder_open.codePoint : Icons.folder.codePoint,
color: "blue",
),
children: [
Node(
label: 'personal',
key: 'd3',
code snippet
TreeView(
controller: _treeViewController,
allowParentSelect: _allowParentSelect,
supportParentDoubleTap: _supportParentDoubleTap,
onExpansionChanged: (key, expanded) =>
_expandNode(key, expanded),
onNodeTap: (key) {
debugPrint('Selected: $key');
setState(() {
_selectedNode = key;
_treeViewController =
_treeViewController.copyWith(selectedKey: key);
});
},
theme: _treeViewTheme,
)
working demo
full code
import 'package:flutter/cupertino.dart';
import 'package:flutter/material.dart';
import 'package:flutter_treeview/tree_view.dart';
import 'dart:convert';
void main() => runApp(MyApp());
class MyApp extends StatelessWidget {
// This widget is the root of your application.
#override
Widget build(BuildContext context) {
return MaterialApp(
title: 'TreeView Example',
home: MyHomePage(title: 'TreeView Example'),
);
}
}
class MyHomePage extends StatefulWidget {
MyHomePage({Key key, this.title}) : super(key: key);
final String title;
#override
_MyHomePageState createState() => _MyHomePageState();
}
class _MyHomePageState extends State<MyHomePage> {
String _selectedNode;
List<Node> _nodes;
TreeViewController _treeViewController;
bool docsOpen = true;
final Map<ExpanderPosition, Widget> expansionPositionOptions = const {
ExpanderPosition.start: Text('Start'),
ExpanderPosition.end: Text('End'),
};
final Map<ExpanderType, Widget> expansionTypeOptions = const {
ExpanderType.caret: Icon(
Icons.arrow_drop_down,
size: 28,
),
ExpanderType.arrow: Icon(Icons.arrow_downward),
ExpanderType.chevron: Icon(Icons.expand_more),
ExpanderType.plusMinus: Icon(Icons.add),
};
final Map<ExpanderModifier, Widget> expansionModifierOptions = const {
ExpanderModifier.none: ModContainer(ExpanderModifier.none),
ExpanderModifier.circleFilled: ModContainer(ExpanderModifier.circleFilled),
ExpanderModifier.circleOutlined:
ModContainer(ExpanderModifier.circleOutlined),
ExpanderModifier.squareFilled: ModContainer(ExpanderModifier.squareFilled),
ExpanderModifier.squareOutlined:
ModContainer(ExpanderModifier.squareOutlined),
};
ExpanderPosition _expanderPosition = ExpanderPosition.start;
ExpanderType _expanderType = ExpanderType.caret;
ExpanderModifier _expanderModifier = ExpanderModifier.none;
bool _allowParentSelect = false;
bool _supportParentDoubleTap = false;
#override
void initState() {
_nodes = [
Node(
label: 'documents',
key: 'docs',
expanded: docsOpen,
icon: NodeIcon(
codePoint:
docsOpen ? Icons.folder_open.codePoint : Icons.folder.codePoint,
color: "blue",
),
children: [
Node(
label: 'personal',
key: 'd3',
icon: NodeIcon.fromIconData(Icons.input),
children: [
Node(
label: 'Resume.docx',
key: 'pd1',
icon: NodeIcon.fromIconData(Icons.insert_drive_file)),
Node(
label: 'Cover Letter.docx',
key: 'pd2',
icon: NodeIcon.fromIconData(Icons.insert_drive_file)),
]),
Node(
label: 'Inspection.docx',
key: 'd1',
// icon: NodeIcon.fromIconData(Icons.insert_drive_file),
),
Node(
label: 'Invoice.docx',
key: 'd2',
icon: NodeIcon.fromIconData(Icons.insert_drive_file)),
],
),
Node(
label: 'MeetingReport.xls',
key: 'mrxls',
icon: NodeIcon.fromIconData(Icons.insert_drive_file)),
Node(
label: 'MeetingReport.pdf',
key: 'mrpdf',
icon: NodeIcon.fromIconData(Icons.insert_drive_file)),
Node(
label: 'Demo.zip',
key: 'demo',
icon: NodeIcon.fromIconData(Icons.archive)),
];
_treeViewController = TreeViewController(
children: _nodes,
selectedKey: _selectedNode,
);
super.initState();
}
ListTile _makeExpanderPosition() {
return ListTile(
title: Text('Expander Position'),
dense: true,
trailing: CupertinoSlidingSegmentedControl(
children: expansionPositionOptions,
groupValue: _expanderPosition,
onValueChanged: (ExpanderPosition newValue) {
setState(() {
_expanderPosition = newValue;
});
},
),
);
}
SwitchListTile _makeAllowParentSelect() {
return SwitchListTile.adaptive(
title: Text('Allow Parent Select'),
dense: true,
value: _allowParentSelect,
onChanged: (v) {
setState(() {
_allowParentSelect = v;
});
},
);
}
SwitchListTile _makeSupportParentDoubleTap() {
return SwitchListTile.adaptive(
title: Text('Support Parent Double Tap'),
dense: true,
value: _supportParentDoubleTap,
onChanged: (v) {
setState(() {
_supportParentDoubleTap = v;
});
},
);
}
ListTile _makeExpanderType() {
return ListTile(
title: Text('Expander Style'),
dense: true,
trailing: CupertinoSlidingSegmentedControl(
children: expansionTypeOptions,
groupValue: _expanderType,
onValueChanged: (ExpanderType newValue) {
setState(() {
_expanderType = newValue;
});
},
),
);
}
ListTile _makeExpanderModifier() {
return ListTile(
title: Text('Expander Modifier'),
dense: true,
trailing: CupertinoSlidingSegmentedControl(
children: expansionModifierOptions,
groupValue: _expanderModifier,
onValueChanged: (ExpanderModifier newValue) {
setState(() {
_expanderModifier = newValue;
});
},
),
);
}
#override
Widget build(BuildContext context) {
TreeViewTheme _treeViewTheme = TreeViewTheme(
expanderTheme: ExpanderThemeData(
type: _expanderType,
modifier: _expanderModifier,
position: _expanderPosition,
color: Colors.grey.shade800,
size: 20,
),
labelStyle: TextStyle(
fontSize: 16,
letterSpacing: 0.3,
),
parentLabelStyle: TextStyle(
fontSize: 16,
letterSpacing: 0.1,
fontWeight: FontWeight.w800,
color: Colors.blue.shade700,
),
iconTheme: IconThemeData(
size: 18,
color: Colors.grey.shade800,
),
colorScheme: Theme.of(context).brightness == Brightness.light
? ColorScheme.light(
primary: Colors.blue.shade50,
onPrimary: Colors.grey.shade900,
background: Colors.transparent,
onBackground: Colors.black,
)
: ColorScheme.dark(
primary: Colors.black26,
onPrimary: Colors.white,
background: Colors.transparent,
onBackground: Colors.white70,
),
);
return Scaffold(
appBar: AppBar(
title: Text(widget.title),
elevation: 0,
),
body: GestureDetector(
onTap: () {
FocusScope.of(context).requestFocus(FocusNode());
},
child: Container(
color: Colors.grey.shade200,
padding: EdgeInsets.all(20),
height: double.infinity,
child: Column(
children: <Widget>[
Container(
height: 300,
child: Column(
children: <Widget>[
_makeExpanderPosition(),
_makeExpanderType(),
_makeExpanderModifier(),
_makeAllowParentSelect(),
_makeSupportParentDoubleTap(),
],
),
),
Expanded(
child: Container(
decoration: BoxDecoration(
color: Colors.white,
borderRadius: BorderRadius.circular(10),
),
padding: EdgeInsets.all(10),
child: TreeView(
controller: _treeViewController,
allowParentSelect: _allowParentSelect,
supportParentDoubleTap: _supportParentDoubleTap,
onExpansionChanged: (key, expanded) =>
_expandNode(key, expanded),
onNodeTap: (key) {
debugPrint('Selected: $key');
setState(() {
_selectedNode = key;
_treeViewController =
_treeViewController.copyWith(selectedKey: key);
});
},
theme: _treeViewTheme,
),
),
),
GestureDetector(
onTap: () {
debugPrint('Close Keyboard');
FocusScope.of(context).unfocus();
},
child: Container(
padding: EdgeInsets.only(top: 20),
alignment: Alignment.center,
child: Text(_treeViewController.getNode(_selectedNode) == null
? ''
: _treeViewController.getNode(_selectedNode).label),
),
)
],
),
),
),
bottomNavigationBar: SafeArea(
top: false,
child: ButtonBar(
alignment: MainAxisAlignment.spaceEvenly,
children: <Widget>[
CupertinoButton(
child: Text('Node'),
onPressed: () {
setState(() {
_treeViewController = _treeViewController.copyWith(
children: _nodes,
);
});
},
),
CupertinoButton(
child: Text('JSON'),
onPressed: () {
setState(() {
_treeViewController =
_treeViewController.loadJSON(json: US_STATES_JSON);
});
},
),
CupertinoButton(
child: Text('Toggle'),
onPressed: _treeViewController.selectedNode != null &&
_treeViewController.selectedNode.isParent
? () {
setState(() {
_treeViewController = _treeViewController
.withToggleNode(_treeViewController.selectedKey);
});
}
: null,
),
CupertinoButton(
child: Text('Edit'),
onPressed: () {
TextEditingController editingController = TextEditingController(
text: _treeViewController.selectedNode.label);
showCupertinoDialog(
context: context,
builder: (context) {
return CupertinoAlertDialog(
title: Text('Edit Label'),
content: Container(
height: 80,
alignment: Alignment.center,
padding: EdgeInsets.all(10),
child: CupertinoTextField(
controller: editingController,
autofocus: true,
),
),
actions: <Widget>[
CupertinoDialogAction(
child: Text('Cancel'),
isDestructiveAction: true,
onPressed: () => Navigator.of(context).pop(),
),
CupertinoDialogAction(
child: Text('Update'),
isDefaultAction: true,
onPressed: () {
if (editingController.text.isNotEmpty) {
setState(() {
Node _node = _treeViewController.selectedNode;
_treeViewController =
_treeViewController.withUpdateNode(
_treeViewController.selectedKey,
_node.copyWith(
label: editingController.text));
});
debugPrint(editingController.text);
}
Navigator.of(context).pop();
},
),
],
);
});
},
),
],
),
),
);
}
_expandNode(String key, bool expanded) {
String msg = '${expanded ? "Expanded" : "Collapsed"}: $key';
debugPrint(msg);
Node node = _treeViewController.getNode(key);
if (node != null) {
List<Node> updated;
if (key == 'docs') {
updated = _treeViewController.updateNode(
key,
node.copyWith(
expanded: expanded,
icon: NodeIcon(
codePoint: expanded
? Icons.folder_open.codePoint
: Icons.folder.codePoint,
color: expanded ? "blue600" : "grey700",
)),
);
} else {
updated = _treeViewController.updateNode(
key, node.copyWith(expanded: expanded));
}
setState(() {
if (key == 'docs') docsOpen = expanded;
_treeViewController = _treeViewController.copyWith(children: updated);
});
}
}
}
class ModContainer extends StatelessWidget {
final ExpanderModifier modifier;
const ModContainer(this.modifier, {Key key}) : super(key: key);
#override
Widget build(BuildContext context) {
double _borderWidth = 0;
BoxShape _shapeBorder = BoxShape.rectangle;
Color _backColor = Colors.transparent;
Color _backAltColor = Colors.grey.shade700;
switch (modifier) {
case ExpanderModifier.none:
break;
case ExpanderModifier.circleFilled:
_shapeBorder = BoxShape.circle;
_backColor = _backAltColor;
break;
case ExpanderModifier.circleOutlined:
_borderWidth = 1;
_shapeBorder = BoxShape.circle;
break;
case ExpanderModifier.squareFilled:
_backColor = _backAltColor;
break;
case ExpanderModifier.squareOutlined:
_borderWidth = 1;
break;
}
return Container(
decoration: BoxDecoration(
shape: _shapeBorder,
border: _borderWidth == 0
? null
: Border.all(
width: _borderWidth,
color: _backAltColor,
),
color: _backColor,
),
width: 15,
height: 15,
);
}
}
const List<Map<String, dynamic>> US_STATES = [
{
"label": "A",
"children": [
{"label": "Alabama", "key": "AL"},
{"label": "Alaska", "key": "AK"},
{"label": "American Samoa", "key": "AS"},
{"label": "Arizona", "key": "AZ"},
{"label": "Arkansas", "key": "AR"}
]
},
{
"label": "C",
"children": [
{"label": "California", "key": "CA"},
{"label": "Colorado", "key": "CO"},
{"label": "Connecticut", "key": "CT"},
]
},
{
"label": "D",
"children": [
{"label": "Delaware", "key": "DE"},
{"label": "District Of Columbia", "key": "DC"},
]
},
{
"label": "F",
"children": [
{"label": "Federated States Of Micronesia", "key": "FM"},
{"label": "Florida", "key": "FL"},
]
},
{
"label": "G",
"children": [
{"label": "Georgia", "key": "GA"},
{"label": "Guam", "key": "GU"},
]
},
{
"label": "H",
"children": [
{"label": "Hawaii", "key": "HI"},
]
},
{
"label": "I",
"children": [
{"label": "Idaho", "key": "ID"},
{"label": "Illinois", "key": "IL"},
{"label": "Indiana", "key": "IN"},
{"label": "Iowa", "key": "IA"},
]
},
{
"label": "K",
"children": [
{"label": "Kansas", "key": "KS"},
{"label": "Kentucky", "key": "KY"},
]
},
{
"label": "L",
"children": [
{"label": "Louisiana", "key": "LA"},
]
},
{
"label": "M",
"children": [
{"label": "Maine", "key": "ME"},
{"label": "Marshall Islands", "key": "MH"},
{"label": "Maryland", "key": "MD"},
{"label": "Massachusetts", "key": "MA"},
{"label": "Michigan", "key": "MI"},
{"label": "Minnesota", "key": "MN"},
{"label": "Mississippi", "key": "MS"},
{"label": "Missouri", "key": "MO"},
{"label": "Montana", "key": "MT"},
]
},
{
"label": "N",
"children": [
{"label": "Nebraska", "key": "NE"},
{"label": "Nevada", "key": "NV"},
{"label": "New Hampshire", "key": "NH"},
{"label": "New Jersey", "key": "NJ"},
{"label": "New Mexico", "key": "NM"},
{"label": "New York", "key": "NY"},
{"label": "North Carolina", "key": "NC"},
{"label": "North Dakota", "key": "ND"},
{"label": "Northern Mariana Islands", "key": "MP"},
]
},
{
"label": "O",
"children": [
{"label": "Ohio", "key": "OH"},
{"label": "Oklahoma", "key": "OK"},
{"label": "Oregon", "key": "OR"},
]
},
{
"label": "P",
"children": [
{"label": "Palau", "key": "PW"},
{"label": "Pennsylvania", "key": "PA"},
{"label": "Puerto Rico", "key": "PR"},
]
},
{
"label": "R",
"children": [
{"label": "Rhode Island", "key": "RI"},
]
},
{
"label": "S",
"children": [
{"label": "South Carolina", "key": "SC"},
{"label": "South Dakota", "key": "SD"},
]
},
{
"label": "T",
"children": [
{"label": "Tennessee", "key": "TN"},
{"label": "Texas", "key": "TX"},
]
},
{
"label": "U",
"children": [
{"label": "Utah", "key": "UT"},
]
},
{
"label": "V",
"children": [
{"label": "Vermont", "key": "VT"},
{"label": "Virgin Islands", "key": "VI"},
{"label": "Virginia", "key": "VA"},
]
},
{
"label": "W",
"children": [
{"label": "Washington", "key": "WA"},
{"label": "West Virginia", "key": "WV"},
{"label": "Wisconsin", "key": "WI"},
{"label": "Wyoming", "key": "WY"}
]
},
];
String US_STATES_JSON = jsonEncode(US_STATES);

Issues with geo indexes in mongodb

We have a collection of ~1M items that we query using the $nearSphere selector. It takes around between 3 seconds and 20 seconds to return 200 items.
From the explain plan of the request, we can see that is going through the same index 6 times.
Is it the expected behavior of mongodb query planner?
We would like to know if there is a way to force mongo to filter first by some field like endDate to reduce the set and then use $nearSphere?
On our monitoring system we can see some pagefault and assert but they might be related to the lack of IOPS of our hard drive.
Thank you for your help.
Here is the explan plan (I removed the rejected plans and troncates the BinData lines)
{
"queryPlanner": {
"plannerVersion": 1.0,
"namespace": "myCollection.Post",
"indexFilterSet": false,
"parsedQuery": {
"$and": [
{
"$or": [
{
"availableToUsers": {
"$eq": "M76zJCedq4"
}
},
{
"$nor": [
{
"availableToUsers": {
"$exists": true
}
}
]
}
]
},
{
"startDate": {
"$lt": ISODate(
"2019-03-01T01:02:00.000+0000"
)
}
},
{
"availableSubmitNumber": {
"$gt": 0.0
}
},
{
"endDate": {
"$gt": ISODate(
"2019-03-01T01:02:00.000+0000"
)
}
},
{
"name": {
"$in": ["Post1", "Post2"]
}
},
{
"$nor": [
{
"acceptedByUserId": {
"$eq": "M76zJCedq4"
}
}
]
},
{
"locationGeoPoint": {
"$nearSphere": [
174.9084055,
-36.9293289
]
}
}
]
},
"winningPlan": {
"stage": "FETCH",
"filter": {
"$and": [
{
"$or": [
{
"availableToUsers": {
"$eq": "M76zJCedq4"
}
},
{
"$nor": [
{
"availableToUsers": {
"$exists": true
}
}
]
}
]
},
{
"$nor": [
{
"acceptedByUserId": {
"$eq": "M76zJCedq4"
}
}
]
}
]
},
"inputStage": {
"stage": "GEO_NEAR_2D",
"keyPattern": {
"locationGeoPoint": "2d",
"endDate": 1.0,
"startDate": 1.0,
"availableSubmitNumber": 1.0,
"name": 1.0
},
"indexName": "locationGeoPoint_2d_endDate_1_startDate_1_availableSubmitNumber_1_name_1",
"indexVersion": 2.0,
"inputStages": [
{
"stage": "FETCH",
"inputStage": {
"stage": "IXSCAN",
"filter": {
"$and": [
{
"endDate": {
"$gt": ISODate(
"2019-03-01T01:02:00.000+0000"
)
}
},
{
"startDate": {
"$lt": ISODate(
"2019-03-01T01:02:00.000+0000"
)
}
},
{
"availableSubmitNumber": {
"$gt": 0.0
}
},
{
"name": {
"$in": ["Post1", "Post2"]
}
}
]
},
"keyPattern": {
"locationGeoPoint": "2d",
"endDate": 1.0,
"startDate": 1.0,
"availableSubmitNumber": 1.0,
"name": 1.0
},
"indexName": "locationGeoPoint_2d_endDate_1_startDate_1_availableSubmitNumber_1_name_1",
"isMultiKey": false,
"isUnique": false,
"isSparse": false,
"isPartial": false,
"indexVersion": 2.0,
"direction": "forward",
"indexBounds": {
"locationGeoPoint": [
"[BinData(128, BEB167B000000000), BinData(128, BEB167BFFFFFFFFF)]"
],
"endDate": [
"[MinKey, MaxKey]"
],
"startDate": [
"[MinKey, MaxKey]"
],
"availableSubmitNumber": [
"[MinKey, MaxKey]"
],
"name": [
"[MinKey, MaxKey]"
]
}
}
},
{
"stage": "FETCH",
"inputStage": {
"stage": "IXSCAN",
"filter": {
"$and": [
{
"endDate": {
"$gt": ISODate(
"2019-03-01T01:02:00.000+0000"
)
}
},
{
"startDate": {
"$lt": ISODate(
"2019-03-01T01:02:00.000+0000"
)
}
},
{
"availableSubmitNumber": {
"$gt": 0.0
}
},
{
"name": {
"$in": ["Post1", "Post2"]
}
}
]
},
"keyPattern": {
"locationGeoPoint": "2d",
"endDate": 1.0,
"startDate": 1.0,
"availableSubmitNumber": 1.0,
"name": 1.0
},
"indexName": "locationGeoPoint_2d_endDate_1_startDate_1_availableSubmitNumber_1_name_1",
"isMultiKey": false,
"isUnique": false,
"isSparse": false,
"isPartial": false,
"indexVersion": 2.0,
"direction": "forward",
"indexBounds": {
"locationGeoPoint": [
"[BinData(128, BEB1658000000000), BinData(128, BEB165BFFFFFFFFF)]",
"[BinData(128, BEB165C000000000), BinData(128, BEB165FFFFFFFFFF)]"
],
"endDate": [
"[MinKey, MaxKey]"
],
"startDate": [
"[MinKey, MaxKey]"
],
"availableSubmitNumber": [
"[MinKey, MaxKey]"
],
"name": [
"[MinKey, MaxKey]"
]
}
}
},
{
"stage": "FETCH",
"inputStage": {
"stage": "IXSCAN",
"filter": {
"$and": [
{
"endDate": {
"$gt": ISODate(
"2019-03-01T01:02:00.000+0000"
)
}
},
{
"startDate": {
"$lt": ISODate(
"2019-03-01T01:02:00.000+0000"
)
}
},
{
"availableSubmitNumber": {
"$gt": 0.0
}
},
{
"name": {
"$in": ["Post1", "Post2"]
}
}
]
},
"keyPattern": {
"locationGeoPoint": "2d",
"endDate": 1.0,
"startDate": 1.0,
"availableSubmitNumber": 1.0,
"name": 1.0
},
"indexName": "locationGeoPoint_2d_endDate_1_startDate_1_availableSubmitNumber_1_name_1",
"isMultiKey": false,
"isUnique": false,
"isSparse": false,
"isPartial": false,
"indexVersion": 2.0,
"direction": "forward",
"indexBounds": {
"locationGeoPoint": [
"[BinData(128, BEB14BC000000000), BinData(128, BEB14BFFFFFFFFFF)]",
"[BinData(128, BEB14C0000000000), BinData(128, BEB14FFFFFFFFFFF)]",
"[BinData(128, BEB1580000000000), BinData(128, BEB15BFFFFFFFFFF)]",
"[BinData(128, BEB1600000000000), BinData(128, BEB163FFFFFFFFFF)]",
"[BinData(128, BEB1640000000000), BinData(128, BEB164FFFFFFFFFF)]",
"[BinData(128, BEB1650000000000), BinData(128, BEB1653FFFFFFFFF)]",
"[BinData(128, BEB1654000000000), BinData(128, BEB1657FFFFFFFFF)]",
"[BinData(128, BEB1680000000000), BinData(128, BEB16BFFFFFFFFFF)]"
],
"endDate": [
"[MinKey, MaxKey]"
],
"startDate": [
"[MinKey, MaxKey]"
],
"availableSubmitNumber": [
"[MinKey, MaxKey]"
],
"name": [
"[MinKey, MaxKey]"
]
}
}
},
{
"stage": "FETCH",
"inputStage": {
"stage": "IXSCAN",
"filter": {
"$and": [
{
"endDate": {
"$gt": ISODate(
"2019-03-01T01:02:00.000+0000"
)
}
},
{
"startDate": {
"$lt": ISODate(
"2019-03-01T01:02:00.000+0000"
)
}
},
{
"availableSubmitNumber": {
"$gt": 0.0
}
},
{
"name": {
"$in": ["Post1", "Post2"]
}
}
]
},
"keyPattern": {
"locationGeoPoint": "2d",
"endDate": 1.0,
"startDate": 1.0,
"availableSubmitNumber": 1.0,
"name": 1.0
},
"indexName": "locationGeoPoint_2d_endDate_1_startDate_1_availableSubmitNumber_1_name_1",
"isMultiKey": false,
"isUnique": false,
"isSparse": false,
"isPartial": false,
"indexVersion": 2.0,
"direction": "forward",
"indexBounds": {
"locationGeoPoint": [
"[BinData(128, BE9BE00000000000), BinData(128, BE9BEFFFFFFFFFFF)]",
"[BinData(128, BE9BF80000000000), BinData(128, BE9BFBFFFFFFFFFF)]",
"[BinData(128, BEB1100000000000), BinData(128, BEB11FFFFFFFFFFF)]",
"[BinData(128, BEB1300000000000), BinData(128, BEB13FFFFFFFFFFF)]",
"[BinData(128, BEB1400000000000), BinData(128, BEB143FFFFFFFFFF)]"
],
"endDate": [
"[MinKey, MaxKey]"
],
"startDate": [
"[MinKey, MaxKey]"
],
"availableSubmitNumber": [
"[MinKey, MaxKey]"
],
"name": [
"[MinKey, MaxKey]"
]
}
}
},
{
"stage": "FETCH",
"inputStage": {
"stage": "IXSCAN",
"filter": {
"$and": [
{
"endDate": {
"$gt": ISODate(
"2019-03-01T01:02:00.000+0000"
)
}
},
{
"startDate": {
"$lt": ISODate(
"2019-03-01T01:02:00.000+0000"
)
}
},
{
"availableSubmitNumber": {
"$gt": 0.0
}
},
{
"name": {
"$in": ["Post1", "Post2"]
}
}
]
},
"keyPattern": {
"locationGeoPoint": "2d",
"endDate": 1.0,
"startDate": 1.0,
"availableSubmitNumber": 1.0,
"name": 1.0
},
"indexName": "locationGeoPoint_2d_endDate_1_startDate_1_availableSubmitNumber_1_name_1",
"isMultiKey": false,
"isUnique": false,
"isSparse": false,
"isPartial": false,
"indexVersion": 2.0,
"direction": "forward",
"indexBounds": {
"locationGeoPoint": [
"[BinData(128, BE9B800000000000), BinData(128, BE9BBFFFFFFFFFFF)]",
"[BinData(128, BE9BC00000000000), BinData(128, BE9BCFFFFFFFFFFF)]"
],
"endDate": [
"[MinKey, MaxKey]"
],
"startDate": [
"[MinKey, MaxKey]"
],
"availableSubmitNumber": [
"[MinKey, MaxKey]"
],
"name": [
"[MinKey, MaxKey]"
]
}
}
}
]
}
}
},
"serverInfo": {
"port": 27017.0,
"version": "4.0.3",
"gitVersion": "7ea530946fa7880364d88c8d8b6026bbc9ffa48c"
},
"ok": 1.0,
"operationTime": Timestamp(1551940718,
4),
"$clusterTime": {
"clusterTime": Timestamp(1551940718,
4),
"signature": {
"hash": BinData(0,
"AAAAAAAAAAAAAAAAAAAAAAAAAAA="
),
"keyId": NumberLong(0)
}
}
}

How to calculate YTD and MTD in mongodb?

How to calculate Month-To-Date(MTD) and Year-To-Date(YTD) in mongodb in a single query? sample data below, in this data requestedOn is a date field, I want to calculate MTD & YTD, on the assumption of financial year on "1st Jan of the year"(For example financial year for year 2016 is "01-Jan-2016" :
{
"_id": {
"$oid": "5808578b33fa6f161c9747f8"
},
"_class": "exceltest.TestBean",
"requestedOn": "2000-03-01",
"bookName": "Test6",
"revenue": 10.0,
"unitsSold": 1,
"bookCategory": [
{
"categoryCode": "Cooking/"
},
{
"categoryCode": "Cooking/Beverages"
},
{
"categoryCode": "Food Receipe/"
},
{
"categoryCode": "Food Receipe/Bartending"
},
{
"categoryCode": "Cooking/Beverages/Bartending"
},
{
"categoryCode": "Food Receipe/Taste"
}
]
}{
"_id": {
"$oid": "5808578b33fa6f161c9747f9"
},
"_class": "exceltest.TestBean",
"requestedOn": "2000-03-01",
"bookName": "Test1",
"revenue": 11.0,
"unitsSold": 2,
"bookCategory": [
{
"categoryCode": "Cooking/"
},
{
"categoryCode": "Cooking/Beverages"
},
{
"categoryCode": "Food Receipe/"
},
{
"categoryCode": "Food Receipe/Bartending"
},
{
"categoryCode": "Cooking/Beverages/Bartending"
},
{
"categoryCode": "Food Receipe/Taste"
}
]
}{
"_id": {
"$oid": "5808578b33fa6f161c9747fa"
},
"_class": "exceltest.TestBean",
"requestedOn": "2000-06-01",
"bookName": "Test2",
"revenue": 12.0,
"unitsSold": 3,
"bookCategory": [
{
"categoryCode": "Cooking/"
},
{
"categoryCode": "Cooking/Beverages"
},
{
"categoryCode": "Food Receipe/"
},
{
"categoryCode": "Food Receipe/Bartending"
},
{
"categoryCode": "Cooking/Beverages/Bartending"
},
{
"categoryCode": "Food Receipe/Taste"
}
]
}{
"_id": {
"$oid": "5808578b33fa6f161c9747fb"
},
"_class": "exceltest.TestBean",
"requestedOn": "2000-07-01",
"bookName": "Test3",
"revenue": 13.0,
"unitsSold": 4,
"bookCategory": [
{
"categoryCode": "Cooking/"
},
{
"categoryCode": "Cooking/Beverages"
},
{
"categoryCode": "Food Receipe/"
},
{
"categoryCode": "Food Receipe/Bartending"
},
{
"categoryCode": "Cooking/Beverages/Bartending"
},
{
"categoryCode": "Food Receipe/Taste"
}
]
}{
"_id": {
"$oid": "5808578b33fa6f161c9747fc"
},
"_class": "exceltest.TestBean",
"requestedOn": "2009-09-01",
"bookName": "Test4",
"revenue": 14.0,
"unitsSold": 5,
"bookCategory": [
{
"categoryCode": "Cooking/"
},
{
"categoryCode": "Cooking/Beverages"
},
{
"categoryCode": "Food Receipe/"
},
{
"categoryCode": "Food Receipe/Bartending"
},
{
"categoryCode": "Cooking/Beverages/Bartending"
},
{
"categoryCode": "Food Receipe/Taste"
}
]
}{
"_id": {
"$oid": "5808578b33fa6f161c9747fd"
},
"_class": "exceltest.TestBean",
"requestedOn": "2009-06-01",
"bookName": "Test5",
"revenue": 15.0,
"unitsSold": 6,
"bookCategory": [
{
"categoryCode": "Cooking/"
},
{
"categoryCode": "Cooking/Beverages"
},
{
"categoryCode": "Food Receipe/"
},
{
"categoryCode": "Food Receipe/Bartending"
},
{
"categoryCode": "Cooking/Beverages/Bartending"
},
{
"categoryCode": "Food Receipe/Taste"
}
]
}{
"_id": {
"$oid": "5808578b33fa6f161c9747fe"
},
"_class": "exceltest.TestBean",
"requestedOn": "2004-06-01",
"bookName": "Test10",
"revenue": 16.0,
"unitsSold": 7,
"bookCategory": [
{
"categoryCode": "Cooking/"
},
{
"categoryCode": "Cooking/Beverages"
},
{
"categoryCode": "Food Receipe/"
},
{
"categoryCode": "Food Receipe/Bartending"
},
{
"categoryCode": "Cooking/Beverages/Bartending"
},
{
"categoryCode": "Food Receipe/Taste"
}
]
}{
"_id": {
"$oid": "5808578b33fa6f161c9747ff"
},
"_class": "exceltest.TestBean",
"requestedOn": "2000-01-01",
"bookName": "Test11",
"revenue": 100.0,
"unitsSold": 100,
"bookCategory": [
{
"categoryCode": "Cooking/"
},
{
"categoryCode": "Cooking/Beverages"
},
{
"categoryCode": "Food Receipe/"
},
{
"categoryCode": "Food Receipe/Bartending"
},
{
"categoryCode": "Cooking/Beverages/Bartending"
},
{
"categoryCode": "Food Receipe/Taste"
}
]
}{
"_id": {
"$oid": "580857b833fa6f0c3499e462"
},
"_class": "exceltest.TestBean",
"requestedOn": "2000-02-01",
"bookName": "Test1",
"revenue": 20.0,
"unitsSold": 10,
"bookCategory": [
{
"categoryCode": "Cooking/"
},
{
"categoryCode": "Cooking/Beverages"
},
{
"categoryCode": "Food Receipe/"
},
{
"categoryCode": "Food Receipe/Bartending"
}
]
}{
"_id": {
"$oid": "580857b833fa6f0c3499e463"
},
"_class": "exceltest.TestBean",
"requestedOn": "2001-02-01",
"bookName": "Test2",
"revenue": 19.0,
"unitsSold": 9,
"bookCategory": [
{
"categoryCode": "Cooking/"
},
{
"categoryCode": "Cooking/Beverages"
},
{
"categoryCode": "Food Receipe/"
},
{
"categoryCode": "Food Receipe/Bartending"
}
]
}{
"_id": {
"$oid": "580857b833fa6f0c3499e464"
},
"_class": "exceltest.TestBean",
"requestedOn": "2001-02-01",
"bookName": "Test3",
"revenue": 18.0,
"unitsSold": 8,
"bookCategory": [
{
"categoryCode": "Cooking/"
},
{
"categoryCode": "Cooking/Beverages"
},
{
"categoryCode": "Food Receipe/"
},
{
"categoryCode": "Food Receipe/Bartending"
}
]
}{
"_id": {
"$oid": "580857b833fa6f0c3499e465"
},
"_class": "exceltest.TestBean",
"requestedOn": "2007-06-01",
"bookName": "Test4",
"revenue": 17.0,
"unitsSold": 7,
"bookCategory": [
{
"categoryCode": "Cooking/"
},
{
"categoryCode": "Cooking/Beverages"
},
{
"categoryCode": "Food Receipe/"
},
{
"categoryCode": "Food Receipe/Bartending"
}
]
}{
"_id": {
"$oid": "580857b833fa6f0c3499e466"
},
"_class": "exceltest.TestBean",
"requestedOn": "2005-06-01",
"bookName": "Test5",
"revenue": 16.0,
"unitsSold": 6,
"bookCategory": [
{
"categoryCode": "Cooking/"
},
{
"categoryCode": "Cooking/Beverages"
},
{
"categoryCode": "Food Receipe/"
},
{
"categoryCode": "Food Receipe/Bartending"
}
]
}{
"_id": {
"$oid": "580857b833fa6f0c3499e467"
},
"_class": "exceltest.TestBean",
"requestedOn": "2004-06-01",
"bookName": "Test1",
"revenue": 15.0,
"unitsSold": 5,
"bookCategory": [
{
"categoryCode": "Cooking/"
},
{
"categoryCode": "Cooking/Beverages"
},
{
"categoryCode": "Food Receipe/"
},
{
"categoryCode": "Food Receipe/Bartending"
}
]
}{
"_id": {
"$oid": "580857b833fa6f0c3499e468"
},
"_class": "exceltest.TestBean",
"requestedOn": "2002-06-01",
"bookName": "Test2",
"revenue": 14.0,
"unitsSold": 4,
"bookCategory": [
{
"categoryCode": "Cooking/"
},
{
"categoryCode": "Cooking/Beverages"
},
{
"categoryCode": "Food Receipe/"
},
{
"categoryCode": "Food Receipe/Bartending"
}
]
}{
"_id": {
"$oid": "580857b833fa6f0c3499e469"
},
"_class": "exceltest.TestBean",
"requestedOn": "2001-06-01",
"bookName": "Test3",
"revenue": 13.0,
"unitsSold": 3,
"bookCategory": [
{
"categoryCode": "Cooking/"
},
{
"categoryCode": "Cooking/Beverages"
},
{
"categoryCode": "Food Receipe/"
},
{
"categoryCode": "Food Receipe/Bartending"
}
]
}{
"_id": {
"$oid": "580857b833fa6f0c3499e46a"
},
"_class": "exceltest.TestBean",
"requestedOn": "2000-06-01",
"bookName": "Test4",
"revenue": 12.0,
"unitsSold": 2,
"bookCategory": [
{
"categoryCode": "Cooking/"
},
{
"categoryCode": "Cooking/Beverages"
},
{
"categoryCode": "Food Receipe/"
},
{
"categoryCode": "Food Receipe/Bartending"
}
]
}{
"_id": {
"$oid": "580857b833fa6f0c3499e46b"
},
"_class": "exceltest.TestBean",
"requestedOn": "2008-06-01",
"bookName": "Test5",
"revenue": 11.0,
"unitsSold": 1,
"bookCategory": [
{
"categoryCode": "Cooking/"
},
{
"categoryCode": "Cooking/Beverages"
},
{
"categoryCode": "Food Receipe/"
},
{
"categoryCode": "Food Receipe/Bartending"
}
]
}
Regards
Kris

It is a good practice to keep dates in MongoDB in its native dateformat ISODate().
You can use date formats like $year,$month,$day,$hour etc.
These can be used for grouping , in your case:
db.collectionName.aggregate([
{$group:{_id:{'Date':{$year:'$requestedOn'}},total:{$sum:'$FieldName'}}}
])
to convert string to ISODate , answers can be found at
- [http://stackoverflow.com/questions/15473772/how-to-convert-from-string-to-date-data-type?noredirect=1&lq=1][2]
- [http://stackoverflow.com/questions/15473772/how-to-convert-from-string-to-date-data-type?noredirect=1&lq=1][2]

We Keep Coding

iphone swift flutter scala powershell matlab mongodb postgresql perl eclipse

How to modify the execution plan in Spark? - scala

Sure, you can parse and modify any JSON object in memory, but that has nothing to do with Spark. Related: What JSON library to use in Scala? Any modifications you make wouldn't be persisted within the execution plan itself.

Related

MongoDB aggregation re-group deep nested array of objects

Merge object fields aggregation

Expandable ListView / dynamic TreeView in Flutter

Issues with geo indexes in mongodb

How to calculate YTD and MTD in mongodb?

Categories

Resources