Search Mongo Document based on BSON DateTime - mongodb

My mongo document looks like below -
{
"_id" : ObjectId("5977dab66ea64e345288ceec"),
"transactionid" : "1",
"servicename" : "Reservation Service",
"starttime" : {
"dateTime" : true,
"symbol" : false,
"javaScriptWithScope" : false,
"string" : false,
"regularExpression" : false,
"javaScript" : false,
"double" : false,
"document" : false,
"DBPointer" : false,
"number" : false,
"boolean" : false,
"null" : false,
"bsonType" : "DATE_TIME",
"array" : false,
"int32" : false,
"int64" : false,
"binary" : false,
"value" : NumberLong(1501026996822),
"decimal128" : false,
"objectId" : false,
"timestamp" : false
},
"endtime" : {
"dateTime" : true,
"symbol" : false,
"javaScriptWithScope" : false,
"string" : false,
"regularExpression" : false,
"javaScript" : false,
"double" : false,
"document" : false,
"DBPointer" : false,
"number" : false,
"boolean" : false,
"null" : false,
"bsonType" : "DATE_TIME",
"array" : false,
"int32" : false,
"int64" : false,
"binary" : false,
"value" : NumberLong(1501026997875),
"decimal128" : false,
"objectId" : false,
"timestamp" : false
},
"status" : "Success",
"updatedby" : "muf",
"requestpayload" : "{\n \"service_header\": {\n \t\n \n \t\"service_name\": \"Reservation Service\",\n \t\"source_id\": \"Amex\",\n \t \t\"document_id\":\"1\",\n \t\"transaction_id\":\"1\"\n },\n \t\n \t\"airline\":\"united\",\n\t\"destination\":\"LAX\"\n\n\n}",
"responsepayload" : "[{\"departureDate\":\"2015/02/11\",\"airlineName\":\"United\",\"destination\":\"LAX\",\"price\":345.99,\"planeType\":\"Boeing 737\",\"code\":\"ER45if\",\"origin\":\"MUA\",\"emptySeats\":52},{\"departureDate\":\"2015/04/11\",\"airlineName\":\"United\",\"destination\":\"LAX\",\"price\":346.0,\"planeType\":\"Boeing 777\",\"code\":\"ER45jd\",\"origin\":\"MUA\",\"emptySeats\":12},{\"departureDate\":\"2015/06/11\",\"airlineName\":\"United\",\"destination\":\"LAX\",\"price\":423.0,\"planeType\":\"Boeing 707\",\"code\":\"ER0945\",\"origin\":\"MUA\",\"emptySeats\":0}]",
"exception" : "",
"requesthost" : "localhost",
"requestport" : "8086",
"requestpath" : "/flights"
}
I am trying to query below document using below query -
db.getCollection('Transaction').find({ starttime: { "$gt" : { "$date" : (new ISODate("2018-07-25T23:56:36.822Z")*1) } } })
I do get the result. However when I try to use less than and greater than together I don't get any result. I am not sure what is best way to query above document.
db.getCollection('Transaction').find({ starttime: { "$gt" : { "$date" : (new ISODate("2015-07-25T23:56:36.822Z")*1) } , "$lt" : { "$date" : (new ISODate("2018-07-25T23:56:36.822Z")*1) } } })

Related

MongoDB performing COLLSCAN even with an index specified in the compound index

I have a database with the following compound index NameHandleCreationTime
"metadata" : {
"formatVersion" : 8,
"infoObj" : "{ \"v\" : 2, \"key\" : { \"Name\" : 1, \"Handle\" : 1, \"CreationTime\" : -1 }, \"name\" : \"NameHandleCreationTime\", \"ns\" : \"intenv.UserCollection_test\", \"collation\" : { \"locale\" : \"en\", \"caseLevel\" : false, \"caseFirst\" : \"off\", \"strength\" : 1, \"numericOrdering\" : false, \"alternate\" : \"non-ignorable\", \"maxVariable\" : \"punct\", \"normalization\" : false, \"backwards\" : false, \"version\" : \"57.1\" } }"
},
...
So I thought if I make a query with Name: xyz to get a single document from the UserCollection, I'd get it through an IXSCAN, because the compound index's first field is Name. However, the logs show that I am COLLSCANing and even winning plan is a COLLSCAN.
//db.getCollection("intenv.UserCollection_test").find({Name:"xyz"}).collation({locale:"en"}).explain()
{
"queryPlanner" : {
"plannerVersion" : 1,
"namespace" : "intenv.UserCollection_test",
"indexFilterSet" : false,
"parsedQuery" : {
"Name" : {
"$eq" : "xyz"
}
},
"collation" : {
"locale" : "en",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : 3,
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
},
"winningPlan" : {
"stage" : "COLLSCAN",
"filter" : {
"Name" : {
"$eq" : "xyz"
}
},
"direction" : "forward"
},
"rejectedPlans" : [ ]
},
...
What is going on...?
Also, I toyed around with collation (https://docs.mongodb.com/manual/reference/collation/) while writing this question: setting collation({ locale: "en", strength: 1}) gives me an IXSCAN as the winning plan...
Why is this?
The collation requested in the query is different than the collation the index was created with, so it can't be used for that query.

Mongodb Searching sharding key is very slow

I set up a mongodb with 3 shards by using bitnami's chart: mongodb-sharded
and I got two problems.
My sharding collection has more than 38 millions records.
The first problem is some mongodb pods take a lot of memory, more than 10G from the whole memory of 32G in a node. I found some similar issues from stackoverflow and I would try to tackle this problem.
The other problem is: when I do a search which matches the sharding key, it's very slow and sometimes it will be time out.
Here is my collection's indexes:
ice-shard-mongodb-sharded-shard-0:PRIMARY> db.LogInfo.getIndexes()
[
{
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : 2,
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
}
},
{
"v" : 2,
"key" : {
"Properties.Time" : 1
},
"name" : "Properties.Time_1",
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : 2,
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
}
},
{
"v" : 2,
"key" : {
"Properties.Plot" : 1,
"Properties.Time" : 1
},
"name" : "Properties.Plot_1_Properties.Time_1",
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : 2,
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
}
},
{
"v" : 2,
"key" : {
"Properties.Scenario" : 1,
"Properties.Time" : 1
},
"name" : "Properties.Scenario_1_Properties.Time_1",
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : 2,
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
}
},
{
"v" : 2,
"key" : {
"Properties.SecsFunction" : 1,
"Properties.Time" : 1
},
"name" : "Properties.SecsFunction_1_Properties.Time_1",
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : 2,
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
}
},
{
"v" : 2,
"key" : {
"Properties.EapId" : "hashed",
"Properties.Time" : 1
},
"name" : "Properties.EapId_hashed_Properties.Time_1"
}
]
The key "Properties.EapId_hashed_Properties.Time_1" is the sharding key.
If I search
db.LogInfo.find({"Properties.SecsFunction": "s6f11", "Properties.Time": {$gte: ISODate("2021-03-10")}})
This search matches an index and it will search all the 3 shards. It's very fast within 0.01 seconds.
But if I search
db.LogInfo.find(
{
"Properties.EapId": "12eap012",
"Properties.Time": {
"$gte": ISODate("2021-03-17")
}
}
)
which should use the sharding key, I think, it's very flow. Sometimes it will take more than 5 seconds and sometimes it will be time out. I've checked the indexes several times but I can't figure out why.
Please advise. Thanks.
update: Mongodb version: 4.4.3
update: shards are built as replica set = 3.
update: below is the query plan for the problematic search. (I drop the executionStat() due to stackoverflow's 30000 characters limit.)
{
"queryPlanner" : {
"mongosPlannerVersion" : NumberInt(1),
"winningPlan" : {
"stage" : "SHARD_MERGE",
"shards" : [
{
"shardName" : "ice-shard-mongodb-sharded-shard-1",
"connectionString" : "ice-shard-mongodb-sharded-shard-1/ice-shard-mongodb-sharded-shard1-data-0.ice-shard-mongodb-sharded-headless.ice-system.svc.cluster.local:27017,ice-shard-mongodb-sharded-shard1-data-1.ice-shard-mongodb-sharded-headless.ice-system.svc.cluster.local:27017,ice-shard-mongodb-sharded-shard1-data-2.ice-shard-mongodb-sharded-headless.ice-system.svc.cluster.local:27017",
"serverInfo" : {
"host" : "ice-shard-mongodb-sharded-shard1-data-1",
"port" : NumberInt(27017),
"version" : "4.4.3",
"gitVersion" : "913d6b62acfbb344dde1b116f4161360acd8fd13"
},
"plannerVersion" : NumberInt(1),
"namespace" : "LogCenter.LogInfo",
"indexFilterSet" : false,
"parsedQuery" : {
"$and" : [
{
"Properties.EapId" : {
"$eq" : "12eap012"
}
},
{
"Properties.Time" : {
"$gte" : ISODate("2021-03-17T00:00:00.000+0000")
}
}
]
},
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : NumberInt(2),
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
},
"winningPlan" : {
"stage" : "SHARDING_FILTER",
"inputStage" : {
"stage" : "FETCH",
"filter" : {
"Properties.EapId" : {
"$eq" : "12eap012"
}
},
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"Properties.Time" : 1.0
},
"indexName" : "Properties.Time_1",
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : NumberInt(2),
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
},
"isMultiKey" : false,
"multiKeyPaths" : {
"Properties.Time" : [
]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : NumberInt(2),
"direction" : "forward",
"indexBounds" : {
"Properties.Time" : [
"[new Date(1615939200000), new Date(9223372036854775807)]"
]
}
}
}
},
"rejectedPlans" : [
]
},
{
"shardName" : "ice-shard-mongodb-sharded-shard-0",
"connectionString" : "ice-shard-mongodb-sharded-shard-0/ice-shard-mongodb-sharded-shard0-data-0.ice-shard-mongodb-sharded-headless.ice-system.svc.cluster.local:27017,ice-shard-mongodb-sharded-shard0-data-1.ice-shard-mongodb-sharded-headless.ice-system.svc.cluster.local:27017,ice-shard-mongodb-sharded-shard0-data-2.ice-shard-mongodb-sharded-headless.ice-system.svc.cluster.local:27017",
"serverInfo" : {
"host" : "ice-shard-mongodb-sharded-shard0-data-0",
"port" : NumberInt(27017),
"version" : "4.4.3",
"gitVersion" : "913d6b62acfbb344dde1b116f4161360acd8fd13"
},
"plannerVersion" : NumberInt(1),
"namespace" : "LogCenter.LogInfo",
"indexFilterSet" : false,
"parsedQuery" : {
"$and" : [
{
"Properties.EapId" : {
"$eq" : "12eap012"
}
},
{
"Properties.Time" : {
"$gte" : ISODate("2021-03-17T00:00:00.000+0000")
}
}
]
},
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : NumberInt(2),
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
},
"winningPlan" : {
"stage" : "SHARDING_FILTER",
"inputStage" : {
"stage" : "FETCH",
"filter" : {
"Properties.EapId" : {
"$eq" : "12eap012"
}
},
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"Properties.Time" : 1.0
},
"indexName" : "Properties.Time_1",
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : NumberInt(2),
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
},
"isMultiKey" : false,
"multiKeyPaths" : {
"Properties.Time" : [
]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : NumberInt(2),
"direction" : "forward",
"indexBounds" : {
"Properties.Time" : [
"[new Date(1615939200000), new Date(9223372036854775807)]"
]
}
}
}
},
"rejectedPlans" : [
]
},
{
"shardName" : "ice-shard-mongodb-sharded-shard-2",
"connectionString" : "ice-shard-mongodb-sharded-shard-2/ice-shard-mongodb-sharded-shard2-data-0.ice-shard-mongodb-sharded-headless.ice-system.svc.cluster.local:27017,ice-shard-mongodb-sharded-shard2-data-1.ice-shard-mongodb-sharded-headless.ice-system.svc.cluster.local:27017,ice-shard-mongodb-sharded-shard2-data-2.ice-shard-mongodb-sharded-headless.ice-system.svc.cluster.local:27017",
"serverInfo" : {
"host" : "ice-shard-mongodb-sharded-shard2-data-0",
"port" : NumberInt(27017),
"version" : "4.4.3",
"gitVersion" : "913d6b62acfbb344dde1b116f4161360acd8fd13"
},
"plannerVersion" : NumberInt(1),
"namespace" : "LogCenter.LogInfo",
"indexFilterSet" : false,
"parsedQuery" : {
"$and" : [
{
"Properties.EapId" : {
"$eq" : "12eap012"
}
},
{
"Properties.Time" : {
"$gte" : ISODate("2021-03-17T00:00:00.000+0000")
}
}
]
},
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : NumberInt(2),
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
},
"winningPlan" : {
"stage" : "SHARDING_FILTER",
"inputStage" : {
"stage" : "FETCH",
"filter" : {
"Properties.EapId" : {
"$eq" : "12eap012"
}
},
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"Properties.Time" : 1.0
},
"indexName" : "Properties.Time_1",
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : NumberInt(2),
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
},
"isMultiKey" : false,
"multiKeyPaths" : {
"Properties.Time" : [
]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : NumberInt(2),
"direction" : "forward",
"indexBounds" : {
"Properties.Time" : [
"[new Date(1615939200000), new Date(9223372036854775807)]"
]
}
}
}
},
"rejectedPlans" : [
]
}
]
}
},
"serverInfo" : {
"host" : "ice-shard-mongodb-sharded-mongos-59576fb8b8-n47n5",
"port" : NumberInt(27017),
"version" : "4.4.3",
"gitVersion" : "913d6b62acfbb344dde1b116f4161360acd8fd13"
},
"ok" : 1.0,
"operationTime" : Timestamp(1616372015, 12),
"$clusterTime" : {
"clusterTime" : Timestamp(1616372015, 14),
"signature" : {
"hash" : BinData(0, "vVxiZGic6un9QIInadoHCfBCX2Y="),
"keyId" : NumberLong(6924833040434724866)
}
}
}
I found the root cause why my search didn't use the sharding key. It's all about the collation.
Sharding key always use collation : { locale : "simple" } to do a binary comparison, while in my case my collection and indexes are designed to use "collation" : { "locale" : "en_US", "strength" : 2 }.
After redesigning my collection without any collation, now everything is fine.

MongoDB Redash - FieldPath field names may not contain '.'

I am new to mongodb. I am trying to take the daily count. I want count of those element where the fields I_C or L2_C exist and for the last 3 months.
I have written this code, but it is showing error -
"FieldPath field names may not contain '.'".
What would be the correct way to go for this?
Here is the schema of the data I am trying to query on.
{
"_id" : ObjectId("5b83af839bc195a7cfbabefb"),
"firstName" : null,
"lastName" : null,
"realm" : null,
"username" : null,
"password" : "$2a$10$WVTWLevvoJXcMs/x9o/qMOEV56owh2ppjZXJt4pSR8PIDuBeRehwW",
"credentials" : null,
"challenges" : null,
"email" : "ritesh.rnagpure#gmail.com",
"emailVerified" : null,
"verificationToken" : null,
"status" : null,
"created" : null,
"lastUpdated" : ISODate("2019-08-28T12:15:07.808Z"),
"dateCreated" : ISODate("2018-08-27T08:00:02.595Z"),
"source" : "web",
"isFromAPUrl" : false,
"personal" : {
"phone" : "9975891779",
"referralcode" : "",
"iccode" : "221930",
"isiccode" : "yes",
"referrerEmail" : "zohaibrulz#gmail.com",
"referrerName" : "Momin Zohaib Hasanali",
"residency" : {
"type" : "Indian",
"status" : "Status_Resident_Individual",
"nationality" : "Nationality_Indian",
"nationalityOther" : ""
},
"isNameFromIT" : true,
"name" : "RITESH RAJENDRA NAGPURE",
"birthdate" : {
"date" : "7",
"month" : "4",
"year" : "1993"
},
"previousPan" : "AJEPN0774B",
"previousDOB" : "07/04/1993",
"address" : {
"correspondence" : {
"line1" : "Flat No-404, G Wing, Sai Satyam Residency,",
"line2" : "Near Don Bosco School, Kalyan West, Opposite Raunak city, Kalyan, Thane",
"city" : "Kalyan",
"state" : "Maharashtra",
"pincode" : "421301",
"proof" : ""
},
"permanent" : {
"line1" : "Flat No-404, G Wing, Sai Satyam Residency,",
"line2" : "Near Don Bosco School, Kalyan West, Opposite Raunak city, Kalyan, Thane",
"city" : "Kalyan",
"state" : "Maharashtra",
"pincode" : "421301",
"proof" : "",
"proofOtherText" : ""
},
"sameAsCorrespondence" : true
},
"isaddressFromQR" : false,
"isaadhar" : false,
"iskra" : false,
"fatherOrSpouseName" : "RAJENDRA",
"ipvdetails" : {
"ipvrequire" : false,
"ipvstatus" : "done"
},
"aadharNumber" : "875847348082",
"mothername" : "SAVITA",
"gender" : "male",
"polexposed" : "no",
"maritalStatus" : "single",
"incomeType" : "annual",
"income" : "below_1L",
"occupation" : "Private Sector",
"tradingExperience" : "< 1 Year",
"promocode" : "CASHBACK",
"requireddoc" : {
"ispanrequire" : true,
"isuidrequire" : true,
"ischequerequire" : true,
"isphotorequire" : true,
"iscoraddressrequire" : false,
"issignaturerequire" : true,
"isaddressrequire" : false,
"isbankstatementrequire" : true,
"isoptionaldoc" : false
},
"uploadeddoc" : {
"ispan" : true,
"isuid" : true,
"isaddress" : false,
"isphoto" : true,
"iscoraddress" : false,
"ischeque" : true,
"issignature" : true,
"isoptionaldoc" : false,
"isbankstatement" : true,
"issignaturebmp" : true
},
"docname" : {
"uid" : "5b83af839bc195a7cfbabefb_uid.pdf",
"pan" : "5b83af839bc195a7cfbabefb_pan.jpg",
"photo" : "5b83af839bc195a7cfbabefb_photo.jpg",
"address" : "",
"coraddress" : "",
"cheque" : "5b83af839bc195a7cfbabefb_cheque.pdf",
"bankstatement" : "5b83af839bc195a7cfbabefb_bankstm.pdf",
"signature" : "5b83af839bc195a7cfbabefb_sign.jpg",
"other" : "",
"optionaldoc" : ""
},
"doctype" : {
"pan" : "pan",
"uid" : "uid",
"add" : "uid",
"cheque" : "bankstatement",
"bankstm" : "bankstatement",
"photo" : "photo"
},
"isaddressmodified" : true,
"isAllEsigned" : true,
"isEsignMerged" : true,
"iskycFile" : true,
"isKycEsigned" : true,
"isApplicationCompleted" : true,
"applicationStatus" : "completed",
"isPanVerified" : true,
"panStatus" : "E",
"panName" : "RITESH RAJENDRA NAGPURE",
"rejectionReasonList" : [],
"makerAssigned" : false,
"checkerAssigned" : true,
"checkerOfficer" : "robot",
"checkerTime" : ISODate("2019-05-29T11:30:33.004Z")
},
"txn" : {
"L1_L2" : ISODate("2018-08-27T11:06:35.251Z"),
"L1_L2_officer" : "prajakta.rane",
"L2_R" : ISODate("2018-08-27T10:43:19.063Z"),
"L2_R_officer" : "avilon.pereira",
"R_L1" : ISODate("2018-08-27T10:48:51.377Z"),
"L2_I" : ISODate("2018-08-27T11:43:12.718Z"),
"L2_I_officer" : "rajkumar.kesari",
"I_C" : ISODate("2018-08-27T12:58:52.343Z"),
"I_C_officer" : "suraj.pandey"
},
"sales" : {
"openStateOfficer" : "bhaskar.geera",
"aadhaar" : false,
"assignStep" : "registration",
"assignTo" : "bhaskar.geera",
"callTime" : "2018-08-27T11:35:15.446Z",
"calledMessage" : "Ringing",
"calledStatus" : "Ringing_Not_Picked",
"iscalled" : true,
"callCount" : 5
},
"utm_params" : {},
"landingPage" : "",
"suiteID" : "b9245931-a66f-3647-11b8-5b83af881208",
"_scopeMeta" : {},
"panPageVisited" : true,
"kra" : {
"pan" : "AJEPN0774B",
"dob" : "07/04/1993",
"name" : "",
"fatherOrSpouseName" : "",
"agency" : "",
"email" : "",
"mobile" : "",
"address" : {
"correspondence" : {
"line1" : "",
"line2" : "",
"city" : "",
"state" : "",
"pincode" : "",
"proof" : ""
},
"permanent" : {
"line1" : "",
"line2" : "",
"city" : "",
"state" : "",
"pincode" : "",
"proof" : ""
}
},
"gender" : "",
"maritalStatus" : "",
"appStatus" : "005",
"appUpdateStatus" : "005",
"authMode" : "0"
},
"aadhar" : {},
"overlayOpenData" : {
"showSegmentsOverlay" : false,
"showBrokerageOverlay" : false,
"showReviewOverlay" : true,
"showVerifyOverlay" : true
},
"currentStep" : "review/verify",
"namePageVisited" : true,
"options" : {
"BSE_EQ" : true,
"NSE_EQ" : true,
"BCD_FO" : false,
"NCD_FO" : true,
"NSE_FO" : true,
"MCX" : true,
"MF" : true
},
"pricingPlan" : "BASIC-D0",
"bank" : {
"primary" : {
"ifsc" : "SBIN0017460",
"name" : "STATE BANK OF INDIA",
"address" : "MADHUBAN TIRTHDHAM COMPLEX,SHOP NO 4,ADHARWADI ROAD,KALYAN WEST,DISTT.THANE.MAHARASHTRA 421301",
"branch" : "ADHARWADI ROAD BRANCH",
"city" : "THANE",
"state" : "MAHARASHTRA",
"country" : "INDIA",
"micr" : "",
"pincode" : "421301",
"type" : "savings",
"account" : "31527410599"
}
},
"kycSplitStart" : 63,
"kycDocsLength" : 4,
"eSignProperties" : {
"signername" : "Ritesh Rajendra Nagpure"
},
"formSubmittedDate" : ISODate("2018-08-27T10:48:51.377Z"),
"ucccode" : "177762",
"crmPaymentSynced" : true,
"passwordResetCode" : "",
"lastPaymentDate" : ISODate("2018-10-26T09:55:24.149Z"),
"registrationWorkflowId" : 0,
"currentWorkflowId" : 0,
"isPaymentFromLedger" : false,
"lastLoginDate" : ISODate("2019-08-28T12:14:58.569Z"),
"viewedPreSales" : true
}
I have written this query -
"collection": "user",
"aggregate": [
{
"$match": {
"$or" : [
{
"$and" : [
{"txn.I_C" : {"$exists": true}}, {"txn.I_C" : {"$gt" : {"$humanTime" :"2019-07-01"}}}]
},
{
"$and" : [
{"txn.L2_C" : {"$exists": true}}, {"txn.L2_C" : {"$gt" : {"$humanTime" :"2019-07-01"}}}]
}
]}
},
{ "$addFields" : {"date" :{"$cond": { "if": {"txn.I_C" : {"$exists" : true}}, "then": "txn.I_C", "else": "txn.L2_C"}}}},
{
"$project" : {
"_id" : 0,
"datePartDay": {
"$concat": [
{
"$substr": [
{
"$year": "$date"
},
0,
4
]
},
"-",
{
"$substr": [
{
"$month": "$date"
},
0,
2
]
},
"-",
{
"$substr": [
{
"$dayOfMonth": "$date"
},
0,
2
]
}
]
}
}
},
{
"$group": {
"_id": "$datePartDay",
"count": {
"$sum": 1
}
}
}
]
}
Try this way, please:
"collection": "user",
"aggregate": [
{
"$match": {
"$or" : [
{
"$and" : [
{"txn.I_C" : {"$exists": true}}, {"txn.I_C" : {"$gt" : {"$humanTime" :"2019-07-01"}}}]
},
{
"$and" : [
{"txn.L2_C" : {"$exists": true}}, {"txn.L2_C" : {"$gt" : {"$humanTime" :"2019-07-01"}}}]
}
]}
},
{ "$addFields" : {"date" :{"$cond": {"if": {"$ne" : ["$txn.I_C", undefined]}, "then": "$txn.I_C", "else": "$txn.L2_C"}}}},
{
"$project" : {
"_id" : 0,
"datePartDay": {
"$concat": [
{
"$substr": [
{
"$year": "$date"
},
0,
4
]
},
"-",
{
"$substr": [
{
"$month": "$date"
},
0,
2
]
},
"-",
{
"$substr": [
{
"$dayOfMonth": "$date"
},
0,
2
]
}
]
}
}
},
{
"$group": {
"_id": "$datePartDay",
"count": {
"$sum": 1
}
}
}
]
}

mongodb update single object in array

How to update Single object in array of objects by rules query is that object must by in document that has translation_key == as input that i read and object must have same language (id) if not i must create new object . ( mongodb, mongoose)
Language are required and uniueq in translations
{
"_id" : ObjectId("5bfd5324725fb12bc4863cd8"),
"deleted" : false,
"deleted_at" : null,
"noAuth" : false,
"hidden" : false,
"translation_key" : "Standard",
"translation_type" : "text",
"translation" : [
{
"system" : {
"android" : false,
"ios" : false,
"web" : false,
"api" : false
},
"language" : ObjectId("5bf52e06edb9902e2113d8b3"),
"text":'',
"active" : false,
"prepared_string" : false,
"_id" : ObjectId("5bfd910c2998d929644abd90"),
"params" : []
},
{
"system" : {
"android" : false,
"ios" : false,
"web" : true,
"api" : false
},
"active" : false,
"prepared_string" : false,
"_id" : ObjectId("5bfdb5d22998d929644ac2af"),
"language" : ObjectId("5bf52e06edb9902e6471d8c1"),
"text" : "Standard",
"app_version" : "2.0",
"params" : []
}
],
"create_date" : ISODate("2018-11-27T14:22:28.635Z"),
"update_date" : ISODate("2018-11-27T14:22:28.635Z"),
"__v" : 3
}
Model.findOneAndUpdate(
{
_id: "document_id",
"translation.language": "languageId",
},
{
$set: {
"translation.$.text: "text",
"translation.$.active": "status",
"translation.$.system.android" : true
},
{ new: true, upsert: true, }
)
.lean();

Modifying or Unwinding deep nested arrays with mongo query

After having made my own aggregation pipeline, I have arrived at the following result:
[
{
"_id": "53712c7238b8d900008ef71c",
"address": [
{
"_id": "53712c7238b8d900008ef71b",
"phoneNumber": "4092348294",
...
}
],
"name": "TestDealer",
"vehicles": [
{
"_id": "53712fa138b8d900008ef720",
"createdAt": "2014-05-12T20:08:00.000Z",
"tags": [
"vehicle"
],
"opts": [
{
"_id": "53712fa138b8d900008ef71e",
"hasSunroof": false,
...
}
],
"listed": true,
"disclosures": [
{
"_id": "53712fa138b8d900008ef71f",
"waterDamage": false,
...
}
],
"details": [
{
"_id": "53712fa138b8d900008ef71d",
"year": 2007,
...
}
]
},
{
"_id": "5373c6439058859af7904a17",
"createdAt": "2014-05-14T19:17:00.000Z",
"tags": [
"vehicle"
],
"opts": [
{
"_id": "5373c6439058859af7904a15",
"hasSunroof": false,
....
"other": []
}
],
"listed": true,
"disclosures": [
{
"_id": "5373c6439058859af7904a16",
"waterDamage": false,
...
}
],
"details": [
{
"_id": "5373c6439058859af7904a14",
"year": 2013,
...
}
]
}
]
}
]
As you can see, vehicles is an array of subdocuments (vehicles) that follow a specific schema:
details: an array of details subdocs
opts: an array of opts subdocs
disclosures: an array of disclosures subdocs
For each of details, disclosures and opts I would like to return the first subdoc in the array as opposed to an array. However, I'm not sure where to start. I can't really $unwind vehicles because it is possible that it might be empty.
Here's is what my final data should look like:
[
{
"_id": "53712c7238b8d900008ef71c",
"address": [
{
"_id": "53712c7238b8d900008ef71b",
"phoneNumber": "4092348294",
...
}
],
"name": "TestDealer",
"vehicles": [
{
"_id": "53712fa138b8d900008ef720",
"createdAt": "2014-05-12T20:08:00.000Z",
"tags": [
"vehicle"
],
"opts": {
"_id": "53712fa138b8d900008ef71e",
"hasSunroof": false,
...
},
"listed": true,
"disclosures": {
"_id": "53712fa138b8d900008ef71f",
"waterDamage": false,
...
},
"details": {
"_id": "53712fa138b8d900008ef71d",
"year": 2007,
...
}
},
{
"_id": "5373c6439058859af7904a17",
"createdAt": "2014-05-14T19:17:00.000Z",
"tags": [
"vehicle"
],
"opts": {
"_id": "5373c6439058859af7904a15",
"hasSunroof": false,
....
"other": []
},
"listed": true,
"disclosures": {
"_id": "5373c6439058859af7904a16",
"waterDamage": false,
...
},
"details": {
"_id": "5373c6439058859af7904a14",
"year": 2013,
...
}
}
]
}
]
EDIT:
Here is the aggregation I do to get my data (in CoffeeScript):
{
$match:
_id: ObjectId(params.id)
}
{
$unwind: '$inventories'
}
{
$match:
'inventories._id': 'active'
}
{
$redact:
$cond:
if:
$eq:['$listed',false]
then: '$$PRUNE'
else: '$$DESCEND'
}
{
$project:
address: 1
name: '$dealerName'
email: '$_id.email'
vehicles: '$inventories.vehicles'
}
As well as the original data:
{ "_id" : ObjectId( "53712c7238b8d900008ef71c" ),
"email" : "test#test.com",
"createdAt" : Date( 1399925280000 ),
"password" : "$2a$10$3dhicJuONkXeI9mQflXF0.JftAx7mLQaY2./6a3O7xdrs5qfH2zMW",
"firstName" : "TestName",
"lastName" : "testName",
"dealerName" : "TestDealer",
"dealerType" : "Franchised Dealer",
"emailIsVerified" : false,
"lastLoggedInAt" : Date( 1400247300000 ),
"expiresAt" : Date( 4524062882000 ),
"deletedAt" : null,
"updatedAt" : Date( 1400251500000 ),
"inventories" : [
{ "title" : "activeInventory",
"_id" : "active",
"vehicles" : [
{ "_id" : ObjectId( "53712fa138b8d900008ef720" ),
"createdAt" : Date( 1399925280000 ),
"tags" : [
"vehicle" ],
"opts" : [
{ "_id" : ObjectId( "53712fa138b8d900008ef71e" ),
"hasSunroof" : false,
"hasSnowTires" : false,
"hasPowerWindows" : false,
"hasNavigationSystem" : false,
"hasManuals" : false,
"hasMags" : false,
"hasLeather" : false,
"hasKeylessEntry" : false,
"hasHeatedSeats" : false,
"hasDvdSystem" : false,
"hasAirConditioning" : false,
"has2SetsOfKeys" : false,
"other" : [] } ],
"listed" : true,
"disclosures" : [
{ "_id" : ObjectId( "53712fa138b8d900008ef71f" ),
"waterDamage" : false,
"vehicleLien" : false,
"usVehicle" : false,
"totalLossByInsurer" : false,
"theftOrRecovery" : false,
"taxiOrLimo" : false,
"structuralPartsAreDamagedOrAlteredOrRepaired" : false,
"suspensionOrSubFrameNeedsRepair" : false,
"repainted" : false,
"previousDamageExceeding3k" : false,
"powerTrainNeedsRepair" : false,
"policeCruiser" : false,
"outOfProvinceOrState" : false,
"originalOwner" : false,
"originalManufacturerVinPlate" : false,
"originalSpecificationsChanged" : false,
"odometerRolledBackOrReplaced" : false,
"manufacturerWarrantyCancelled" : false,
"manufacturerBadgesChanged" : false,
"fuelSystemNeedsRepair" : false,
"fireDamage" : false,
"engineNeedsRepair" : false,
"emergencyVehicle" : false,
"electricalSystemNeedsRepair" : false,
"dailyRental" : false,
"computerNeedsRepair" : false,
"antiLockBrakesDamagedOrInoperable" : false,
"airbagIsMissingOrInoperable" : false,
"airConditioningNeedsRepair" : false } ],
"details" : [
{ "_id" : ObjectId( "53712fa138b8d900008ef71d" ),
"year" : 2007,
"vin" : "JN8AZ08W27W649264",
"trim" : "SE",
"transmission" : "Automatic",
"price" : 132123,
"model" : "Murano 4D Utility AWD",
"mileageReading" : 125000,
"make" : "Nissan",
"interiorColor" : "Black",
"history" : "Carproof Verified ($40.00)",
"hasAccident" : false,
"fuelType" : "Biodiesel",
"exteriorColor" : "Blue",
"driveTrain" : "FWD",
"description" : "dsadas",
"cylinders" : 4,
"mileageType" : "kms" } ] },
{ "_id" : ObjectId( "53713bfc1429925f0faf79d0" ),
"createdAt" : Date( 1399929780000 ),
"tags" : [
"vehicle" ],
"opts" : [
{ "_id" : ObjectId( "53713bfc1429925f0faf79ce" ),
"hasSunroof" : false,
"hasSnowTires" : false,
"hasPowerWindows" : false,
"hasNavigationSystem" : false,
"hasManuals" : false,
"hasMags" : false,
"hasLeather" : false,
"hasKeylessEntry" : false,
"hasHeatedSeats" : false,
"hasDvdSystem" : false,
"hasAirConditioning" : false,
"has2SetsOfKeys" : false,
"other" : [] } ],
"listed" : false,
"disclosures" : [
{ "_id" : ObjectId( "53713bfc1429925f0faf79cf" ),
"waterDamage" : false,
"vehicleLien" : false,
"usVehicle" : false,
"totalLossByInsurer" : false,
"theftOrRecovery" : false,
"taxiOrLimo" : false,
"structuralPartsAreDamagedOrAlteredOrRepaired" : false,
"suspensionOrSubFrameNeedsRepair" : false,
"repainted" : false,
"previousDamageExceeding3k" : false,
"powerTrainNeedsRepair" : false,
"policeCruiser" : false,
"outOfProvinceOrState" : false,
"originalOwner" : false,
"originalManufacturerVinPlate" : false,
"originalSpecificationsChanged" : false,
"odometerRolledBackOrReplaced" : false,
"manufacturerWarrantyCancelled" : false,
"manufacturerBadgesChanged" : false,
"fuelSystemNeedsRepair" : false,
"fireDamage" : false,
"engineNeedsRepair" : false,
"emergencyVehicle" : false,
"electricalSystemNeedsRepair" : false,
"dailyRental" : false,
"computerNeedsRepair" : false,
"antiLockBrakesDamagedOrInoperable" : false,
"airbagIsMissingOrInoperable" : false,
"airConditioningNeedsRepair" : false } ],
"details" : [
{ "_id" : ObjectId( "53713bfc1429925f0faf79cd" ),
"year" : 2007,
"vin" : "JN8AZ08W27W649264",
"trim" : "SE",
"transmission" : "Manual",
"price" : 13241234,
"model" : "Murano 4D Utility AWD",
"mileageReading" : 1312312,
"make" : "Mercedes-Benz",
"interiorColor" : "Black",
"history" : "Carproof Claims ($25.00)",
"hasAccident" : true,
"fuelType" : "Diesel",
"exteriorColor" : "Black",
"driveTrain" : "RWD",
"description" : "werwe",
"cylinders" : 4,
"mileageType" : "kms" } ] },
{ "_id" : ObjectId( "5373c6439058859af7904a17" ),
"createdAt" : Date( 1400095020000 ),
"tags" : [
"vehicle" ],
"opts" : [
{ "_id" : ObjectId( "5373c6439058859af7904a15" ),
"hasSunroof" : false,
"hasSnowTires" : false,
"hasPowerWindows" : false,
"hasNavigationSystem" : false,
"hasManuals" : false,
"hasMags" : false,
"hasLeather" : false,
"hasKeylessEntry" : false,
"hasHeatedSeats" : false,
"hasDvdSystem" : false,
"hasAirConditioning" : false,
"has2SetsOfKeys" : false,
"other" : [] } ],
"listed" : true,
"disclosures" : [
{ "_id" : ObjectId( "5373c6439058859af7904a16" ),
"waterDamage" : false,
"vehicleLien" : false,
"usVehicle" : false,
"totalLossByInsurer" : false,
"theftOrRecovery" : false,
"taxiOrLimo" : false,
"structuralPartsAreDamagedOrAlteredOrRepaired" : false,
"suspensionOrSubFrameNeedsRepair" : false,
"repainted" : false,
"previousDamageExceeding3k" : false,
"powerTrainNeedsRepair" : false,
"policeCruiser" : false,
"outOfProvinceOrState" : false,
"originalOwner" : false,
"originalManufacturerVinPlate" : false,
"originalSpecificationsChanged" : false,
"odometerRolledBackOrReplaced" : false,
"manufacturerWarrantyCancelled" : false,
"manufacturerBadgesChanged" : false,
"fuelSystemNeedsRepair" : false,
"fireDamage" : false,
"engineNeedsRepair" : false,
"emergencyVehicle" : false,
"electricalSystemNeedsRepair" : false,
"dailyRental" : false,
"computerNeedsRepair" : false,
"antiLockBrakesDamagedOrInoperable" : false,
"airbagIsMissingOrInoperable" : false,
"airConditioningNeedsRepair" : false } ],
"details" : [
{ "_id" : ObjectId( "5373c6439058859af7904a14" ),
"year" : 2013,
"vin" : "12345678901234567",
"trim" : "SE",
"transmission" : "Automatic",
"price" : 13564,
"model" : "Red",
"mileageReading" : 13453,
"make" : "Daihatsu",
"interiorColor" : "Black",
"history" : "Carproof Verified ($40.00)",
"hasAccident" : true,
"fuelType" : "Diesel",
"exteriorColor" : "Black",
"driveTrain" : "FWD",
"description" : "wrw",
"cylinders" : 3,
"mileageType" : "kms" } ] } ],
"tags" : [
"inventory",
"active",
"vehicles" ] },
{ "title" : "soldInventory",
"_id" : "sold",
"vehicles" : [],
"tags" : [
"inventory",
"sold",
"vehicles" ] },
{ "title" : "deletedInventory",
"_id" : "deleted",
"vehicles" : [],
"tags" : [
"inventory",
"deleted",
"vehicles" ] } ],
"address" : [
{ "_id" : ObjectId( "53712c7238b8d900008ef71b" ),
"phoneNumber" : "4092348294",
"country" : "Canada",
"zip" : "301111",
"region" : "Quebec",
"city" : "Montreal",
"street2" : "Apt. 101",
"street1" : "1213 Street",
"additionalNumbers" : [] } ],
"__v" : 3 }
If you are using 2.6 (which you must be since you use $redact) and if the array is always guaranteed to be empty if there are no tags or opts, you can use this technique to make sure you don't lose vehicles which have empty arrays when you $unwind:
In a $project phase add the following for all arrays you want to $unwind to keep just the first element:
opts:{$cond:{if:{$eq:[{$size:"$opts"},0]}, then:{$literal:[ "none" ]}, else:"$opts"}}
For each sub-array you can now group keeping the first one, and then repeat again for the next sub-array.
You'll need to be careful about unwinding and re-grouping for each array separately to keep from ending up with any duplication, if you are not aggregating anything else, you can do it in a single group with as many $unwinds as there are sub-arrays.