Related
I'm facing a problem with the lookup in the second (student) table that matches all incoming output records of the first(test) table. I have two collections "tests" and "students". "Test" collection contains all school tests and the "student" table contains student's attended tests. Student table contains "pastTest"(test attended in past with status "pass" or "fail")array. I want to retrieve student who passed all incoming tests (we retrieve from the tests table)
test table: _id (primary ket)
student.pastTests.testId (need to match with test._id)
Test Document:
{
"_id" : ObjectId("5c9b5c1005729b2bf23f3290"),
"testDate" : {
"term" : 1,
"week" : 7
},
"retestDate" : {
"term" : 1,
"week" : 10
},
"testOrder" : "1.1",
"testDateScheduled" : true,
"retestDateScheduled" : true
}
Student Document:
{
"_id" : ObjectId("5c92dd994e8e6b2c1647d0d0"),
"completedYears" : [],
"firstName" : "Andrew",
"lastName" : "Jonhson",
"teacherId" : ObjectId("5bf36b1076696374e65feb4f"),
"yearGroup" : "0",
"schoolId" : 40001,
"currentTest" : ObjectId("5c9b5c1005729b2bf23f3290"),
"pastTests" : [
{
"_id" : ObjectId("5d3570645045863d373f6db1"),
"testId" : ObjectId("5c9b5c1005729b2bf23f3290"),
"status" : "pass"
},
{
"_id" : ObjectId("5d425af07708f5636c3bec1c"),
"testId" : ObjectId("5c9b5fc460e39c2c58e44109"),
"status" : "pass"
},
{
"_id" : ObjectId("5d5e54a875fab079f4d03570"),
"testId" : ObjectId("5c9b6492bb581c2ceb553fef"),
"status" : "fail"
},
],
"createdAt" : ISODate("2019-03-21T00:40:57.401Z"),
"updatedAt" : ISODate("2020-09-24T19:55:38.291Z"),
"__v" : 0,
"holdTests" : [],
"completedTests" : [],
"className" : "dd",
}
Query:
db.getCollection('tests').aggregate([
{
$match: {
yearGroup: '-1',
$or : [
{
$and: [
{'retestDateScheduled': true},
{ 'retestDate.term': { $lt: 4 } },
]
},
{
$and: [
{'testDateScheduled': true},
{ 'testDate.term': { $lt: 4 } },
]
}
]
}
},
{
$lookup: {
from: 'students',
let: {testId: '$_id', schoolId: 49014, yearGroup: '-1'},
pipeline: [
]
}
}
])
Note: Initial match query returns all tests of the term-1, now I have to retrieve students who passed in all tests of the term-1.
Lookup stage is pending - facing problem with lookup in second (student) table who match all incoming output records of first(test) collection
Thanks in advance !!
Try this:
db.tests.aggregate([
{
$match: {
// Your match condition
}
},
{
$group: {
_id: null,
term_1_testIds: { $push: "$_id" },
test_count: { $sum: 1 }
}
},
{
$lookup: {
from: "students",
let: { term_1_testIds: '$term_1_testIds', schoolId: 40001, totalTestCount: "$test_count" },
pipeline: [
{
$match: {
$expr: { $eq: ["$schoolId", "$$schoolId"] }
}
},
{ $unwind: "$pastTests" },
{
$match: {
"pastTests.status": "pass",
$expr: { $in: ["$pastTests.testId", "$$term_1_testIds"] }
}
},
{
$group: {
_id: "$_id",
firstName: { $first: "$firstName" },
yearGroup: { $first: "$yearGroup" },
schoolId: { $first: "$schoolId" },
currentTest: { $first: "$currentTest" },
passedTestCount: { $sum: 1 },
pastTests: { $push: "$pastTests" }
}
},
{
$match: {
$expr: { $eq: ["$passedTestCount", "$$totalTestCount"] }
}
}
],
as: "students"
}
}
]);
Output:
{
"_id" : null,
"term_1_testIds" : [
ObjectId("5c9b5c1005729b2bf23f3290"),
ObjectId("5c9b5fc460e39c2c58e44109"),
ObjectId("5c9b6492bb581c2ceb553fef")
],
"test_count" : 3,
"students" : [
{
"_id" : ObjectId("5c92dd994e8e6b2c1647d0d1"),
"firstName" : "Dheemanth",
"yearGroup" : "0",
"schoolId" : 40001,
"currentTest" : ObjectId("5c9b5c1005729b2bf23f3290"),
"passedTestCount" : 3,
"pastTests" : [
{
"_id" : ObjectId("5d3570645045863d373f6db1"),
"testId" : ObjectId("5c9b5c1005729b2bf23f3290"),
"status" : "pass"
},
{
"_id" : ObjectId("5d425af07708f5636c3bec1c"),
"testId" : ObjectId("5c9b5fc460e39c2c58e44109"),
"status" : "pass"
},
{
"_id" : ObjectId("5d5e54a875fab079f4d03570"),
"testId" : ObjectId("5c9b6492bb581c2ceb553fef"),
"status" : "pass"
}
]
}
]
}
This how my tests collection looks like
/* 1 createdAt:3/27/2019, 5:24:58 PM*/
{
"_id" : ObjectId("5c9b6492bb581c2ceb553fef"),
"name" : "Test 3"
},
/* 2 createdAt:3/27/2019, 5:04:28 PM*/
{
"_id" : ObjectId("5c9b5fc460e39c2c58e44109"),
"name" : "Test 2"
},
/* 3 createdAt:3/27/2019, 4:48:40 PM*/
{
"_id" : ObjectId("5c9b5c1005729b2bf23f3290"),
"name" : "Test 1"
}
This is how my students collection looks like:
/* 1 createdAt:3/21/2019, 6:10:57 AM*/
{
"_id" : ObjectId("5c92dd994e8e6b2c1647d0d1"),
"firstName" : "Dheemanth",
"yearGroup" : "0",
"schoolId" : 40001,
"currentTest" : ObjectId("5c9b5c1005729b2bf23f3290"),
"pastTests" : [
{
"_id" : ObjectId("5d3570645045863d373f6db1"),
"testId" : ObjectId("5c9b5c1005729b2bf23f3290"),
"status" : "pass"
},
{
"_id" : ObjectId("5d425af07708f5636c3bec1c"),
"testId" : ObjectId("5c9b5fc460e39c2c58e44109"),
"status" : "pass"
},
{
"_id" : ObjectId("5d5e54a875fab079f4d03570"),
"testId" : ObjectId("5c9b6492bb581c2ceb553fef"),
"status" : "pass"
}
]
},
/* 2 createdAt:3/21/2019, 6:10:57 AM*/
{
"_id" : ObjectId("5c92dd994e8e6b2c1647d0d0"),
"firstName" : "Andrew",
"yearGroup" : "0",
"schoolId" : 40001,
"currentTest" : ObjectId("5c9b5c1005729b2bf23f3290"),
"pastTests" : [
{
"_id" : ObjectId("5d3570645045863d373f6db1"),
"testId" : ObjectId("5c9b5c1005729b2bf23f3290"),
"status" : "pass"
},
{
"_id" : ObjectId("5d425af07708f5636c3bec1c"),
"testId" : ObjectId("5c9b5fc460e39c2c58e44109"),
"status" : "pass"
},
{
"_id" : ObjectId("5d5e54a875fab079f4d03570"),
"testId" : ObjectId("5c9b6492bb581c2ceb553fef"),
"status" : "fail"
}
]
}
Also:
In your first $match stage, $and operator is redundant inside $or array it should be like this:
{
$match: {
yearGroup: '-1',
$or: [
{
'retestDateScheduled': true,
'retestDate.term': { $lt: 4 }
},
{
'testDateScheduled': true,
'testDate.term': { $lt: 4 }
}
]
}
}
I have two collections:
quiz_customer_record collection
{
"_id" : ObjectId("5f6ec91cbf74d27430b9c24f"),
"quiz_id" : "5f3a33185a1cd35632b8c98c",
"user_id" : "5efae8bed5c5f06f30a057ff",
"name" : "ABC",
"qualification" : "ttt",
"time_required" : "0:13 Mins",
"questions_attempted" : 2,
"total_quiz_questions" : 2,
"attempt_date" : "2020-09-26T04:52:48.169Z"
}
/* 4 */
{
"_id" : ObjectId("5f6eca82bf74d27430b9c252"),
"quiz_id" : "5f3a33185a1cd35632b8c98c",
"user_id" : "5f6ec9ba3b502398598a5ade",
"name" : "Test",
"qualification" : "BSC",
"time_required" : "0:6 Mins",
"questions_attempted" : 2,
"total_quiz_questions" : 2,
"attempt_date" : "2020-09-26T04:58:46.060Z"
}
dummy collection
/* 1 */
{
"_id" : ObjectId("5f6ec906bf74d27430b9c24d"),
"user_id" : "5efae8bed5c5f06f30a057ff",
"question_id" : "5f6ec888bf74d27430b9c248",
"quiz_id" : "5f3a33185a1cd35632b8c98c",
"selected_answer" : [
"rgdfgdfg"
],
"attempt_date" : "2020-09-26T04:52:25.977Z",
"correct_answer" : [
"rgdfgdfg"
],
"result" : true
}
/* 2 */
{
"_id" : ObjectId("5f6eca82bf74d27430b9c250"),
"user_id" : "5f6ec9ba3b502398598a5ade",
"question_id" : "5f6ec888bf74d27430b9c248",
"quiz_id" : "5f3a33185a1cd35632b8c98c",
"selected_answer" : [
"rgdfgdfg"
],
"attempt_date" : "2020-09-26T04:58:46.060Z",
"correct_answer" : [
"rgdfgdfg"
],
"result" : true
}
/* 3 */
{
"_id" : ObjectId("5f6eca82bf74d27430b9c251"),
"user_id" : "5f6ec9ba3b502398598a5ade",
"question_id" : "5f6ec8b4bf74d27430b9c24b",
"quiz_id" : "5f3a33185a1cd35632b8c98c",
"selected_answer" : [
"sdfsdf"
],
"attempt_date" : "2020-09-26T04:58:46.060Z",
"correct_answer" : [
"sdfsdf"
],
"result" : true
}
From the 2nd(dummy collection i want the total records per user)
I am using this query in which i need modifications:
db.quiz_customer_record.aggregate([{ $match: { quiz_id:"5f3a33185a1cd35632b8c98c"}},
{
$sort: { attempt_date: -1 }
},
{
$group: {
_id: "$user_id",
result1: { $first: "$attempt_date" },
quiz_id: { $first: "$quiz_id" },
o_id: { $first: "$_id" }
}
},
{
$project: {
_id: "$o_id",
user_id: "$_id",
result1: 1
}
}
])
this will give the result as:
/* 1 */
{
"attempt_date" : "2020-09-26T04:52:48.169Z",
"_id" : ObjectId("5f6ec91cbf74d27430b9c24f"),
"user_id" : "5efae8bed5c5f06f30a057ff"
}
/* 2 */
{
"attempt_date" : "2020-09-26T04:58:46.060Z",
"_id" : ObjectId("5f6eca82bf74d27430b9c252"),
"user_id" : "5f6ec9ba3b502398598a5ade"
}
Expected Result: (as per user_id I need the count of records from dummy collection where quiz_id and attempt_date(result1 from above query) matches)
/* 1 */
{
"attempt_date" : "2020-09-26T04:52:48.169Z",
"_id" : ObjectId("5f6ec91cbf74d27430b9c24f"),
"user_id" : "5efae8bed5c5f06f30a057ff",
"total_dummy_rec":0
}
/* 2 */
{
"attempt_date" : "2020-09-26T04:58:46.060Z",
"_id" : ObjectId("5f6eca82bf74d27430b9c252"),
"user_id" : "5f6ec9ba3b502398598a5ade",
"total_dummy_rec":2
}
where total_dummy_rec is the count of total record per user in "dummy" collection.
I am confused on how to approach so i can achieve this result. Help me find a solution. Thank you!
You can add 2 stages after your pipeline stages,
$lookup to join dummy collection, where pass required field in let and in pipeline match condition
moved $project at last and count total document in dummy using $size
{
$lookup: {
from: "dummy",
let: {
quiz_id: "$quiz_id",
user_id: "$_id",
attempt_date: "$attempt_date"
},
pipeline: [
{
$match: {
$expr: {
$and: [
{ $eq: ["$$quiz_id", "$quiz_id"] },
{ $eq: ["$$user_id", "$user_id"] },
{ $eq: ["$$attempt_date", "$attempt_date"] }
]
}
}
}
],
as: "dummy"
}
},
{
$project: {
_id: "$o_id",
user_id: "$_id",
result1: 1,
total_dummy_rec: {
$size: "$dummy"
}
}
}
Playground
What I have been trying to get my head around is to perform some kind of partitioning(split by predicate) in a mongo query. My current query looks like:
db.posts.aggregate([
{"$match": { $and:[ {$or:[{"toggled":false},{"toggled":true, "status":"INACTIVE"}]} , {"updatedAt":{$gte:1549786260000}} ] }},
{"$unwind" :"$interests"},
{"$group" : {"_id": {"iid": "$interests", "pid":"$publisher"}, "count": {"$sum" : 1}}},
{"$project":{ _id: 0, "iid": "$_id.iid", "pid": "$_id.pid", "count": 1 }}
])
This results in the following output:
{
"count" : 3.0,
"iid" : "INT456",
"pid" : "P789"
}
{
"count" : 2.0,
"iid" : "INT789",
"pid" : "P789"
}
{
"count" : 1.0,
"iid" : "INT123",
"pid" : "P789"
}
{
"count" : 1.0,
"iid" : "INT123",
"pid" : "P123"
}
All good so far, but then I had realized that for the documents that match the specific filter {"toggled":true, "status":"INACTIVE"}, I would rather decrement the count (-1). (considering the eventual value can be negative as well.)
Is there a way to somehow partition the data after match to make sure different grouping operations are performed for both the collection of documents?
Something that sounds similar to what I am looking for is
$mergeObjects, or maybe $reduce, but not much that I can relate from the documentation examples.
Note: I can sense, one straightforward way to deal with this would be to perform two queries, but I am looking for a single query to perform the operation.
Sample documents for the above output would be:
/* 1 */
{
"_id" : ObjectId("5d1f7******"),
"id" : "CON123",
"title" : "Game",
"content" : {},
"status" : "ACTIVE",
"toggle":false,
"publisher" : "P789",
"interests" : [
"INT456"
],
"updatedAt" : NumberLong(1582078628264)
}
/* 2 */
{
"_id" : ObjectId("5d1f8******"),
"id" : "CON456",
"title" : "Home",
"content" : {},
"status" : "INACTIVE",
"toggle":true,
"publisher" : "P789",
"interests" : [
"INT456",
"INT789"
],
"updatedAt" : NumberLong(1582078628264)
}
/* 3 */
{
"_id" : ObjectId("5d0e9******"),
"id" : "CON654",
"title" : "School",
"content" : {},
"status" : "ACTIVE",
"toggle":false,
"publisher" : "P789",
"interests" : [
"INT123",
"INT456",
"INT789"
],
"updatedAt" : NumberLong(1582078628264)
}
/* 4 */
{
"_id" : ObjectId("5d207*******"),
"id" : "CON789",
"title":"Stack",
"content" : { },
"status" : "ACTIVE",
"toggle":false,
"publisher" : "P123",
"interests" : [
"INT123"
],
"updatedAt" : NumberLong(1582078628264)
}
What I am looking forward to as a result though is
{
"count" : 1.0, (2-1)
"iid" : "INT456",
"pid" : "P789"
}
{
"count" : 0.0, (1-1)
"iid" : "INT789",
"pid" : "P789"
}
{
"count" : 1.0,
"iid" : "INT123",
"pid" : "P789"
}
{
"count" : 1.0,
"iid" : "INT123",
"pid" : "P123"
}
This aggregation gives the desired result.
db.posts.aggregate( [
{ $match: { updatedAt: { $gte: 1549786260000 } } },
{ $facet: {
FALSE: [
{ $match: { toggle: false } },
{ $unwind : "$interests" },
{ $group : { _id : { iid: "$interests", pid: "$publisher" }, count: { $sum : 1 } } },
],
TRUE: [
{ $match: { toggle: true, status: "INACTIVE" } },
{ $unwind : "$interests" },
{ $group : { _id : { iid: "$interests", pid: "$publisher" }, count: { $sum : -1 } } },
]
} },
{ $project: { result: { $concatArrays: [ "$FALSE", "$TRUE" ] } } },
{ $unwind: "$result" },
{ $replaceRoot: { newRoot: "$result" } },
{ $group : { _id : "$_id", count: { $sum : "$count" } } },
{ $project:{ _id: 0, iid: "$_id.iid", pid: "$_id.pid", count: 1 } }
] )
[ EDIT ADD ]
The output from the query using the input data from the question post:
{ "count" : 1, "iid" : "INT123", "pid" : "P789" }
{ "count" : 1, "iid" : "INT123", "pid" : "P123" }
{ "count" : 0, "iid" : "INT789", "pid" : "P789" }
{ "count" : 1, "iid" : "INT456", "pid" : "P789" }
[ EDIT ADD 2 ]
This query gets the same result with different approach (code):
db.posts.aggregate( [
{
$match: { updatedAt: { $gte: 1549786260000 } }
},
{
$unwind : "$interests"
},
{
$group : {
_id : {
iid: "$interests",
pid: "$publisher"
},
count: {
$sum: {
$switch: {
branches: [
{ case: { $eq: [ "$toggle", false ] },
then: 1 },
{ case: { $and: [ { $eq: [ "$toggle", true] }, { $eq: [ "$status", "INACTIVE" ] } ] },
then: -1 }
]
}
}
}
}
},
{
$project:{
_id: 0,
iid: "$_id.iid",
pid: "$_id.pid",
count: 1
}
}
] )
[ EDIT ADD 3 ]
NOTE:
The facet query runs the two facets (TRUE and FALSE) on the same set of documents; it is like two queries running in parallel. But, there is some duplication of code as well as additional stages for shaping the documents down the pipeline to get the desired output.
The second query avoids the code duplication, and there are much lesser stages in the aggregation pipeline. This will make difference when the input dataset has a large number of documents to process - in terms of performance. In general, lesser stages means lesser iterations of the documents (as a stage has to scan the documents which are output from the previous stage).
In Db I have some sample data:
Object 1
"_id" : ObjectId("5b5934bb49b")
"payment" : {
"paid_total" : 500,
"name" : "havi",
"payment_mode" : "cash",
"pd_no" : "PD20725001",
"invoices" : [
{
"invoice_number" : "IN11803831583"
}
],
"type" : "Payment"
}
Object 2
"_id" : ObjectId("5b5934ee31e"),
"patient" : {
"invoice_date" : "2018-07-26",
"invoiceTotal" : 2000,
"pd_no" : "PD20725001",
"type" : "Invoice",
"invoice_number" : "IN11803831583"
}
Note: All the Data is In same Collection
As the above shown data I have many objects in my database. How can I get the Sum from the data above of invoiceTotal and sum of paid_total and then subtract the paid_total from invoiceTotal and show the balance amount for matching pd_no and invoice_number.
The output I expect looks like
invoiceTotal : 2000
paid_total : 500
Balance : 1500
Sample Input :
{
"_id" : ObjectId("5b596969a88e07f00d6dac17"),
"payment" : {
"paid_total" : 500,
"name" : "havi",
"payment_mode" : "cash",
"pd_no" : "PD20725001",
"invoices" : [
{
"invoice_number" : "IN11803831583"
}
],
"type" : "Payment"
}
}
{
"_id" : ObjectId("5b596986a88e07f00d6dac18"),
"patient" : {
"invoice_date" : "2018-07-26",
"invoiceTotal" : 2000,
"pd_no" : "PD20725001",
"type" : "Invoice",
"invoice_number" : "IN11803831583"
}
}
Use this aggregate query :
db.test.aggregate([
{
$project : {
_id : 0,
pd_no : { $ifNull: ["$payment.pd_no", "$patient.pd_no" ] },
invoice_no : { $ifNull: [ { $arrayElemAt : ["$payment.invoices.invoice_number", 0] },"$patient.invoice_number" ] },
type : { $ifNull: [ "$payment.type", "$patient.type" ] },
paid_total : { $ifNull: [ "$payment.paid_total", 0 ] },
invoice_total : { $ifNull: [ "$patient.invoiceTotal", 0 ] },
}
},
{
$group : {
_id : {
pd_no : "$pd_no",
invoice_no : "$invoice_no"
},
paid_total : {$sum : "$paid_total"},
invoice_total : {$sum : "$invoice_total"}
}
},
{
$project : {
_id : 0,
pd_no : "$_id.pd_no",
invoice_no : "$_id.invoice_no",
invoice_total : "$invoice_total",
paid_total : "$paid_total",
balance : {$subtract : ["$invoice_total" , "$paid_total"]}
}
}
])
In this query we are first finding the pd_no and invoice_no, which we are then using to group the documents. Next, we are getting the invoice_total and paid_total and then subtracting them to get the balance.
Output :
{
"pd_no" : "PD20725001",
"invoice_no" : "IN11803831583",
"invoice_total" : 2000,
"paid_total" : 500,
"balance" : 1500
}
I assume that you will only have documents with invoiceTotal or paid_total and never both at the same time.
you need first to get an amount to get the balance so if paid total it needs to be negative and positive on the case of the invoice total, and you can do this by using first the $project on the pipeline.
collection.aggregate([
{
$project : {
'patient.invoiceTotal': 1,
'payment.paid_total': 1,
ammount: {
$ifNull: ['$patient.invoiceTotal', { $multiply: [-1, '$payment.paid_total']}]
}
}
},
{
$group: {
_id: 'myGroup',
invoiceTotal: { $sum: '$patient.invoiceTotal' },
paid_total: { $sum: '$payment.paid_total' },
balance: { $sum: '$ammount' }
}
}
])
I'm trying to return the total of requests by type based on their status:
If there is no status set, the request should be added to requested
If the status is ordered, the request should be added to ordered
If the status is arrived, the request should be added to arrived
caseRequest.aggregate([{
$group: {
_id: "$product",
suggested: {
$sum: {
$cond: [{
$ifNull: ["$status", true]
},
1, 0
]}
},
ordered: {
$sum: {
$cond: [{
$eq: ["$status", "ordered"]
},
1, 0
]
}
},
arrived: {
$sum: {
$cond: [{
$eq: ["$status", "arrived"]
},
1, 0
]
}
}
}
}
But for some reason it doesn't find any request status ordered or arrived. If in the database I have 48 requests, 45 of them without status, 2 with ordered and 1 with arrived, it returns:
[
{
_id: "xxx",
suggested: 48,
ordered: 0,
arrived: 0,
},
...
]
Try this approach,
Return the total number of requests by type based on their status
Now the simplest way to get the count of different status is to use aggregate pipeline with $group on the status field
db.stackoverflow.aggregate([{ $group: {_id: "$status", count: {$sum:1}} }])
We will be getting a result similar to this
{ "_id" : "", "count" : 2 }
{ "_id" : "arrived", "count" : 3 }
{ "_id" : "ordered", "count" : 4 }
The schema which is used to retrieve these records is very simple so that it will be easier to understand. The schema will have a parameter on the top level of the document and the value of status can be "ordered", "arrived" or empty
Schema
{ "_id" : ObjectId("5798c348d345404e7f9e0ced"), "status" : "ordered" }
The collection is populated with 9 records, with status as ordered, arrived and empty
db.stackoverflow.find()
{ "_id" : ObjectId("5798c348d345404e7f9e0ced"), "status" : "ordered" }
{ "_id" : ObjectId("5798c349d345404e7f9e0cee"), "status" : "ordered" }
{ "_id" : ObjectId("5798c34ad345404e7f9e0cef"), "status" : "ordered" }
{ "_id" : ObjectId("5798c356d345404e7f9e0cf0"), "status" : "arrived" }
{ "_id" : ObjectId("5798c357d345404e7f9e0cf1"), "status" : "arrived" }
{ "_id" : ObjectId("5798c358d345404e7f9e0cf2"), "status" : "arrived" }
{ "_id" : ObjectId("5798c35ad345404e7f9e0cf3"), "status" : "ordered" }
{ "_id" : ObjectId("5798c361d345404e7f9e0cf4"), "status" : "" }
{ "_id" : ObjectId("5798c362d345404e7f9e0cf5"), "status" : "" }
db.stackoverflow.count()
9
Hope it Helps!!