I'm migrating from MySQL to MongoDB. In this process I want to rewrite the following MySQL query:
This query is used to add additional information to the minutes table (adding the program, title, broadcaster and program_id (prid) to the other table)
UPDATE etl_staging.sko_minutes minut
JOIN etl_staging.sko_daily AS daily ON
minut.channel = daily.channel AND
(minut.C BETWEEN daily.start_datetime and daily.end_datetime)
set minut.sko_frequency = daily.sko_frequency,
minut.programma = daily.programma,
minut.titel = daily.titel,
minut.omroep = daily.omroep,
minut.prid = daily.prid
WHERE daily.doelgroep = '6+'
AND daily.universe = 'Currency'
Now i'm trying to rewrite it to the following... Only i found out that i could not make a join in MongoDB. Was looking at something like this. But that returns an empty minutes array as field to the minutes document.
db.sko_minutes.aggregate(
[
{ $lookup: {
from: "sko_daily",
let: { channel: "$channel", start_datetime: "$start_datetime", end_datetime: "$end_datetime",
doelgroep: "$doelgroep", universe: "$universe"},
pipeline: [
{ $match: {
$expr: {
$and: [
{ $eq: ["$$universe", 'Currency']},
{ $eq: ["$$doelgroep", '6+']},
{ $eq: ["$channel", "$$channel"]},
{ $gte: ["$datetime", "$$start_datetime"]},
{ $lte: ["$datetime", "$$end_datetime"]}
]
}
}}
],
as: 'minutes'
}}
]
)
Does anybody have an idea what would be the best approach for this problem?
The documents of the sko_minutes looks like this:
{
"_id": {"$oid": "5fbb8b85336e42949248fb1b"},
"abs": 0,
"channel": "vicetv",
"date": {"$date": "2020-11-22T00:00:00.000Z"},
"datetime": {"$date": "2020-11-22T09:20:00.000Z"},
"hour": 9,
"kta": 0,
"minutes": "20",
"start_time": "09:20:00"
},
And the daily document looks like this:
{
"_id": {"$oid": "5fbb8afb4cab8a4ce5acac5f"},
"abs": 0,
"brk": 0,
"channel": "net5",
"date": "2020-11-18",
"doelgroep": "20-34",
"duration": "47",
"end_date": "2020-11-18",
"end_datetime": {"$date": "2020-11-18T01:12:00.000Z"},
"end_hour": "1",
"end_minutes": "12",
"end_time": "25:12:59",
"hour": "0",
"kdh": 0,
"kta": 0,
"minutes": "12",
"omroep": "net5",
"prid": "694949124",
"programma": "vtwonenverbouwenofverhuizen",
"sko_frequency": "overige herhaling",
"start_datetime": {"$date": "2020-11-18T00:12:00.000Z"},
"start_time": "24:12:00",
"titel": "Vtwonen verbouwen of verhuizen",
"universe": "UGK1-6",
"waardering": 0,
"webtv_gaas": 0,
"webtv_nstreams": 0
},
Brings part of sko_daily to sko_minutes
let fields are from sko_minutes: date, datetime, channel
this fields are matched to daily
You also require a subset of sko_daily, using filters.
Mind you that I tweaked the date fields, and some data, just to make mongoplayground work.
Playground
db.sko_minutes.aggregate([
{
$lookup: {
/**$$ are from sko_minutes*/
from: "sko_daily",
let: {
"start_datetime": "$date",
"end_datetime": "$datetime",
channel: "$channel"
},
pipeline: [
{
$match: {
$expr: {
$and: [
{
$eq: [
"$universe",
"Currency"
]
},
{
$eq: [
"$doelgroep",
"6+"
]
},
{
$eq: [
"$$channel",
"$channel"
]
},
{
$gte: [
"$$start_datetime",
"$start_datetime"
]
},
{
$lte: [
"$$end_datetime",
"$end_datetime"
]
}
]
}
}
}
],
as: "minutes"
}
}
])
Related
I have this sample data:
[
{
"customers": [
{"id": 100, "name": "a"},
{"id": 200, "name": "b"},
{"id": 300, "name": "c"},
{"id": 400, "name": "d"}
],
"sales": [
{
"sale_id": 9999,
"persons_related": [
{"id": 100},
{"id": 900},
{"id": 800}
]
},
{
"sale_id": 9998,
"persons_related": [
{"id": 500},
{"id": 550},
{"id": 560}
]
},
]
}
]
It represents two collections, customers and sales.
Imagine that I am working with the customers collection, I have selected just the customer 100
db.collection.aggregate([
{ $project: {
"customer": { "$arrayElemAt": [ "$customers", 0 ]
}}
}
Which returns:
[
{
"_id": ObjectId("5a934e000102030405000000"),
"customer": {
"id": 100,
"name": "a"
}
}
]
And I want to find up the sales where this id appears, so I lookup against the same collection, adding this stage to the pipeline:
{ $lookup: {
from: "collection",
let: { id: "$customer.id" },
pipeline: [
{ $match: {
$and: [
{ $expr: { $in: [ "$$id", "$sales.persons_related.id" ] } }
]
}}
],
as: "sales"
}
}
I need to use this lookup version (the one with let and pipeline, and not the other one with
localField/foreignField) because I need to add additional filters in the match stage of the pipeline. However, this part:
{ $expr: { $in: [ "$$id", "$sales.persons_related.id" ] } }
Doesn't work as expected, I have tried with other operators ($eq) with same result.
The expected output (using a pipeline in the lookup) should be:
[
{
"_id": ObjectId("5a934e000102030405000000"),
"customer": {
"id": 100,
"name": "a"
},
"sales": [
{
"sale_id": 9999,
"persons_related": [
{"id": 100},
{"id": 900},
{"id": 800}
]
}
]
}
]
Can you please lend me a hand? You can test on this mongo playground
Just FYI mongoplayground provides a multiple collection option, so instead of trying to hack your syntax to simulate it you can just us the dropdown at the top right side to change it.
Your syntax is fine, here is a working playground example:
db.customers.aggregate([
{
$match: {
id: 100
}
},
{
$lookup: {
from: "sales",
let: {
id: "$id"
},
pipeline: [
{
$match: {
$and: [
{
$expr: {
$in: [
"$$id",
"$persons_related.id"
]
}
}
]
}
}
],
as: "sales"
}
}
])
$lookup also flattens arrays so you can just use the simpler syntax for it and it simplifies the code:
db.customers.aggregate([
{
$match: {
id: 100
}
},
{
$lookup: {
from: "sales",
localField: "id",
foreignField: "persons_related.id",
as: "sales"
}
}
])
Mongo Playground
I want to filter the dataset to extract documents which were created 7 days ago OR a Month ago OR Documents created at any date.
filter documents based on createdAt field in document.
Dataset:-
[
{
"_id": ObjectId("6257047cffd61ab62864c1ae"),
"type": "A",
"source": "B",
"user": ObjectId("622b55ff0b0af6b049c387d3"),
"createdAt": ISODate("2022-04-17T07:55:00.368Z"),
"updatedAt": ISODate("2022-04-17T07:55:00.368Z"),
},
{
"_id": ObjectId("6257047cffd61ab62864c1ad"),
"type": "B",
"source": "A",
"user": ObjectId("622b55ff0b0af6b049c387d3"),
"createdAt": ISODate("2022-04-23T07:55:00.368Z"),
"updatedAt": ISODate("2022-04-23T07:55:00.368Z"),
},
{
"_id": ObjectId("6257047cffd61ab62864c1ce"),
"type": "A",
"source": "C",
"user": ObjectId("622b55ff0b0af6b049c387d3"),
"createdAt": ISODate("2022-04-17T07:55:00.368Z"),
"updatedAt": ISODate("2022-04-17T07:55:00.368Z"),
},
{
"_id": ObjectId("6257047cffd61ab62864c1cb"),
"type": "A",
"source": "B",
"user": ObjectId("622b56250b0af6b049c387d6"),
"createdAt": ISODate("2022-04-24T07:55:00.368Z"),
"updatedAt": ISODate("2022-04-24T07:55:00.368Z"),
},
{
"_id": ObjectId("6257047cffd61ab62864c1cb"),
"type": "A",
"source": "B",
"user": ObjectId("622b56250b0af6b049c387d6"),
"createdAt": ISODate("2022-03-24T07:55:00.368Z"),
"updatedAt": ISODate("2022-03-24T07:55:00.368Z"),
},
{
"_id": ObjectId("6257047cffd61ab62864c1ce"),
"type": "A",
"source": "C",
"user": ObjectId("622b55ff0b0af6b049c387d3"),
"createdAt": ISODate("2022-03-17T07:55:00.368Z"),
"updatedAt": ISODate("2022-03-17T07:55:00.368Z"),
},
]
MongoDB aggregate query:-
db.collection.aggregate([
{
$addFields: {
paramType: "All",
paramSource: "All",
paramCreatedAt:"All",
}
},
{
$match: {
$and: [
{
user: ObjectId("622b55ff0b0af6b049c387d3")
},
{
$or: [
{
paramType: {
$eq: "All"
}
},
{
$expr: {
$eq: [
"$paramType",
"$type"
],
}
}
]
},
{
$or: [
{
paramSource: {
$eq: "All"
}
},
{
$expr: {
$eq: [
"$paramSource",
"$source"
]
}
}
]
}
]
}
},
{
$setWindowFields: {
output: {
totalCount: {
$count: {}
}
}
}
},
{
$sort: {
createdAt: -1
}
},
{
$skip: 0
},
{
$limit: 6
},
{
"$project": {
"paramSource": false,
"paramType": false,
}
}
])
how to filter to get documents created in the last 7 days or 30 days or any date.
paramCreatedAt will take one of the following values [All dates, 7 days ago, a month ago]
Example:-
If the All dates filter is applied then display all records.
If 7 days filter is applied display records created from the current date (which can be any day not necessary that it should be sunday) to 7 days back.
If 30 days filter applied then display records created in last 30 days
Your skeleton is pretty neat and you are actually quite close. For the date filtering, just use $dateDiff to return the date difference in days and compare it with the days interval your selected(i.e. 7 days or 30 days) by using $switch
db.collection.aggregate([
{
$addFields: {
paramType: "All",
paramSource: "All",
paramCreatedAt: "All dates"// [All dates, 7 days ago, a month ago]
}
},
{
$match: {
$and: [
{
user: ObjectId("622b55ff0b0af6b049c387d3")
},
{
$or: [
{
paramType: {
$eq: "All"
}
},
{
$expr: {
$eq: [
"$paramType",
"$type"
],
}
}
]
},
{
$or: [
{
paramSource: {
$eq: "All"
}
},
{
$expr: {
$eq: [
"$paramSource",
"$source"
]
}
}
]
},
{
$or: [
{
paramCreatedAt: {
$eq: "All dates"
}
},
{
$expr: {
$and: [
{
"$in": [
"$paramCreatedAt",
[
"7 days ago",
"a month ago"
]
]
},
{
$lte: [
{
"$dateDiff": {
"startDate": "$createdAt",
"endDate": "$$NOW",
"unit": "day"
}
},
{
"$switch": {
"branches": [
{
"case": {
$eq: [
"$paramCreatedAt",
"7 days ago"
]
},
"then": 7
},
{
"case": {
$eq: [
"$paramCreatedAt",
"a month ago"
]
},
"then": 30
}
]
}
}
]
}
]
}
}
]
}
]
}
},
{
$setWindowFields: {
output: {
totalCount: {
$count: {}
}
}
}
},
{
$sort: {
createdAt: -1
}
},
{
$skip: 0
},
{
$limit: 6
},
{
"$project": {
"paramSource": false,
"paramType": false,
}
}
])
Here is the Mongo playground for your reference.
Here's an alternate approach using $facet. $facet is very handy because it allows you to "match and group in parallel" and create overlapping buckets of documents. A single pipeline with $group and $cond on the aggregation field works well for "if/then/elif/elif/else" constructions where overlaps are not desired and an order of precedence is desired.
db.foo.aggregate([
// Initial filter(s):
{$match: {user: ObjectId("622b55ff0b0af6b049c387d3")}},
// Create a single version of "now" from the perspective of the
// CLIENT to use in queries to follow.
// To create such a target date from the perspective of the SERVER,
// use {$addFields: {DD: '$$NOW'}}
// Probably overkill but OK.
{$addFields: {DD: new ISODate()}},
{$facet: {
"all": [ ], // not exciting! :-)
"exactly_7_days_ago": [
{$match: {$expr:
{$eq: [7, {$floor: {$divide:[{$subtract:['$DD', '$createdAt'] }, 1000 * 60 * 60 * 24]}} ]}
}}
],
"everything_from_last_month": [
{$match: {$expr:
{$eq: [1, {$subtract:[{$month: '$DD'}, {$month: '$createdAt'} ]} ]}
}}
],
"only_one_day_from_last_month": [
{$match: {$expr:
{$and: [
{$eq: [1, {$subtract:[{$month: '$DD'}, {$month: '$createdAt'}]} ]},
{$eq: [0, {$subtract:[{$dayOfMonth: '$DD'}, {$dayOfMonth: '$createdAt'} ]} ]}
]}
}}
],
}}
]);
I have two collections properties and property_prices and the relation is one property many prices. So I am trying to join them and then find min value from property_prices.monthly_unit_price.unit_price. So I could get the Properties with their prices and min unit_price value from entire property pricing.
Property Collection
{
"_id": "1",
"status": "Approved",
"name": "My Property Lake"
}
Property Price Collection where monthly_unit_price have objects from Jan - Dec
{
"property_prices": [
{
"property_id": "1",
"block_id": "ABC",
"monthly_unit_price": [{ "month": "Jan", "unit_price": 100 }, { "month": "Dec", "unit_price": "1200" }],
},
{
"property_id": "1",
"block_id": "DEF",
"monthly_unit_price": [{ "month": "Jan", "unit_price": "200" }, { "month": "Dec", "unit_price": "2400" }],
}
]
}
Basically I want to get the min value from property_prices unit_price for property_id 1
So I tried using aggregate and lookup but I cant get the min value for entire property from property_prices.
Here is what I tried
await Property.aggregate([
{
$lookup: {
from: 'property_prices',
as: 'property_prices',
let: { property_id: '$_id' },
pipeline: [
{
$match: {
$expr: {
$and: [
{ $eq: ['$property_id', '$$property_id'] },
{ $eq: ['$status', 'Completed'] },
]
}
}
},
]
},
},
{
$unwind: "$property_prices"
},
{
$group: {
_id: '$property_prices.property_id',
minInvestment: { "$min": "$property_prices.monthly_unit_price.unit_price" }
}
},
]);
Result I am expecting is
{
"_id": "1",
"status": "Approved",
"name": "My Property Lake",
"property_prices": [
{
"property_id": "1",
"block_id": "ABC",
"monthly_unit_price": [{ "month": "Jan", "unit_price": 100 }, { "month": "Dec", "unit_price": "1200" }],
},
{
"property_id": "1",
"block_id": "DEF",
"monthly_unit_price": [{ "month": "Jan", "unit_price": "200" }, { "month": "Dec", "unit_price": "2400" }],
}
],
"minInvestment":100
}
You are on the right track, you just need to "massage" the document structure a little bit more due to the fact it's a nested array. here is a quick example of doing so using the $map and $reduce operators.
Notice I also had to cast the values to number type using $toInt, I recommend these sort of things to be handled at update/insertion time instead.
db.properties.aggregate([
{
$lookup: {
from: "property_prices",
as: "property_prices",
let: {
property_id: "$_id"
},
pipeline: [
{
$match: {
$expr: {
$and: [
{
$eq: [
"$property_id",
"$$property_id"
]
},
{
$eq: [
"$status",
"Completed"
]
}
]
}
}
}
]
}
},
{
$addFields: {
minInvestment: {
$min: {
$reduce: {
input: {
$map: {
input: "$property_prices",
as: "property",
in: {
$map: {
input: "$$property.monthly_unit_price",
as: "price",
in: {
$toInt: "$$price.unit_price"
}
}
}
}
},
initialValue: [],
in: {
"$concatArrays": [
"$$value",
"$$this"
]
}
}
}
}
}
}
])
Mongo Playground
I have three documents. First one is percentages, the others discardeditems and filtereditems. Sample data of these documents are as follows.
db = {
"percentages": [
{
"_id": 1,
"base": "A",
"buy": "BUY_1",
"sell": "SELL_1",
"item": "ITEM_B",
"ask": 100,
"bid": 114,
"percentage": 14
},
{
"_id": 2,
"base": "B",
"buy": "BUY_2",
"sell": "SELL_2",
"item": "ITEM_G",
"ask": 50,
"bid": 90,
"percentage": 80
},
{
"_id": 3,
"base": "A",
"buy": "BUY_2",
"sell": "SELL_2",
"item": "ITEM_G",
"ask": 10,
"bid": 15,
"percentage": 50
}
],
"discardeditems": [
{
"_id": 1,
"buy": "BUY_1",
"sell": "SELL_1",
"item": "ITEM_B"
},
{
"_id": 2,
"buy": "BUY_2",
"sell": "SELL_2",
"item": "ITEM_G"
}
],
"filtereditems": [
{
"_id": 2,
"buy": "BUY_2",
"sell": "SELL_2",
"item": "ITEM_G",
"percentage": "55"
}
]
}
Actually, I want to compare percentages document with discardeditems and filtereditems documents. If the buy, sell and item values in percentages document are equal to those in discardeditems document, I would like to add isdiscardeditem:true to percentages document.
And if the buy, sell, and item values in filtereditems are equal to those in percentages document and the percentage value in filtereditems document is greater than in percentages document, I no longer want to show this record.
The final version of the data I want to see should be as follows;
{
"_id": 1,
"base": "A",
"buy": "BUY_1",
"sell": "SELL_1",
"item": "ITEM_B",
"ask": 100,
"bid": 114,
"percentage": 14,
"isdiscarded": true
},
{
"_id": 2,
"base": "B",
"buy": "BUY_2",
"sell": "SELL_2",
"item": "ITEM_G",
"ask": 50,
"bid": 90,
"percentage": 80,
"isdiscarded": true
}
Percentages document count was three. But now i want to show two record. The other record of percentages document must not come because of the less than the percentage in filtereditems.
I can add isdiscardeditem using $lookup, $match $addFields keyword but I have not been successful in showing two record. https://mongoplayground.net/p/_XZoqGTnIyz
How can I write?
You can try below aggregation:
db.percentages.aggregate([
{
$lookup: {
from: "discardeditems",
let: {
item_src: "$item",
buy_src: "$buy",
sell_src: "$sell"
},
pipeline: [
{
$match: {
$expr: {
$and: [
{ $eq: [ "$$item_src", "$item" ] },
{ $eq: [ "$$buy_src", "$buy" ] },
{ $eq: [ "$$sell_src", "$sell" ] },
]
}
}
}
],
as: "discarded"
}
},
{
$lookup: {
from: "filtereditems",
let: {
item_src: "$item",
buy_src: "$buy",
sell_src: "$sell",
percentage_src: "$percentage"
},
pipeline: [
{
$match: {
$expr: {
$and: [
{ $eq: [ "$$item_src", "$item" ] },
{ $eq: [ "$$buy_src", "$buy" ] },
{ $eq: [ "$$sell_src", "$sell" ] },
{ $lt: [ "$$percentage_src", { $toInt: "$percentage" } ] }
]
}
}
}
],
as: "filtered"
}
},
{
$match: {
filtered: { $eq: [] }
}
},
{
$addFields: {
isdiscarded: { $gt: [ { $size: "$discarded" }, 0 ] }
}
},
{
$project: {
discarded: 0,
filtered: 0
}
}
])
Please note that percentage fields have to have the same type so $toInt is needed for conversion.
Mongo Playground
I have been searching on stackoverflow and cannot find exactly what I am looking for and hope someone can help. I want to submit a single query, get multiple counts back, for a single document, based on array of that document.
My data:
db.myCollection.InsertOne({
"_id": "1",
"age": 30,
"items": [
{
"id": "1",
"isSuccessful": true,
"name": null
},{
"id": "2",
"isSuccessful": true,
"name": null
},{
"id": "3",
"isSuccessful": true,
"name": "Bob"
},{
"id": "4",
"isSuccessful": null,
"name": "Todd"
}
]
});
db.myCollection.InsertOne({
"_id": "2",
"age": 22,
"items": [
{
"id": "6",
"isSuccessful": true,
"name": "Jeff"
}
]
});
What I need back is the document and the counts associated to the items array for said document. In this example where the document _id = "1":
{
"_id": "1",
"age": 30,
{
"totalIsSuccessful" : 2,
"totalNotIsSuccessful": 1,
"totalSuccessfulNull": 1,
"totalNameNull": 2
}
}
I have found that I can get this in 4 queries using something like this below, but I would really like it to be one query.
db.test1.aggregate([
{ $match : { _id : "1" } },
{ "$project": {
"total": {
"$size": {
"$filter": {
"input": "$items",
"cond": { "$eq": [ "$$this.isSuccessful", true ] }
}
}
}
}}
])
Thanks in advance.
I am assuming your expected result is invalid since you have an object literal in the middle of another object and also you have totalIsSuccessful for id:1 as 2 where it seems they should be 3. With that said ...
you can get similar output via $unwind and then grouping with $sum and $cond:
db.collection.aggregate([
{ $match: { _id: "1" } },
{ $unwind: "$items" },
{ $group: {
_id: "_id",
age: { $first: "$age" },
totalIsSuccessful: { $sum: { $cond: [{ "$eq": [ "$items.isSuccessful", true ] }, 1, 0 ] } },
totalNotIsSuccessful: { $sum: { $cond: [{ "$ne": [ "$items.isSuccessful", true ] }, 1, 0 ] } },
totalSuccessfulNull: { $sum: { $cond: [{ "$eq": [ "$items.isSuccessful", null ] }, 1, 0 ] } },
totalNameNull: { $sum: { $cond: [ { "$eq": [ "$items.name", null ]}, 1, 0] } } }
}
])
The output would be this:
[
{
"_id": "_id",
"age": 30,
"totalIsSuccessful": 3,
"totalNameNull": 2,
"totalNotIsSuccessful": 1,
"totalSuccessfulNull": 1
}
]
You can see it working here