Why mongodb upsert counter doesn't reflect actual changes? - mongodb

Consider following query
var collectionName = "test";
db.createCollection(collectionName);
db.getCollection(collectionName).insert({
"_id" : 1, "char" : "Gandalf", "class" : "barbarian", "lvl" : 20
});
db.getCollection(collectionName).bulkWrite([
{
insertOne: {
"document": {
"_id" : 2, "char" : "Dithras", "class" : "barbarian", "lvl" : 4
}
}
},
{
updateOne: {
filter: { "_id": 1},
update: { $set: {"class" : "mage"} },
upsert: true
}
}
])
Which results in:
{
"acknowledged" : true,
"deletedCount" : 0.0,
"insertedCount" : 1.0,
"matchedCount" : 1.0,
"upsertedCount" : 0.0,
"insertedIds" : {
"0" : 2.0
},
"upsertedIds" : {}
}
my question is why updating of a document with id:1 doesn't get into upsertedIds? Isn't this document just being updated with upsert? Or am I missing anything?
According to the documentation it only adds information to upsert if it doesn't find any document (so it's actually more like inserted), but this case I don't know which items got updated.
Is it possible to get which documents got modified when executing a query?
To avoid XY problem: I want to see bulk operation items taht failed (e.g. when trying to update non-existing document with upsert:false) and log IDs that triggered the failures.

You can collect document ids when you prepare your bulk update, then check which of them don't exist after the update. E.g. for document ids 1, 2, and 3 that you updated with upsert:false
db.test.aggregate([
{ $group: { _id: null }},
{ $project: { _id: [ 1, 2, 3 ] }},
{ $unwind: "$_id" },
{ $lookup: {
from: "test",
localField: "_id",
foreignField: "_id",
as: "exists"
}},
{ $match: { exists: { $size:0 }}},
{ $project: { exists:0 }}
])
will return id's of filters that didn't match any document, i.e. "failed" in your terminology.

Related

MongoDb how change id of document to diffrent

Hello how to change an id of document to diffrent when some ids are equal?
I want to change some of employees id_director to object id thats is equal to id_director. I mean for example, when some document have id_director = 100 then changed it to _id with value of employee where id_employee = 100.
I was trying like this:
var employes = db.employees.find({"id_director": {$ne: null}});
while (employes.hasNext()) {
emp = employes.next();
employe = db.employees.findOne({"id_director":emp.id_employe});
emp.id_director = employe._id
db.employees.save(emp)
}
For example
I have two documents in one collection:
employees
{
"_id" : ObjectId("6224a5767b9cdbdcb681b1ef"),
"id_employe" : 180,
"id_director" : 100,
"name" : "Mark"
}
{
"_id" : ObjectId("6224a5767b9cdbdcb681b1f0"),
"id_employe" : 100,
"id_director" : null,
"name" : "Peter"
}
Expected document:
{
"_id" : ObjectId("6224a5767b9cdbdcb681b1ef"),
"id_employe" : 180,
"id_director" : ObjectId("6224a5767b9cdbdcb681b1f0"),
"name" : "Mark"
}
first of all you need to have peter's id then update mark document:
var peter= db.employees.findOne({_id: peterId});
db.employees.updateOne({
_id: markId
}, {
$set: {
id_director: peter._id
}
})
You can perform a self $lookup to lookup for the director_id, then perform some wrangling and $merge back to the collection.
db.collection.aggregate([
{
"$match": {
"id_director": {
$ne: null
}
}
},
{
"$lookup": {
"from": "collection",
"localField": "id_director",
"foreignField": "id_employe",
pipeline: [
{
"$limit": 1
},
{
$project: {
_id: 1
}
}
],
"as": "id_director"
}
},
{
$unwind: "$id_director"
},
{
$project: {
_id: 1,
id_employe: 1,
id_director: "$id_director._id",
name: 1
}
},
{
"$merge": {
"into": "collection",
"on": "_id",
"whenMatched": "replace"
}
}
])
Here is the Mongo playground for your reference.

MongoDB aggregate return count of documents or 0

I have the following aggregate query:
db.user.aggregate()
.match({"business_account_id" : ObjectId("5e3377bcb1dbae5124e4b6bf")})
.lookup({
'localField': 'profile_id',
'from': 'profile',
'foreignField' : '_id',
'as': 'profile'
})
.unwind("$profile")
.match({"profile.type" : "consultant"})
.group({_id:"$business_account_id", count:{$sum:1}})
My goal is to count how many consultant users belong to a given company.
Using the query above, if there is at least one user belonging to the provided business_account_id I get a correct count value.
But if there are none users, the .match({"business_account_id" : ObjectId("5e3377bcb1dbae5124e4b6bf")}) will return an empty (0 documents) result.
How can I get a count: 0 if the there are no users assigned to the company ?
I tried many approach based on other threads but I coundn't get a count: 0
UPDATE 1
A simple version of my problem:
user collection
{
"_id" : ObjectId("5e36beb7b1dbae5124e4b6dc"),
"business_account_id" : ObjectId("5e3377bcb1dbae5124e4b6bf"),
},
{
"_id" : ObjectId("5e36d83db1dbae5124e4b732"),
"business_account_id" : ObjectId("5e3377bcb1dbae5124e4b6bf"),
}
Using the following aggregate query:
db.getCollection("user").aggregate([
{ "$match" : {
"business_account_id" : ObjectId("5e3377bcb1dbae5124e4b6bf")
}
},
{ "$group" : {
"_id" : "$business_account_id",
"count" : { "$sum" : 1 }
}
}
]);
I get:
{
"_id" : ObjectId("5e3377bcb1dbae5124e4b6bf"),
"count" : 2
}
But if I query for an ObjectId that doesn't exist, such as:
db.getCollection("user").aggregate([
{ "$match" : {
"business_account_id" : ObjectId("5e335c873e8d40676928656d")
}
},
{ "$group" : {
"_id" : "$business_account_id",
"count" : { "$sum" : 1 }
}
}
]);
I get an result completely empty. I would expect to get:
{
"_id" : ObjectId("5e335c873e8d40676928656d"),
"count" : 0
}
The root of the problem is if there is no document in the user collection that satisfies the initial $match there is nothing to pass to the next stage of the pipeline. If the business_account_id actually exists somewhere (perhaps another collection?) run the aggregation against that collection so that the initial match finds at least one document. Then use $lookup to find the users. If you are using MongoDB 3.6+, you can might combine the user and profile lookups. Lastly, use $size to count the elements in the users array.
(You will probably need to tweak the collection and field names)
db.businesses.aggregate([
{$match:{_id : ObjectId("5e3377bcb1dbae5124e4b6bf")}},
{$project: { _id:1 }},
{$lookup:{
from: "users",
let: {"busId":"$_id"},
as: "users",
pipeline: [
{$match: {$expr:{$eq:[
"$$busId",
"$business_account_id"
]}}},
{$lookup:{
localField: "profile_id",
from: "profile",
foreignField : "_id",
as: "profile"
}},
{$match: { "profile.type" : "consultant"}}
]
}},
{$project: {
_id: 0,
business_account_id: "$_id",
count:{$size:"$users"}
}}
])
Playground
Since you match non-existing business_account_id value, aggregation process will stop.
Workaround: We perform 2 aggregations in parallel with $facet operator to get default value if matching has no result.
Note: Make sure user collection has at least 1 record, otherwise this won't work
db.user.aggregate([
{
$facet: {
not_found: [
{
$project: {
"_id": ObjectId("5e3377bcb1dbae5124e4b6bf"),
"count": { $const: 0 }
}
},
{
$limit: 1
}
],
found: [
{
"$match": {
"business_account_id": ObjectId("5e3377bcb1dbae5124e4b6bf")
}
},
{
"$group": {
"_id": "$business_account_id",
"count": { "$sum": 1 }
}
}
]
}
},
{
$replaceRoot: {
newRoot: {
$mergeObjects: [
{
$arrayElemAt: ["$not_found", 0]
},
{
$arrayElemAt: ["$found", 0]
}
]
}
}
}
])
MongoPlayground

Mongo Aggregation does not analyze all documents

I'm trying to get some statistics using Mongo's aggregation Framework but my queries do not seem to work right. So I have a collection of documents, each document having the structure below:
{
"_id" : ObjectId("5e46af306d0f5d63d4de6d0f"),
"homeEmperor" : {
"name" : "Home",
"units" : {
"Mage" : 15,
"Warrior" : 15,
"Swordmaster" : 15,
"Rogue" : 15,
"Warlock" : 15
}
},
"awayEmperor" : {
"name" : "Away",
"units" : {
"Druid" : 15,
"Ranger" : 15,
"Priest" : 15,
"Monk" : 15,
"Dragon" : 15
}
},
"dateCreated" : ISODate("2020-02-06T00:00:00.000Z"),
"winner" : "away",
"battleType" : "mutual",
"turns" : 45,
"_class" : "com.deathstar.Datahouse.domain.mongo.HistoricMongoRecord"
}
So what I'm doing at the moment is using this query to get the number of wins each unit has
db.getCollection('WarHistory').aggregate([
{ $match: { battleType: "mutual" } },
{ $project: {"winners": {$cond: { if: { $eq: [ "$winner", "home" ] }, then: "$homeEmperor.units", else: "$awayEmperor.units"} }} },
{ $project: {"winnerUnits": { $objectToArray: "$winners" }} },
{ $group: { _id: {unitType: "$winnerUnits.k"}} },
{ $unwind: "$_id.unitType" },
{ $group: { _id: {unitType: "$_id.unitType"}, count:{$sum:1} }},
{ $sort : { count : -1 }}
])
and this query to get each unit's participation
db.getCollection('WarHistory').aggregate([
{ $match: { battleType: "mutual" } },
{ $project: { "participants": { $mergeObjects: [ "$homeEmperor.units", "$awayEmperor.units" ] } } },
{ $project: {"participantUnits": { $objectToArray: "$participants" }} },
{ $group: { _id: {unitType: "$participantUnits.k"}} },
{ $unwind: "$_id.unitType" },
{ $group: { _id: {unitType: "$_id.unitType"}, count:{$sum:1} }},
{ $sort : { count : -1 }}
])
The output of both queries is as shown below which is the way I want it:
{
"_id" : {
"unitType" : "Pirate"
},
"count" : 1331.0
}
After that I divide them and so on. I also have a Java service which produces these documents.
The problem is that while at first the queries seem to work fine, after some point they seem to stop counting all documents, so I see the "Count" counter in the Collection increasing but the query results remain exactly the same.
I've checked and the documents continue to have the same structure along the way.
I also tried the "allowDiskUse:true" parameter in case the problem was the memory capacity(I have 16GB RAM), but it really made no difference.
Can anyone please confirm that the queries above can do what I want them to?
Any help would be really appreciated! Thanks for your time!

Lookup and sort the foreign collection

so I have a collection users, and each document in this collection, as well as other properties, has an array of ids of documents in the other collection: workouts.
Every document in the collection workouts has a property named date.
And here's what I want to get:
For a specific user, I want to get an array of {workoutId, workoutDate} for the workouts that belong to that user, sorted by date.
This is my attempt, which is working fine.
Users.aggregate([
{
$match : {
_id : ObjectId("whateverTheUserIdIs")
}
},
{
$unwind : {
path : "$workouts"
}
}, {
$lookup : {
from : "workouts",
localField : "workouts",
foreignField : "_id",
as : "workoutDocumentsArray"
}
}, {
$project : {
_id : false,
workoutData : {
$arrayElemAt : [
$workoutDocumentsArray,
0
]
}
}
}, {
$project : {
date : "$workoutData.date",
id : "$workoutData._id"
}
}, {
$sort : {date : -1}
}
])
However I refuse to believe I need all this for what would be such a simple query in SQL!? I believe I must at least be able to merge the two $project stages into one? But I've not been able to figure out how looking at the docs.
Thanks in advance for taking the time! ;)
====
EDIT - This is some sample data
Collection users:
[{
_id:xxx,
workouts: [2,4,6]
},{
_id: yyy,
workouts: [1,3,5]
}]
Colleciton workouts:
[{
_id:1,
date: 1/1/1901
},{
_id:2,
date: 2/2/1902
},{
_id:3,
date: 3/3/1903
},{
_id:4,
date: 4/4/1904
},{
_id:5,
date: 5/5/1905
},{
_id:6,
date: 6/6/1906
}]
And after running my query, for example for user xxx, I would like to get only the workouts that belong to him (whose ids appear in his workouts array), so the result I want would look like:
[{
id:6,
date: 6/6/1906
},{
id:4,
date: 4/4/1904
},{
id:2,
date: 2/2/1902
}]
You don't need to $unwind the workouts array as it already contains array of _ids and use $replaceRoot instead of doing $project
Users.aggregate([
{ "$match": { "_id" : ObjectId("whateverTheUserIdIs") }},
{ "$lookup": {
"from" : "workouts",
"localField" : "workouts",
"foreignField" : "_id",
"as" : "workoutDocumentsArray"
}},
{ "$unwind": "$workoutDocumentsArray" },
{ "$replaceRoot": { "newRoot": "$workoutDocumentsArray" }}
{ "$sort" : { "date" : -1 }}
])
or even with new $lookup syntax
Users.aggregate([
{ "$match" : { "_id": ObjectId("whateverTheUserIdIs") }},
{ "$lookup" : {
"from" : "workouts",
"let": { "workouts": "$workouts" },
"pipeline": [
{ "$match": { "$expr": { "$in": ["$_id", "$$workouts"] }}},
{ "$sort" : { "date" : -1 }}
]
"as" : "workoutDocumentsArray"
}},
{ "$unwind": "$workoutDocumentsArray" },
{ "$replaceRoot": { "newRoot": "$workoutDocumentsArray" }}
])

Issues with merging arrays of objects in MoongoDb

I trying to build an aggregation quarry in MoongoDb that will merge arrays from
2 different collection (one of the collections is of type TTL). And I facing with 2 issues that I can’t resolve.
First Issue:
I would like to merge the TakenSeats fields of my temp collations and permanent collection and set the result instead of my correct TakenSeats field, Using my aggregation in the bottom i manage to merge the arrays with the $push operator, But I cant replace the result field with the TakenSeats field that is in my permanent document.
Second Issue:
In case that I don’t have any documents in my temp collection, how can I still receive the document from the permanent one?
Sample of document in the permanent collection: (extracting data from one document)
{
"_id" : ObjectId("5b6b656818883ec018d1542d"),
"showsHall" : [
ObjectId("5b64cb758ad5f81a6cb7e6ae")
],
"movie" : [
ObjectId("5b6b614218883ec018d15428")
],
"takenSeats" : [
{
"id" : 11
},
{
"id" : 12
}
],
"showDate" : "8/14/2018",
"showStartTime" : "3:00 PM",
"showEndTime" : "5:00 PM",
"creteDate" : ISODate("2018-08-08T21:49:28.020Z"),
"__v" : 0
}
From the TTL collection: (extracting data from multiple documents)
{
"_id" : ObjectId("5b6f35023f64851baa70c61b"),
"createdAt" : ISODate("2018-08-11T19:12:02.951Z"),
"showId" : [
ObjectId("5b6b656818883ec018d1542d")
],
"takenSeats" : [
{
"id" : 22
},
{
"id" : 25
}
]
}
This is the aggregation that I used:
db.getCollection('shows').aggregate([
{ $match: { _id: ObjectId("5b6b656818883ec018d1542d") } },
{
$lookup: {
from: "temp",
localField: "_id",
foreignField: "showId",
as: "fromItems"
}
},
{ $unwind: "$fromItems" },
{ "$project": {"takenSeats": { "$setUnion": ["$takenSeats", "$fromItems.takenSeats"]}, _id: 1, showsHall: 1, movie: 1, takenSeats: 1 , showDate: 1, showStartTime: 1, showEndTime: 1 }},
{$unwind:"$takenSeats"},
{$group:{_id: "$_id", takenSeats: {$push : "$takenSeats"} }},
])
Result:
[Edit]
I manage to maintain my original data with $first operator.
But now i cant resolve issue no 2 (prevent result if null), I tried to use preserveNullAndEmptyArrays
in both of the unwind stages but the result is that it pushes an empty array.
My wanted result is that it should push to a new array only if there is values to push
This is my aggregation :
db.getCollection('shows').aggregate([
{ $match: { _id: ObjectId("5b6b656818883ec018d1542d") } },
{
$lookup: {
from: "temp",
localField: "_id",
foreignField: "showId",
as: "fromItems"
}
},
{ $unwind:{path:"$fromItems" ,preserveNullAndEmptyArrays:true}},
{ "$project": {"takenSeats": { "$setUnion": ["$takenSeats", "$fromItems.takenSeats"]}, _id: 1, showsHall: 1, movie: 1, showDate: 1, showStartTime: 1, showEndTime: 1 }},
{$unwind:{path:"$takenSeats" ,preserveNullAndEmptyArrays:true}},
,
{$group:{
_id: "$_id",
showsHall : { $first: '$showsHall' },
movie : { $first: '$movie' },
showDate : { $first: '$showDate' },
showStartTime : { $first: '$showStartTime' },
showEndTime : { $first: '$showEndTime' },
takenSeats: {$push : "$takenSeats"}
}
}
])
This is the result that i getting if there is no documents in the temp collection
{
"_id" : ObjectId("5b6b656818883ec018d1542d"),
"showsHall" : [
ObjectId("5b64cb758ad5f81a6cb7e6ae")
],
"movie" : [
ObjectId("5b6b614218883ec018d15428")
],
"showDate" : "8/14/2018",
"showStartTime" : "3:00 PM",
"showEndTime" : "5:00 PM",
"takenSeats" : [
null
]
}
Here Please add ifNull Condition for solution 2
db.getCollection('shows').aggregate([
{ $match: { _id: ObjectId("5b6b656818883ec018d1542d") } },
{
$lookup: {
from: "tempShows",
localField: "_id",
foreignField: "showId",
as: "fromItems"
}
},
{ $unwind:{path:"$fromItems" ,preserveNullAndEmptyArrays:true}},
{ "$project": {"takenSeats": { $ifNull: [{ "$setUnion": ["$takenSeats", "$fromItems.takenSeats"]}, '$takenSeats'] } ,_id: 1, showsHall: 1, movie: 1, showDate: 1, showStartTime: 1, showEndTime: 1 }},
{$unwind:{path:"$takenSeats" ,preserveNullAndEmptyArrays:true}},
{$group:{
_id: "$_id",
showsHall : { $first: '$showsHall' },
movie : { $first: '$movie' },
showDate : { $first: '$showDate' },
showStartTime : { $first: '$showStartTime' },
showEndTime : { $first: '$showEndTime' },
takenSeats: {$push : "$takenSeats"}
}
}
])