MongoDB - Get IDs of inserted and existing documents after "Insert if not exist" operation on multiple documents - mongodb

I have to insert multiple documents if they don't already exist, but the important thing is that in the query results I need to have IDs of both the inserted and already existing items.
I'm trying with the following bulkWrite operation:
// external_id is a unique id other than the mongo _id
let items = [
{external_id: 123, name: "John"},
{external_id: 456, name: "Mike"},
{external_id: 789, name: "Joseph"}
];
db.collection("my_collection")
.bulkWrite(
items.map((item) => {
return {
updateOne: {
filter: { external_id: item.external_id },
update: { $setOnInsert: item},
upsert: true,
},
};
})
);
The problem is that the BulkWriteResult return only the _id of the inserted items in upsertedIds, while for the existing items return only the nMatched number.
The other solution I have think about is to make (1) a find over an array of ids, (2) check the results for the ones already existing, and (3) then insertMany for the new ones:
let ids = [123, 456, 789];
let items = [
{external_id: 123, name: "John"},
{external_id: 456, name: "Mike"},
{external_id: 789, name: "Joseph"}
];
// STEP 1: Find alredy existings items
db.collection("my_collection")
.find({ external_id: { $in: ids } })
.toArray(function (err, existingItems) {
// If John already exist
// existingItems = [{_id: ObjectId, external_id: 123, name: "John"}]
// STEP 2: Check which item has to be created
let itemsToBeCreated = items.filter((item) =>
!existingItems.some((ex) => ex.external_id === item.external_id)
);
// STEP 3: Insert new items
db.collection("my_collection")
.insertMany(itemsToBeCreated, function (err, result) {
// FINALLY HERE I GET ALL THE IDs OF THE EXISTING AND INSERTED ITEMS
});
});
With this solution I'm concerned about performance, because these operations are fired 100K times a day for 10 items each, and about 90% of the times the items are new. So 900K new items and 100K already existing.
I would like to know if there is a better way of achieving this.
Thanks in advance

Related

Indexing has no effect on db.find()

I've Just started to work with MongoDB, so you might find my question really stupid.I tried to search a lot before posting my query here, Any Help would be Appreciated.
I also came across this link StackOverFlow Link, which advised to apply .sort() on every query, but that would increase the query time.
So I tried to index my collection using .createIndexes({_id:-1}), to sort data in descending order of creation time(newest to oldest), After that when I used the .find() method to get data in sorted format(newest to Oldest) I did'nt get the desired result , I still had to sort the data :( .
// connecting db
mongoose.connect(dbUrl, dbOptions);
const db = mongoose.connection;
// listener on db events
db.on('open', ()=>{console.log('DB SUCESSFULLY CONNECTED !!');});
db.on('error', console.error.bind(console, 'connection error:'));
// creating Schema for a person
const personSchma = new mongoose.Schema(
{ name: String,
age : Number}
)
// creating model from person Schema
const person = mongoose.model('person', personSchma);
// Chronological Order of Insertion Of Data
// {name: "kush", age:22}
// {name: "clutch", age:22}
// {name: "lauv", age:22}
person.createIndexes({_id:-1}, (err)=>{
if (err){
console.log(err);
}
})
person.find((err, persons)=>{
console.log(persons)
// Output
// [
// { _id: 6026eadd58a2b124d85b0f8d, name: 'kush', age: 22, __v: 0 },
// { _id: 6026facdf200f8261005f8e0, name: 'clutch', age: 22, __v: 0 },
// { _id: 6026facdf200f8261005f8e1, name: 'lauv', age: 22, __v: 0 }
// ]
})
person.find().sort({_id:-1}).lean().limit(100).then((persons)=>{
console.log(persons);
// Output
// [
// { _id: 6026facdf200f8261005f8e1, name: 'lauv', age: 22, __v: 0 },
// { _id: 6026facdf200f8261005f8e0, name: 'clutch', age: 22, __v: 0 },
// { _id: 6026eadd58a2b124d85b0f8d, name: 'kush', age: 22, __v: 0 }
// ]
})
Indexes are special data structure, which can be used to run the queries efficiently. While running the query, MongoDB tries to see which index should be used for running the query efficiently and then that index will be used.
Creating an index with {_id:-1} will create an auxiliary data structure(index) which will be sorted newest first. It doesn't affect the order of the data which we are storing.
To sort the data in descending order(newest first) we will have to explicitly add the sort operation in your query and make sure that an index for descending order _id is present.

using mongoose to update a specific sub doc property's value [duplicate]

Is there a way to update values in an object?
{
_id: 1,
name: 'John Smith',
items: [{
id: 1,
name: 'item 1',
value: 'one'
},{
id: 2,
name: 'item 2',
value: 'two'
}]
}
Lets say I want to update the name and value items for item where id = 2;
I have tried the following w/ mongoose:
var update = {name: 'updated item2', value: 'two updated'};
Person.update({'items.id': 2}, {'$set': {'items.$': update}}, function(err) { ...
Problem with this approach is that it updates/sets the entire object, therefore in this case I lose the id field.
Is there a better way in mongoose to set certain values in an array but leave other values alone?
I have also queried for just the Person:
Person.find({...}, function(err, person) {
person.items ..... // I might be able to search through all the items here and find item with id 2 then update the values I want and call person.save().
});
You're close; you should use dot notation in your use of the $ update operator to do that:
Person.update({'items.id': 2}, {'$set': {
'items.$.name': 'updated item2',
'items.$.value': 'two updated'
}}, function(err) { ...
model.update(
{ _id: 1, "items.id": "2" },
{
$set: {
"items.$.name": "yourValue",
"items.$.value": "yourvalue",
}
}
)
MongoDB Document
There is a mongoose way for doing it.
const itemId = 2;
const query = {
item._id: itemId
};
Person.findOne(query).then(doc => {
item = doc.items.id(itemId );
item["name"] = "new name";
item["value"] = "new value";
doc.save();
//sent respnse to client
}).catch(err => {
console.log('Oh! Dark')
});
There is one thing to remember, when you are searching the object in array on the basis of more than one condition then use $elemMatch
Person.update(
{
_id: 5,
grades: { $elemMatch: { grade: { $lte: 90 }, mean: { $gt: 80 } } }
},
{ $set: { "grades.$.std" : 6 } }
)
here is the docs
For each document, the update operator $set can set multiple values, so rather than replacing the entire object in the items array, you can set the name and value fields of the object individually.
{'$set': {'items.$.name': update.name , 'items.$.value': update.value}}
Below is an example of how to update the value in the array of objects more dynamically.
Person.findOneAndUpdate({_id: id},
{
"$set": {[`items.$[outer].${propertyName}`]: value}
},
{
"arrayFilters": [{ "outer.id": itemId }]
},
function(err, response) {
...
})
Note that by doing it that way, you would be able to update even deeper levels of the nested array by adding additional arrayFilters and positional operator like so:
"$set": {[`items.$[outer].innerItems.$[inner].${propertyName}`]: value}
"arrayFilters":[{ "outer.id": itemId },{ "inner.id": innerItemId }]
More usage can be found in the official docs.
cleaner solution using findOneAndUpdate
await Person.findOneAndUpdate(
{ _id: id, 'items.id': 2 },
{
$set: {
'items.$.name': 'updated item2',
'items.$.value': 'two updated',
}
},
);
In Mongoose, we can update array value using $set inside dot(.) notation to specific value in following way
db.collection.update({"_id": args._id, "viewData._id": widgetId}, {$set: {"viewData.$.widgetData": widgetDoc.widgetData}})
Having tried other solutions which worked fine, but the pitfall of their answers is that only fields already existing would update adding upsert to it would do nothing, so I came up with this.
Person.update({'items.id': 2}, {$set: {
'items': { "item1", "item2", "item3", "item4" } }, {upsert:
true })
I had similar issues. Here is the cleanest way to do it.
const personQuery = {
_id: 1
}
const itemID = 2;
Person.findOne(personQuery).then(item => {
const audioIndex = item.items.map(item => item.id).indexOf(itemID);
item.items[audioIndex].name = 'Name value';
item.save();
});
Found this solution using dot-object and it helped me.
import dot from "dot-object";
const user = await User.findByIdAndUpdate(id, { ...dot.dot(req.body) });
I needed to update an array element with dynamic key-value pairs.
By mapping the update object to new keys containing the $ update operator, I am no longer bound to know the updated keys of the array element and instead assemble a new update object on the fly.
update = {
name: "Andy",
newKey: "new value"
}
new_update = Object.fromEntries(
Object.entries(update).map(
([k, v], i) => ["my_array.$." + k, v]
)
)
console.log({
"$set": new_update
})
In mongoose we can update, like simple array
user.updateInfoByIndex(0,"test")
User.methods.updateInfoByIndex = function(index, info) ={
this.arrayField[index]=info
this.save()
}
update(
{_id: 1, 'items.id': 2},
{'$set': {'items.$[]': update}},
{new: true})
Here is the doc about $[]: https://docs.mongodb.com/manual/reference/operator/update/positional-all/#up.S[]

How to check whether each item in an array exists or not

I'm trying to create a watch list where users can watch items. I was trying to create it by adding a watchlist field to my users collection. The watchlist would be an array of IDs corresponding to other items.
Users Collection:
id: ObjectId
name: string
watchlist: array i.e. [9872, 342, 4545, 234, 8745]
The question I have is related to querying this structure. I want to be able to write a query where I pass in a user id and an array of ~20 IDs and check which of those IDs the user watches (i.e. which of them exists in the watchlist field for that user).
I tried this initially:
db.users.find({
_id: 507c35dd8fada716c89d0013,
watchlist: { $in: [342, 999, 8745, etc...] }
});
But this gives me the list of users that contain any of those watchlist items, which is not what I want. What I actually want is a response containing an array like this:
{
id: 342,
exists: true
},
{
id: 999,
exists: false
},
{
id: 8745,
exists: true
}
I'd even be ok just getting an array of items that match:
{
_id: 507c35dd8fada716c89d0013,
watching: [342, 8745]
}
Is this doable, or would I be better off moving the watchlist to a separate collection with users as an array? (My concern with the latter approach is that a user will only watch a few hundred items, but tens of thousands of users could potentially watch the same item.)
You can easily achieve the second output using $setIntersection operator.
db.users.aggregate(
[ {$match:{"_id": 507c35dd8fada716c89d0013}},
{ $project: { "watching": { $setIntersection: [ "$watchlist", [ 342, 999, 8745 ] ] } } }
]
)

MongoDB: Upsert document in array field

Suppose, I have the following database:
{
_id: 1,
name: 'Alice',
courses: [
{
_id: 'DB103',
credits: 6
},
{
_id: 'ML203',
credits: 4
}
]
},
{
_id: 2,
name: 'Bob',
courses: []
}
I now want to 'upsert' the document with the course id 'DB103' in both documents. Although the _id field should remain the same, the credits field value should change (i.e. to 4). In the first document, the respective field should be changed, in the second one, {_id: 'DB103', credits: 4} should be inserted into the courses array.
Is there any possibility in MongoDB to handle both cases?
Sure, I could search with $elemMatch in courses for 'DB103' and if I haven't found it, insert, otherwise update the value. But these are two steps and I would like to do both in just one.

MongoDB CoffeeScript find document with several fields

How can I check if there are any documents with both name: "John" and age: 40?
This doesn't seem to be working
db.Event.findOne name: "John", age: 40, (error, result) ->
unless result
db.Event.save {name: "John", age: 40}
# document not found, so add it
inserted = true
else
# document found
inserted = false
A callback to manage inserted boolean:
data =
'name': 'John'
'age': 40
inserted = false
db.Event.update data,
{ '$setOnInsert': data },
'upsert': true,
(error, result) ->
inserted = not error
Sounds like you need the $setOnInsert update operator which when used with the update flag { "upsert": true } within an atomic update() operation, the operation inserts a new document and $setOnInsert assigns the specified values to the fields in the document. If the update operation does not result in an insert, $setOnInsert does nothing:
JavaScript:
var data = {
"name": "John",
"age": 40
}
db.Event.update(
data,
{ "$setOnInsert": data },
{ "upsert": true }
)
CoffeeScript:
data =
'name': 'John'
'age': 40
db.Event.update data, { '$setOnInsert': data }, 'upsert': true