How to get a string in ObjectId in MongoDB v3.6? - mongodb

I have an aggregation query whose MongoDB repsonse is :
_id: ObjectId('5e822d6c87502b3a9b751786')
I would like to get the string inside the ObjectId which is 5e822d6c87502b3a9b751786.
[ Problem ]
I have searched this question but so far there are only three operators that are capable to do this, namely $toString, $toObjectId, and $convert :
$project: {
_id: {
$toString: "$_id"
}
}
$project: {
_id: {
$toObjectId: "$_id"
}
}
$project: {
_id: {
$convert: {
input: "$_id"
to: "string"
}
}
}
MongoDB v3.6 does not support them if I am not mistaken.
Is there any workaround in MongoDB v3.6 to get a string inside an ObjectId?
Any help is much appreciated :)

For MongoDB v3.6, as $toString and $convert is not available until v4.0, you may need to resort to JS/application-level access to the _id.
db.testCollection.insertMany([
{
"_id": ObjectId("5e822d6c87502b3a9b751786")
}]);
db.testCollection.find().forEach( doc => {
// put your logic for process here
console.log(JSON.stringify(doc._id))
});
output:
"5e822d6c87502b3a9b751786"

Just want to add ray's answer after some findings.
Having defined a model in the app, here are some workaround that worked for me:
Promises
const TestCollection = require('../models/testCollection');
function getStringInObjectId() {
TestCollection.find().then(t => {
t.forEach(doc => {
// Put some logic here...
console.log('string in ObjectId :', JSON.stringify(doc._id));
});
});
}
Async/Await
const TestCollection = require('../models/testCollection');
async function getStringInObjectId() {
const t = await TestCollection.find();
t.forEach(doc => {
// Put some logic here...
console.log('string in ObjectId :', JSON.stringify(doc._id));
});
}
Same goes to aggregation :
Promises
const TestCollection = require('../models/testCollection');
function getStringInObjectId() {
TestCollection.aggregate([
{ $match: { 'name': 'marc' } }, // <- { 'name': 'marc' } is just an example, feel free to change it
// Put some stages here if required...
]).then(t => {
console.log('string in ObjectId : ', t[0]._id.toString());
});
}
Async/Await
const TestCollection = require('../models/testCollection');
async function getStringInObjectId() {
const t = await TestCollection.aggregate([
{ $match: { 'name': 'marc' } }, // <- { 'name': 'marc' } is just an example, feel free to change it
// Put some stages here if required...
]);
console.log('string in ObjectId : ', t[0]._id.toString());
}
Either JSON.stringify() or toString() can be used to convert it to string. Feel free to correct variable names.

Related

How do you consistently migrate a large MongoDB collection?

I was trying to migrate a large MongoDB of ~600k documents, like so:
for await (const doc of db.collection('collection').find({
legacyProp: { $exists: true },
})) {
// additional data fetching from separate collections here
const newPropValue = await fetchNewPropValue(doc._id)
await db.collection('collection').findOneAndUpdate({ _id: doc._id }, [{ $set: { newProp: newPropValue } }, { $unset: ['legacyProp'] }])
}
}
When the migration script finished, data was still being updated for about 30 minutes or so. I've concluded this by computing document count of documents containing legacyProp property:
db.collection.countDocuments({ legacyProp: { $exists: true } })
which was decreasing on subsequent calls. After a while, the updates stopped and the final document count of documents containing legacy prop was around 300k, so the update failed silently resulting in a data loss. I'm curious what exactly happened, and most importantly, how do you update large MongoDB collections without any data loss? Keep in mind, there is additional data fetching involved before every update operation.
My first attempt would be to build function of fetchNewPropValue() in an aggregation pipeline.
Have a look at Aggregation Pipeline Operators
If this is not possible then you can try to put all newPropValue's into array and use it like this. 600k properties should fit easily into your RAM.
const newPropValues = await fetchNewPropValue() // getting all new properties as array [{_id: ..., val: ...}, {_id: ..., val: ...}, ...]
db.getCollection('collection').updateMany(
{ legacyProp: { $exists: true } },
[
{
$set: {
newProp: {
$first: {
$filter: { input: newPropValues, cond: { $eq: ["$_id", "$$this._id"] } }
}
}
}
},
{ $set: { legacyProp: "$$REMOVE", newProp: "$$newProp.val" } }
]
)
Or you can try bulkWrite:
let bulkOperations = []
db.getCollection('collection').find({ legacyProp: { $exists: true } }).forEach(doc => {
const newPropValue = await fetchNewPropValue(doc._id);
bulkOperations.push({
updateOne: {
filter: { _id: doc._id },
update: {
$set: { newProp: newPropValue },
$unset: { legacyProp: "" }
}
}
});
if (bulkOperations.length > 10000) {
db.getCollection('collection').bulkWrite(bulkOperations, { ordered: false });
bulkOperations = [];
}
})
if (bulkOperations.length > 0)
db.getCollection('collection').bulkWrite(bulkOperations, { ordered: false })

Mongoose Query with array of object

I want to query using _id in array of object.
const members = [
{ _id: 60fb6f2859fd441f38e21172 },
{ _id: 60fb70a059fd441f38e21175 },
{ _id: 60fb6d9459fd441f38e2116c }
]
I know if it's an simple array, we can use $in operator. Please help how to query using above array.
const files = await File.find({ _id: { $in: members._id } });

MongoDB find objects sorted by string dateTime

I have a mongoDB with companies having the following structure:
"name": "CORESITE",
"isin": "us21870q1058",
"XDateInserted": "2020-09-16 14:19",
"XDateUpdated": "2020-10-09 14:38",
i wish to query all companies to an array, sorted by XdateUpdated. The following bit of code succeeds in sorting the result by name, however, sorting by XdateUpdated just return me a random order. Any Solutions ?
const allCompanies = await database
.get()
.db(Mongo.db)
.collection(Mongo.CC)
.find(
{
circulate: { $ne: false },
},
{
projection: {
_id: 0,
isin: 1,
name: 1,
XDateInserted: 1,
},
}
)
.sort({ name: -1 })
.toArray();
Here is a solution using aggregation.
You'll first have to convert the string date to Date type. Then Sort it.
const allCompanies = await database
.get()
.db(Mongo.db)
.collection(Mongo.CC)
.aggregate([
{
$addFields: {
updatedDateFormatted: {
$dateFromString: {
dateString: "$XDateUpdated"
}
}
}
},
{
$sort: {
updatedDateFormatted: -1
}
},
{
$project: {
updatedDateFormatted: 0 // To hide the custom field from result
}
}
])
.toArray();

Mongodb update duplicate array value

I am getting in an array of data and then want to insert that into my mongodb. I want to overwrite any duplicate values with the new one (on Im uploading) and if no duplicates just add it onto the current array.
I currently have:
db.cases.updateOne(
{ companyID: 218 },
{
$addToSet: {
cases: [AN ARRAY OF CASES]
},
$currentDate: { lastModified: true }
})
The collection has multiple companies and each has an array of cases - see image below :
The other thing that doesn't seem to work is that the currentDate doesn't seem to change whenever I update the cases, not sure if thats the way I have written the query?
Thank you.
You can use $addToSet (https://docs.mongodb.com/manual/reference/operator/update/addToSet/) with $[] (https://docs.mongodb.com/manual/reference/operator/update/positional-filtered/) to accomplish this.
I scripted an example in Node.js to show you what I mean:
const { MongoClient } = require('mongodb');
async function main() {
/**
* Connection URI. Update <username>, <password>, and <your-cluster-url> to reflect your cluster.
*/
const uri = "mongodb+srv://<username>:<password>#<your-cluster-url>?retryWrites=true&w=majority";
/**
* The Mongo Client you will use to interact with your database
*/
const client = new MongoClient(uri, { useUnifiedTopology: true });
try {
// Connect to the MongoDB cluster
await client.connect();
// Make the appropriate DB calls
await updateArray(client, 'id1')
} finally {
// Close the connection to the MongoDB cluster
await client.close();
}
}
main().catch(console.error);
async function updateArray(client, id) {
const docId = "299";
const mynewdoc = {
id: docId,
priority: 5,
casenumber: 40,
new: "field"
}
const result = await client.db("NameOfYourDb").collection("NameOfYourCollection").updateOne(
{ _id: id },
{
$set: {
"cases.$[element]": mynewdoc
}
},
{
arrayFilters: [{ "element.id": docId }]
}
)
if (result) {
console.log(result);
} else {
console.log(`Not found '${id}'`);
}
const result2 = await client.db("NameOfYourDb").collection("NameOfYourCollection").updateOne(
{ _id: id },
{
$addToSet: {
"cases": mynewdoc
}
}
)
if (result2) {
console.log(result2);
} else {
console.log(`Not found '${id}'`);
}
}
See https://www.mongodb.com/blog/post/quick-start-nodejs-mongodb--how-to-get-connected-to-your-database for an explanation of how the Node.js code is structured.
So I have figured a hack around but it is not exactly what I wanted - maybe someone can use this and expand upon it to make it work better:
db.cases.aggregate(
{ $match: { 'companyID': 218 }},
{ $unwind: '$cases' },
{ $group: { "_id": "$cases.casenumber", cases: {$push:'$cases'}, "count": { $sum:1 }}},
{ $match: { "count": { "$gt": 1 }}}
{ $project: { "cases": { $slice: [ "$cases", -1, { $subtract: [ { $size: "$cases" }, 1 ]}] }}},
{ $project: { "cases": { $arrayElemAt: ['$cases', 0] }}},
{ $group: { _id: 1, cases: {$push:'$cases' }}},
{ $out: 'cases' }
)
The only problem is that using out overwrites the cases document so I need to figure out a way to write it to the cases array inside of the company id (218)
Hopefully this can help someone though.
Thanks.

regex as $filter in projection

I'm trying to find (using a regexp) an array field and returns that element only
This is my data
[
{
"_id": "56d6e8bbf7404bd80a017edb",
"name": "document1",
"tags": [
"A area1",
"B area2",
"C area3"
]
},
{
"_id": "56d6e8bbf7404bd82d017ede",
"name": "document2",
"tags": [
"b_area3",
"b_area4",
"b_area5"
]
}
]
My query
var query=new RegExp('^'+string, "i");
Model.find({tags:query},{'tags.$': 1}, function(err,data){
if(err) console.log(err);
res.json(data);
});
This query selects only the tags field (as I want), but selects the first element. I need the element(s) that matches the query.
EDIT: I tried the mongodb aggregate too, the $filter cond is wrong. I get the error "MongoError: invalid operator $regex"
caseNote.aggregate([
{ $match: {tags:query}},
{ $project: {
tags: {$filter: {
input: 'tags',
as: 'item',
cond: {$regex: ['$$item', query]}
}}
}}
], function (err, result) {
if (err) {
console.log(err);
} else {
res.json(result);
}
});
EDIT2: on #zangw suggestion, this is the mongoose version, but it's incomplete: the tags fields is fine (needs test), but the query still returns the whole document.
caseNote
.aggregate({ $match: {tags: {$in:['area3']}}})
.unwind('tags')
.exec(function(err,d){
res.json(d);
});
According this issue Use $regex as the expression in a $cond, the $regex could NOT be used with cond for current mongo version.
Maybe you can try this one, filter the area3 through $match, then get all matched tags through $group, then remove the _id through $project.
caseNote.aggregate([{$unwind: '$tags'},
{$match: {tags: /area3/}},
{$group: {_id: null, tags: {$push: '$tags'}}},
{$project: {tags: 1, _id: 0}}])
.exec(function(err, tags) {
if (err)
console.log(err);
else
console.log(tags);
});
Results:
{ "tags" : [ "C area3", "b_area3" ] }
As per #zangw's answer,
this ISSUE-SERVER-8892, According this issue Use $regex as the expression in a $cond, the $regex could NOT be used with cond for current mongo version.
MongoDB v4.1.11 has launched new features in ISSUE-SERVER-11947, This features adds three new expressions $regexFind, $regexFindAll and $regexMatch to the aggregation language.
In your example, You can use $regexMatch expression,
Model.aggregate([
{
$project: {
tags: {
$filter: {
input: "$tags",
cond: {
$regexMatch: {
input: "$$this",
regex: query
}
}
}
}
}
}
])
Playground
This is how I solved it. If the query can be parsed to regex the projection is not added to aggregation, but happening after the db request. If the query string is normal string, the projection added.
const { query } = req; // /rea/ or 'C area3'
const convertIfRegex = string => {
const parts = string.split('/')
let regex = string;
let options = '';
if (parts.length > 1) {
regex = parts[1];
options = parts[2];
} else {
return false
}
try {
return new RegExp(regex, options);
} catch (e) {
return null
}
};
const regex = convertIfRegex(queryString);
const aggregations = [{ $match: { tags:query } }]
if (!regex) {
aggregations.push({
$project: {
tags: {$filter: {
input: 'tags',
as: 'item',
cond: {$eq: ['$$item', query]}
}}
}
})
}
let result = await caseNote.aggregate(aggregations);
if (regex) {
result = result.reduce((acc, entry) => {
const tags = entry.tags.filter(({ tag }) => (
tag.match(regex)
))
if (tags.length) return [...acc, { ...entry, tags }];
return acc;
})
}
res.json(result)