Let's say the DB contains tens of thousands of docs having the following structure
{
_id: ObjectId("5ef053e819aaa00013a2bd69"),
approvers: [
{
type: "ONE",
details: {
name: "NameOne",
surname: "SurnameOne"
}
},
{
type: "TWO",
details: {
name: "NameTwo",
surname: "SurnameTwo"
}
},
{
type: "THREE",
// details field is missing
}
]
}
I need to select only such docs where there's no approver of type "TWO" or "ONE" or the approver has missing details
I had an idea to use $not in a combination with $elemMatch:
{
$or: [
{
"approvers.type": {
$not: {
$in: ["ONE", "TWO"]
}
}
},
{
approvers: {
$not: {
$elemMatch: {
type: { $in: ["ONE", "TWO"]},
details: {$exists: true}
}
}
}
}
]
}
The query works but it's super ineffective since the index is not used. Based on my understanding, the DB engine has to do full collection scan and in each doc, check all the array elements.
Actually the collection has 75k records and every approvers array can hold up to 3 elements.
Is there any "trick" to make it more effective or the only option would be to change the data structure ?
This is where separated collection is beneficial
Assuming above are projects, a different structure can be in place
//approvals
[{
_id: ObjectId
projectId: ObjectId // etc the _id in your code
type: "one",
details: "some stuff"
},
{
_id: ObjectId
projectId: ObjectId // etc the _id in your code
type: "two",
details: "some stuff"
},
{
_id: ObjectId
projectId: ObjectId // etc the _id in your code
type: "three",
details: "some stuff"
}]
Then you can can get all the projectId where type $ne ["one", "two"], before retrieving the related projects using $in. This should be achievable via aggregation too though I never tried.
Suppose I want to sort the data based on the current city first and then the remaining country data. Is there any way I achieve that in MongoDB?
Example
[
{ id: 2, name: 'sdf' },
{ id: 3, name: 'sfs' },
{ id: 3, name: 'aaa' },
{ id: 1, name: 'dsd' },
];
What I want as an outcome is the data with id 3 at first and the remaining other.
like
[
{ id: 3, name: 'sfs' },
{ id: 3, name: 'aaa' },
{ id: 1, name: 'dsd' },
{ id: 2, name: 'sdf' },
];
It's just a example,
My actual requirement is to sort the data based on certain category first and then the remaining one
It's not possible within mongodb but you could first fetch the documents from the db and then sort them in Javascript (or whatever other language you're using to present the data).
On a side note, having duplicate values in the "id" field is not a good practice and defies the definition of id itself.
There is no straight way to sort condationaly in MongoDB, as per your example you can try aggregation query,
$facet to separate result for both types of documents
first, to get id: 3 documents
second, to get id is not 3 documents and sort by id in ascending order
$project and $concatArrays to concat both arrays in siquance
$unwind deconstruct all array
$replaceRoot to replace all object to root
db.collection.aggregate([
{
$facet: {
first: [
{ $match: { id: 3 } }
],
second: [
{ $match: { id: { $ne: 3 } } },
{ $sort: { id: 1 } }
]
}
},
{
$project: {
all: { $concatArrays: ["$first", "$second"] }
}
},
{ $unwind: "$all" },
{ $replaceRoot: { newRoot: "$all" } }
])
Playground
I have some records like :
{
id: 1,
phone: "+15555555555",
name: "Acme CO.",
vendorcode: "ACMEC"
},
{
id: 2,
phone: "+15555555555",
name: "Acme corporation company",
vendorcode: "ACMECOMPANY"
},
{
id: 3,
phone: "+15555555555",
name: "Acme Incorporated",
vendorcode: null
}
I want to merge records:
IF phone field matches, merge the records. (can overwrite values with the values of the next record being merged).
But if there are vendorcode non-null values in multiple records, create an arrray of values. So "vendorcode" in the new record would be an array.
I would like the output of the above collection to be something like:
{
phone: "+15555555555",
name: "Acme Co.",
vendorcode: ["ACMEC","ACMECOMPANY"]
}
in a new collection.
How to write an aggregation for this in mongodb?
$group by phone, select first name, phone
$ifNull will return vendorcode if its not null
$addToSet to make array of unique vendorcode
$project to remove _id field
$out to write query result in new collection, this will create a new collection and write this query result
db.collection.aggregate([
{
$group: {
_id: "$phone",
phone: { $first: "$phone" },
name: { $first: "$name" },
vendorcode: {
$addToSet: { $ifNull: ["$vendorcode", "$$REMOVE"] }
}
}
},
{ $project: { _id: 0 } },
{ $out: "newCollectionName" }
])
Playground
Is it possible to reference the root document during an update operation such that a document like this:
{"name":"foo","value":1}
can be updated with new values and have the full (previous) document pushed into a new field (creating an update history):
{"name":"bar","value":2,"previous":[{"name:"foo","value":1}]}
And so on..
{"name":"baz","value":3,"previous":[{"name:"foo","value":1},{"name:"bar","value":2}]}
I figure I'll have to use the new aggregate set operator in Mongo 4.2, but how can I achieve this?
Ideally I don't want to have to reference each field explicitly. I'd prefer to push the root document (minus the _id and previous fields) without knowing what the other fields are.
In addition to the new $set operator, what makes your use case really easier with Mongo 4.2 is the fact that db.collection.update() now accepts an aggregation pipeline, finally allowing the update of a field based on its current value:
// { name: "foo", value: 1 }
db.collection.update(
{},
[{ $set: {
previous: {
$ifNull: [
{ $concatArrays: [ "$previous", [{ name: "$name", value: "$value" }] ] },
[ { name: "$name", value: "$value" } ]
]
},
name: "bar",
value: 2
}}],
{ multi: true }
)
// { name: "bar", value: 2, previous: [{ name: "foo", value: 1 }] }
// and if applied again:
// { name: "baz", value: 3, previous: [{ name: "foo", value: 1 }, { name: "bar", value: 2 } ] }
The first part {} is the match query, filtering which documents to update (in our case probably all documents).
The second part [{ $set: { previous: { $ifNull: [ ... } ] is the update aggregation pipeline (note the squared brackets signifying the use of an aggregation pipeline):
$set is a new aggregation operator and an alias of $addFields. It's used to add/replace a new field (in our case "previous") with values from the current document.
Using an $ifNull check, we can determine whether "previous" already exists in the document or not (this is not the case for the first update).
If "previous" doesn't exist (is null), then we have to create it and set it with an array of one element: the current document: [ { name: "$name", value: "$value" } ].
If "previous" already exist, then we concatenate ($concatArrays) the existing array with the current document.
Don't forget { multi: true }, otherwise only the first matching document will be updated.
As you mentioned "root" in your question and if your schema is not the same for all documents (if you can't tell which fields should be used and pushed in the "previous" array), then you can use the $$ROOT variable which represents the current document and filter out the "previous" array. In this case, replace both { name: "$name", value: "$value" } from the previous query with:
{ $arrayToObject: { $filter: {
input: { $objectToArray: "$$ROOT" },
as: "root",
cond: { $ne: [ "$$root.k", "previous" ] }
}}}
Imho, you are making your life indefinitely more complex for no reason with such complicated data models.
Think of what you really want to achieve. You want to correlate different values in one or more interconnected series which are written to the collection consecutively.
Storing this in one document comes with some strings attached. While it seems to be reasonable in the beginning, let me name a few:
How do you get the most current document if you do not know it's value for name?
How do you deal with very large series, which make the document hit the 16MB limit?
What is the benefit of the added complexity?
Simplify first
So, let's assume you have only one series for a moment. It gets as simple as
[{
"_id":"foo",
"ts": ISODate("2019-07-03T17:40:00.000Z"),
"value":1
},{
"_id":"bar",
"ts": ISODate("2019-07-03T17:45:00.000"),
"value":2
},{
"_id":"baz",
"ts": ISODate("2019-07-03T17:50:00.000"),
"value":3
}]
Assuming the name is unique, we can use it as _id, potentially saving an index.
You can actually get the semantic equivalent by simply doing a
> db.seriesa.find().sort({ts:-1})
{ "_id" : "baz", "ts" : ISODate("2019-07-03T17:50:00Z"), "value" : 3 }
{ "_id" : "bar", "ts" : ISODate("2019-07-03T17:45:00Z"), "value" : 2 }
{ "_id" : "foo", "ts" : ISODate("2019-07-03T17:40:00Z"), "value" : 1 }
Say you only want to have the two latest values, you can use limit():
> db.seriesa.find().sort({ts:-1}).limit(2)
{ "_id" : "baz", "ts" : ISODate("2019-07-03T17:50:00Z"), "value" : 3 }
{ "_id" : "bar", "ts" : ISODate("2019-07-03T17:45:00Z"), "value" : 2 }
Should you really need to have the older values in a queue-ish array
db.seriesa.aggregate([{
$group: {
_id: "queue",
name: {
$last: "$_id"
},
value: {
$last: "$value"
},
previous: {
$push: {
name: "$_id",
value: "$value"
}
}
}
}, {
$project: {
name: 1,
value: 1,
previous: {
$slice: ["$previous", {
$subtract: [{
$size: "$previous"
}, 1]
}]
}
}
}])
Nail it
Now, let us say you have more than one series of data. Basically, there are two ways of dealing with it: put different series in different collections or put all the series in one collection and make a distinction by a field, which for obvious reasons should be indexed.
So, when to use what? It boils down wether you want to do aggregations over all series (maybe later down the road) or not. If you do, you should put all series into one collection. Of course, we have to slightly modify our data model:
[{
"name":"foo",
"series": "a"
"ts": ISODate("2019-07-03T17:40:00.000Z"),
"value":1
},{
"name":"bar",
"series": "a"
"ts": ISODate("2019-07-03T17:45:00.000"),
"value":2
},{
"name":"baz",
"series": "a"
"ts": ISODate("2019-07-03T17:50:00.000"),
"value":3
},{
"name":"foo",
"series": "b"
"ts": ISODate("2019-07-03T17:40:00.000Z"),
"value":1
},{
"name":"bar",
"series": "b"
"ts": ISODate("2019-07-03T17:45:00.000"),
"value":2
},{
"name":"baz",
"series": "b"
"ts": ISODate("2019-07-03T17:50:00.000"),
"value":3
}]
Note that for demonstration purposes, I fell back for the default ObjectId value for _id.
Next, we create an index over series and ts, as we are going to need it for our query:
> db.series.ensureIndex({series:1,ts:-1})
And now our simple query looks like this
> db.series.find({"series":"b"},{_id:0}).sort({ts:-1})
{ "name" : "baz", "series" : "b", "ts" : ISODate("2019-07-03T17:50:00Z"), "value" : 3 }
{ "name" : "bar", "series" : "b", "ts" : ISODate("2019-07-03T17:45:00Z"), "value" : 2 }
{ "name" : "foo", "series" : "b", "ts" : ISODate("2019-07-03T17:40:00Z"), "value" : 1 }
In order to generate the queue-ish like document, we need to add a match state
> db.series.aggregate([{
$match: {
"series": "b"
}
},
// other stages omitted for brevity
])
Note that the index we created earlier will be utilized here.
Or, we can generate a document like this for every series by simply using series as the _id in the $group stage and replace _id with name where appropriate
db.series.aggregate([{
$group: {
_id: "$series",
name: {
$last: "$name"
},
value: {
$last: "$value"
},
previous: {
$push: {
name: "$name",
value: "$value"
}
}
}
}, {
$project: {
name: 1,
value: 1,
previous: {
$slice: ["$previous", {
$subtract: [{
$size: "$previous"
}, 1]
}]
}
}
}])
Conclusion
Stop Being Clever when it comes to data models in MongoDB. Most of the problems with data models I saw in the wild and the vast majority I see on SO come from the fact that someone tried to be Smart (by premature optimization) ™.
Unless we are talking of ginormous series (which can not be, since you settled for a 16MB limit in your approach), the data model and queries above are highly efficient without adding unneeded complexity.
addMultipleData: (req, res, next) => {
let name = req.body.name ? req.body.name : res.json({ message: "Please enter Name" });
let value = req.body.value ? req.body.value : res.json({ message: "Please Enter Value" });
if (!req.body.name || !req.body.value) { return; }
//Step 1
models.dynamic.findOne({}, function (findError, findResponse) {
if (findResponse == null) {
let insertedValue = {
name: name,
value: value
}
//Step 2
models.dynamic.create(insertedValue, function (error, response) {
res.json({
message: "succesfully inserted"
})
})
}
else {
let pushedValue = {
name: findResponse.name,
value: findResponse.value
}
let updateWith = {
$set: { name: name, value: value },
$push: { previous: pushedValue }
}
let options = { upsert: true }
//Step 3
models.dynamic.updateOne({}, updateWith, options, function (error, updatedResponse) {
if (updatedResponse.nModified == 1) {
res.json({
message: "succesfully inserted"
})
}
})
}
})
}
//This is the schema
var multipleAddSchema = mongoose.Schema({
"name":String,
"value":Number,
"previous":[]
})
I am looking into the best way to organize filtering. I have the following document format:
{
_id: "info",
ids: ["id1", "id2", "id3"]
}
{
_id: "id1",
value: 5
}
{
_id: "id2",
value: 1
}
{
_id: "id3",
value: 5
}
I need to make the following query: get all documents by id from doc "info" and then filter them out by value 5. So, that result would be something like:
{
_id: "id1",
value: 5
}
{
_id: "id3",
value: 5
}
I suppose I need to do unwind on ids, but how do I then select all documents that match those values? Or maybe I should just use $in operator somehow to grab all documents and after that do filtering?
Any help is aprpeciated. Thanks.
If it is only MongoDB shell/script, I would do it like this:
db.ids.find({ _id: { $in: db.ids.findOne({ _id: "info" }).ids }, value: 5 })
You also have worse versions using:
or the eval command:
db.runCommand({
eval: function(value) {
var ids = db.ids.findOne({ _id: "info" }).ids;
return db.ids.find({ _id: { $in: ids }, value: value }).toArray();
},
args: [5]
})
or the $where operator (low performance because you execute one find for each candidate result with value 5):
db.ids.find({
value: 5,
$where: "db.ids.findOne({ _id: 'info', ids: this._id })"
})
But if you are trying to run the queries through a MongoDb driver, the story might be different.