I have a simple sample database which I use to develop a simple micro framework for our application. To verify versioning works as expected, I need to update all documents by adding a new field. I used this thread as guide, so what I'm doing should work.
{
"_id": {
"$oid": "63d95015f94d9a88ecbc1a00"
},
"Version": 1,
"bookTitle": "Fancy Book 0",
"author": "Author 0"
}
{
"_id": {
"$oid": "63d95015f94d9a88ecbc1a01"
},
"Version": 1,
"bookTitle": "Fancy Book 1",
"author": "Author 1"
}
... 8 more
Now when I run this in MongoSH:
db.Books.updateMany({}, {$set: {'NewField': true}})
I get this output:
{
acknowledged: true,
insertedId: null,
matchedCount: 0,
modifiedCount: 0,
upsertedCount: 0
}
So what's the issue here? It does execute and it seemingly is correct, but no single document updates. The official documentation states that {} is a selector for all documents, so why does it not match even one of them?
Related
I have the query:
db.changes.find(
{
$or: [
{ _id: ObjectId("60b1e8dc9d0359001bb80441") },
{ _oid: ObjectId("60b1e8dc9d0359001bb80441") },
],
},
{
_id: 1,
}
);
which returns almost instantly.
But the moment I add a sort, the query doesn't return. The query just runs. The longest I could tolerate the query running was over 30 Min, so I'm not entirely sure if it does eventually return.
db.changes
.find(
{
$or: [
{ _id: ObjectId("60b1e8dc9d0359001bb80441") },
{ _oid: ObjectId("60b1e8dc9d0359001bb80441") },
],
},
{
_id: 1,
}
)
.sort({ _id: -1 });
I have the following indexes:
[
{
"_oid" : 1
},
{
"_id" : 1
}
]
and this is what db.currentOp() returns:
{
"host": "xxxx:27017",
"desc": "conn387",
"connectionId": 387,
"client": "xxxx:55802",
"appName": "MongoDB Shell",
"clientMetadata": {
"application": {
"name": "MongoDB Shell"
},
"driver": {
"name": "MongoDB Internal Client",
"version": "4.0.5-18-g7e327a9017"
},
"os": {
"type": "Linux",
"name": "Ubuntu",
"architecture": "x86_64",
"version": "20.04"
}
},
"active": true,
"currentOpTime": "2021-09-24T15:26:54.286+0200",
"opid": 71111,
"secs_running": NumberLong(23),
"microsecs_running": NumberLong(23860504),
"op": "query",
"ns": "myDB.changes",
"command": {
"find": "changes",
"filter": {
"$or": [
{
"_id": ObjectId("60b1e8dc9d0359001bb80441")
},
{
"_oid": ObjectId("60b1e8dc9d0359001bb80441")
}
]
},
"sort": {
"_id": -1.0
},
"projection": {
"_id": 1.0
},
"lsid": {
"id": UUID("38c4c09b-d740-4e44-a5a5-b17e0e04f776")
},
"$readPreference": {
"mode": "secondaryPreferred"
},
"$db": "myDB"
},
"numYields": 1346,
"locks": {
"Global": "r",
"Database": "r",
"Collection": "r"
},
"waitingForLock": false,
"lockStats": {
"Global": {
"acquireCount": {
"r": NumberLong(2694)
}
},
"Database": {
"acquireCount": {
"r": NumberLong(1347)
}
},
"Collection": {
"acquireCount": {
"r": NumberLong(1347)
}
}
}
}
This wasn't always a problem, it's only recently started. I've also rebuilt the indexes, and nothing seems to work. I've tried using .explain(), and that also doesn't return.
Any suggestions would be welcome. For my situation, it's going to be much easier to make changes to the DB than it is to change the query.
This is happening due to the way Mongo chooses what's called a "winning plan", I recommend you read more on this in my other answer which explains this behavior. However it is interesting to see if the Mongo team will consider this specific behavior a feature or a bug.
Basically the $or operator has some special qualities, as specified:
When evaluating the clauses in the $or expression, MongoDB either performs a collection scan or, if all the clauses are supported by indexes, MongoDB performs index scans. That is, for MongoDB to use indexes to evaluate an $or expression, all the clauses in the $or expression must be supported by indexes. Otherwise, MongoDB will perform a collection scan.
It seems that the addition of the sort is disrupting the usage this quality, meaning you're running a collection scan all of a sudden.
What I recommend you do is use the aggregation pipeline instead of the query language, I personally find it has more stable behavior and it might work there. If not maybe just do the sorting in code ..
The server can use a separate index for each branch of the $or, but in order to avoid doing an in-memory sort the indexes used would have to find the documents in the sort order so a merge-sort can be used instead.
For this query, an index on {_id:1} would find documents matching the first branch, and return them in the proper order. For the second branch, and index on {oid:1, _id:1} would do the same.
If you have both of those indexes, the server should be able to find the matching documents quickly, and return them without needing to perform an explicit sort.
I am running a simple query using mongoDB compass using filter and project and I'm having a behaviour I can't explain.
Here is my filter:
{
"$and": [
{
"_id": ObjectId('611ee5ee6b93815ee436969e')
},
{
"type": "article"
}
]
}
I get the following result:
{
"_id": {
"$oid": "611ee5ee6b93815ee436969e"
},
"type": "article",
"history": [],
"liked": [],
"parentId": "61105f00cc11ec10406fd1c4",
"permissionList": [],
"title": "Test",
"wikiId": "610de623fbfa1e58cdba9d2c"
}
As expected I get all the fields, in particular type and wikiId
However if i add the following projection:
{
"_id": 1,
"wikiId": 1,
"parentId": 1,
"title": 1,
"type": 1,
"permissionList": 1,
"liked": 1,
"history": 1
}
I would expect the same result, however i get:
{
"_id": {
"$oid": "611ee5ee6b93815ee436969e"
},
"type": "article",
"history": [],
"liked": [],
"parentId": "61105f00cc11ec10406fd1c4",
"permissionList": [],
"title": "Test"
}
This time i don't have the field wikiId, however it was requested in the projection.
And what's bug me is that if I do this projection instead:
{
"_id": 1,
"wikiId": 1,
"parentId": 1,
"title": 1,
"permissionList": 1,
"liked": 1,
"history": 1
}
Then I got the wikiId field as expected in the result again.
Anyone can provide me an insight of what is going on with those queries and where i'm mistaken.
Edit 1: The fact I want to use a projection is that depending on the type field I can use different document with different field.
In my Java code I'm using #BsonDiscriminator(key = "type") but when I explicitly want a kind of document I'm creating the appropriate projection to be sure. However in this case I just wanted to simplify the issue I'm facing to the simplest.
Thanks
in compass by default filters are having and condition so your filter can be refactored as below:
{ "_id": ObjectId('611ee5ee6b93815ee436969e'), "type": "article" }
And regarding projection you can just simply mention fields to Exclude or include like below:
{
"_id": 1,
"wikiId": 1,
"parentId": 1,
"title": 1,
"type": 1,
"permissionList": 1,
"liked": 1,
"history": 1
}
if you need all fields then there is no need to mention anything by default in compass you get all fields, but lets say you want all but few fields(_id,wikiId) then use below projection:
{
"_id": 0,
"wikiId": 0,
}
Also in compass you can try clicking find again or refresh button to see as sometimes it does not reflects current filter as as we change filter it fetches data so just hit refresh or find and see it should work.
I have an aggregate call in MongoDB (v4.2) where I'm doing a $lookup and $unwind of related sub-docs but I'm having problems figuring out how to sort. The sub-doc has a field called created and so does the top-level doc:
Wine.aggregate(
[
{
"$match": {
"user": ObjectId("<userId>")
}
},
{
"$sort": {
"created": -1
}
},
{
"$lookup": {
"from": "wineuniversals", // the collection name
"localField": "wineUniversal", // field from the wines model
"foreignField": "_id", // field how the two collections are linked
"as": "uWineData" // the object property where the universal wine data is stored
}
},
{
"$unwind": {
"path": "$uWineData"
}
}
]
).exec(function(err, wines) {...});
And here is an example of the docs it returns:
{
"_id": "5dbbc408e78d867664213147",
"photoURL": "51972ee99dec8f31cdb6ff8025a0d3d3",
"user": "554f99352ee62248071b4d0f",
"mode": "past",
"wineUniversal": "5dce6038e78d8676642131fd",
"hidden": false,
"deleted": false,
"eventBlindTasting": false,
"quantity": 1,
"groupDescription": "none",
"comment": "Surprisingly tasty. You'd think it's be olonk based on the gimmicky label.",
"scoreTotal": 91,
"scoreOverallImpression": 4.2,
"scoreFinish": 4,
"scoreTaste": 4,
"scoreAroma": 4.3,
"lastUpdated": "2020-05-21T23:06:54.497Z",
"created": "2020-05-21T23:06:54.498Z", // FIRST INSTANCE OF CREATED
"uWineData": {
"_id": "5dce6038e78d8676642131fd",
"scoreCount": 1,
"averageScore": 0,
"userWines": ["5dbbc408e78d867664213147"],
"expertScore2": "",
"expertReviewer2": null,
"expertScore1": "",
"expertReviewer1": null,
"currency": "USD",
"commonPrice": 12,
"additionalDetails": "",
"designation": "",
"category": "Red",
"varietal": "Cabernet Sauvignon",
"vineyard": "",
"appellation": "California",
"subRegion": "",
"region": "California",
"country": "United States",
"wineryUrl": "https://www.thewalkingdeadwine.com",
"winery": "The Walking Dead Wines",
"vintage": "2016",
"deleted": false,
"lastUpdated": "2019-11-15T08:22:32.437Z",
"created": "2019-11-15T08:22:16.579Z", // SECOND INSTANCE OF CREATED IN SUBDOC
"__v": 1
}
}
It looks like the $sort step is keying off the created field in the subdoc instead of the top-level doc's created field.
Is there a way to reference the created in the top-level doc so my $sort step orders the docs by the top level created field?
Ah ha! Turns out that adding the preserverNullAndEmptyPathways: true to the $unwind fixed the issue. Some of the top-level docs didn't have a sub-doc available yet (eg, the most recent docs hadn't yet had an _id linkage added to them, so there wasn't anything to $unwind, thus the Null/Empty issue).
{
"path": "$uWineData",
"preserveNullAndEmptyArrays": true
}
I got a big array with data in the following format:
{
"application": "myapp",
"buildSystem": {
"counter": 2361.1,
"hostname": "host.com",
"jobName": "job_name",
"label": "2361",
"systemType": "sys"
},
"creationTime": 1517420374748,
"id": "123",
"stack": "OTHER",
"testStatus": "PASSED",
"testSuites": [
{
"errors": 0,
"failures": 0,
"hostname": "some_host",
"properties": [
{
"name": "some_name",
"value": "UnicodeLittle"
},
<MANY MORE PROPERTIES>,
{
"name": "sun",
"value": ""
}
],
"skipped": 0,
"systemError": "",
"systemOut": "",
"testCases": [
{
"classname": "IdTest",
"name": "has correct representation",
"status": "PASSED",
"time": "0.001"
},
<MANY MORE TEST CASES>,
{
"classname": "IdTest",
"name": "normalized values",
"status": "PASSED",
"time": "0.001"
}
],
"tests": 8,
"time": 0.005,
"timestamp": "2018-01-31T17:35:15",
"title": "IdTest"
}
<MANY MORE TEST SUITES >,
]}
Where I can distinct three main structures with big data: TestSuites, Properties, and TestCases. My task is to sum all times from each TestSuite so that I can get the total duration of the test. Since the properties and TestCases are huge, the query cannot complete. I would like to select only the "time" value from TestSuites, but it kind of conflicts with the "time" of TestCases in my query:
db.my_tests.find(
{
application: application,
creationTime:{
$gte: start_date.valueOf(),
$lte: end_date.valueOf()
}
},
{
application: 1,
creationTime: 1,
buildSystem: 1,
"testSuites.time": 1,
_id:1
}
)
Is it possible to project only the "time" properties from TestSuites without loading the whole schema? I already tried testSuites: 1, testSuites.$.time: 1 without success. Please notice that TestSuites is an array of one element with a dictionary.
I already checked this similar post without success:
Mongodb update the specific element from subarray
Following code prints duration of each TestSuite:
query = db.my_collection.aggregate(
[
{$match: {
application: application,
creationTime:{
$gte: start_date.valueOf(),
$lte: end_date.valueOf()
}
}
},
{ $project :
{ duration: { $sum: "$testSuites.time"}}
}
]
).forEach(function(doc)
{
print(doc._id)
print(doc.duration)
}
)
Is it possible to project only the "time" properties from TestSuites
without loading the whole schema? I already tried testSuites: 1,
testSuites.$.time
Answering to your problem of prejecting only the time property of the testSuites document you can simply try projecting it with "testSuites.time" : 1 (you need to add the quotes for the dot notation property references).
My task is to sum all times from each TestSuite so that I can get the
total duration of the test. Since the properties and TestCases are
huge, the query cannot complete
As for your task, i suggest you try out the mongodb's aggregation framework for your calculations documents tranformations. The aggregations framework option {allowDiskUse : true} will also help you if you are proccessing "large" documents.
This should be really easy but I cant seem to get it working;
I just need to remove a sub-document that I had accidentally run and introduced undesirable info in the sub-document.
I tried - db.test.remove({}, {dcoll10:{"$exists": true}, {multi: true}); but this didnt work.
Example of document and sub-doc (dcoll10) is given below;
{
"_id": "SSS",
"ts": { "$date": 1395927614611 },
"dcoll10": [
{
"_id": "SSS",
"type": "1813",
"gro": "0.1",
},
{
"_id": "SSS",
"type": "1813",
"gro": "0.1",
}
],
"assima" : [
{......}
]
}
It sounds like you want the $unset operator which is documented here: http://docs.mongodb.org/manual/reference/operator/update/unset/