Append key value if mongodb collection does not have the value? - mongodb

I have a db with a bunch of collections.
Some of them have 'status' and some don't.
How can I insert 'status':'pending' into the collections that lack 'status' - but not overwrite the collections that already have a status?
Using pymongo / flask / python 2.7
I tried this:
orders = monDB.find('order')
for order in orders:
if not order['status']:
monDB.update('order', {'status':'pending'})
print 'success'
But nothing happens. What am I doing wrong?


Use $exists to check if the field exists and $set to create it if it doesn't. Assuming monDB is your collection:
monDB.update({'status': {'$exists' : False}}, {'$set' : {'status': 'pending'}}, multi=True)
In the mongo shell:
> db.myQueue.update({status: {$exists: false}}, {$set : {status: 'pending'}}, false, true)
See http://docs.mongodb.org/manual/reference/operators/ and http://docs.mongodb.org/manual/applications/update/#crud-update-update.

Related

PyMongo : Array filtered positional operator update gives me error

I am trying to run a python code to update a collection with arrays. The below statement give error wtih pymongo . Please guide
db.students.update({}, {'$set': {"grades.$[element]": 100}}, {'multi': true, 'arrayFilters': [{"element": { '$gte': 100}}]} )
tried : multi=True tried : multi:True
I am getting the below error :
common.validate_boolean("upsert", upsert)
File "F:\Program Files\Python3.7\lib\site-packages\pymongo\common.py", line 159, in validate_boolean
raise TypeError("%s must be True or False" % (option,))
TypeError: upsert must be True or False

Pymongo's syntax is a tad different than Mongo's syntax, you should write it like this:
db.students.update({}, {'$set': {"grades.$[element]": 100}}, multi=True, array_filters=[{"element": {'$gte': 100}}])
Also update is deprecated, and in your case you should use update_many instead.
db.students.update_many({}, {'$set': {"grades.$[element]": 100}}, array_filters=[{"element": {'$gte': 100}}])

logstash-input-mongodb loop on a "restarting error" - Timestamp

I try to use the mongodb plugin as input for logstash.
Here is my simple configuration:
input {
mongodb {
uri => 'mongodb://localhost:27017/testDB'
placeholder_db_dir => '/Users/TEST/Documents/WORK/ELK_Stack/LogStash/data/'
collection => 'logCollection_ALL'
batch_size => 50
}
}
filter {}
output { stdout {} }
But I'm facing a "loop issue" probably due to a field "timestamp" but I don't know what to do.
[2018-04-25T12:01:35,998][WARN ][logstash.inputs.mongodb ] MongoDB Input threw an exception, restarting {:exception=>#TypeError: wrong argument type String (expected LogStash::Timestamp)>}
With also a DEBUG log:
[2018-04-25T12:01:34.893000 #2900] DEBUG -- : MONGODB | QUERY | namespace=testDB.logCollection_ALL selector={:_id=>{:$gt=>BSON::ObjectId('5ae04f5917e7979b0a000001')}} flags=[:slave_ok] limit=50 skip=0 project=nil |
runtime: 39.0000ms
How can I parametrize my logstash config to get my output in the stdout console ?

It's because of field #timestamp that has ISODate data type.
You must remove this field from all documents.
db.getCollection('collection1').update({}, {$unset: {"#timestamp": 1}}, {multi: true})

mongodb status of index creation job

I'm using MongoDB and have a collection with roughly 75 million records.
I have added a compound index on two "fields" by using the following command:
db.my_collection.ensureIndex({"data.items.text":1, "created_at":1},{background:true}).
Two days later I'm trying to see the status of the index creation. Running db.currentOp() returns {}, however when I try to create another index I get this error message:
cannot add index with a background operation in progress.
Is there a way to check the status/progress of the index creation job?
One thing to add - I am using mongodb version 2.0.6. Thanks!

At the mongo shell, type below command to see the current progress:
rs0:PRIMARY> db.currentOp(true).inprog.forEach(function(op){ if(op.msg!==undefined) print(op.msg) })
Index Build (background) Index Build (background): 1431577/55212209 2%
To do a real-time running status log:
> while (true) { db.currentOp(true).inprog.forEach(function(op){ if(op.msg!==undefined) print(op.msg) }); sleep(1000); }
Index Build: scanning collection Index Build: scanning collection: 43687948/47760207 91%
Index Build: scanning collection Index Build: scanning collection: 43861991/47760228 91%
Index Build: scanning collection Index Build: scanning collection: 44993874/47760246 94%
Index Build: scanning collection Index Build: scanning collection: 45968152/47760259 96%

You could use currentOp with a true argument which returns a more verbose output, including idle connections and system operations.
db.currentOp(true)
... and then you could use db.killOp() to Kill the desired operation.

The following should print out index progress:
db
.currentOp({"command.createIndexes": { $exists : true } })
.inprog
.forEach(function(op){ print(op.msg) })
outputs:
Index Build (background) Index Build (background): 5311727/27231147 19%

Unfortunately, DR9885 answer didn't work for me, it has spaces in the code (syntax error) and even if the spaces are removed, it returns nothing.
This works as of Mongo Shell v3.6.0
db.currentOp().inprog.forEach(function(op){ if(op.msg) print(op.msg) })
Didn't read Bajal answer until after I posted mine, but it's almost exactly the same except that it's slightly shorter code and also works.

I like:
db.currentOp({
'msg' :{ $exists: true },
'command': { $exists: true },
$or: [
{ 'command.createIndexes': { $exists: true } },
{ 'command.reIndex': { $exists: true } }
]
}).inprog.forEach(function(op) {
print(op.msg);
});
Output example:
Index Build Index Build: 84826/335739 25%
Documentation suggests:
db.adminCommand(
{
currentOp: true,
$or: [
{ op: "command", "command.createIndexes": { $exists: true } },
{ op: "none", "msg" : /^Index Build/ }
]
}
)
Active Indexing Operations example.

Simple one to just check progress of a single index going on:
db.currentOp({"msg":/Index/}).inprog[0].progress;
outputs:
{ "done" : 86007212, "total" : 96868386 }

Find progress of index jobs, nice one liner:
> db.currentOp().inprog.map(a => a.msg)
[
undefined,
undefined,
undefined,
undefined,
undefined,
undefined,
"Index Build: scanning collection Index Build: scanning collection: 16448156/54469342 30%",
undefined,
undefined
]

MongoDB :error doing update (anon):1552 on Update command

I am using MongoDB Version of 1.2.6
I was doing a multiple update on a collection ( If a field named yesterday exists ,update it or else create field called yesterday for that record )
This is my query
db.userLoginCount.update({yesterday : {$exists : false}}, {$set: {yesterday : '07082013'}},false,true)
Once i run this , i am getting an error as
Tue Jul 9 02:43:05 Error: error doing update (anon):1552
Please tell me what is wring with the above query ??

Can i use mongoexport --query <file> where file is a list of conditions

I have an array of ids stored in a file, and I want to retrieve their data from the mongdb
so i looked into the mongoexport method. it seems --query option can only accept a json instead read a large json or array from a file. In my case, it is about 4000 ids stored in the file. Is there a solution to this?
I was able to use
mongoexport --db db --collection collection --field name --csv -oout ~/data.csv
but how to read query conditions from a file
for example, for mongoid in rails application, query like this is Data.where(:_id.in => array).
or is it possible to do from mongo shell by executing a javscript file
tks

I believe you can use a javascript to output the array you need.
you can use "printjson" command in your script, for example:
create a script.js javascript file as following:
script.js:
printjson( db.albums.find({_id : 18}, {"images" : 1,"_id":0}).toArray() )
Call hi as follow:
mongo test script.js > out.txt
In my local environment albums collection has the following structure:
db.albums.findOne({"_id":18
{
"_id" : 18,
"images" : [
2926,
5377,
8036,
9023,
10119,
11543,
12305,
12556,
12576,
13753,
14414,
14865,
15193,
15933,
17156,
17314,
17391,
20168,
21705,
22016,
22348,
23036,
23452,
24112,
27086,
27310,
27864,
28092,
29184,
29190,
29250,
29354,
29454,
29563,
30366,
30619,
31390,
31825,
31906,
32339,
32674,
33307,
33844,
37475,
37976,
38717,
38774,
39801,
41369,
41752,
44977,
45384,
45643,
46918,
47069,
50099,
52755,
54314,
54497,
62338,
63438,
63572,
63600,
65631,
66953,
67160,
67369,
69802,
71087,
71127,
71282,
73123,
73201,
73954,
74972,
76279,
77054,
78397,
78645,
78936,
79364,
79707,
83065,
83142,
83568,
84160,
85391,
85443,
85488,
86143,
86240,
86949,
89406,
89846,
92591,
92639,
92655,
93844,
93934,
94987,
95324,
95431,
95817,
95864,
96230,
96975,
97026
]
}
>
, so the output I got was:
$ cat out.txt
MongoDB shell version: 2.2.1
connecting to: test
[
{
"images" : [
2926,
5377,
8036,
9023,
10119,
11543,
12305,
12556,
12576,
13753,
14414,
14865,
15193,
15933,
17156,
17314,
17391,
20168,
21705,
22016,
22348,
23036,
23452,
24112,
27086,
27310,
27864,
28092,
29184,
29190,
29250,
29354,
29454,
29563,
30366,
30619,
31390,
31825,
31906,
32339,
32674,
33307,
33844,
37475,
37976,
38717,
38774,
39801,
41369,
41752,
44977,
45384,
45643,
46918,
47069,
50099,
52755,
54314,
54497,
62338,
63438,
63572,
63600,
65631,
66953,
67160,
67369,
69802,
71087,
71127,
71282,
73123,
73201,
73954,
74972,
76279,
77054,
78397,
78645,
78936,
79364,
79707,
83065,
83142,
83568,
84160,
85391,
85443,
85488,
86143,
86240,
86949,
89406,
89846,
92591,
92639,
92655,
93844,
93934,
94987,
95324,
95431,
95817,
95864,
96230,
96975,
97026
]
}
]
Regards,
Moacy