What does "fastmod" means in mongodb logs - mongodb

I have thought fastmod specifies some operations like update-in-place.
In my app I'm doing update by _id using '$' modifiers, for example:
$colleciton->update(
array('_id' => $id),
array(
'$inc' => array('hits' => new MongoInt32(1)),
'$set' => array(
'times.gen' => gettimeofday(true),
'http.code' => new MongoInt32(200)
)
),
array('safe'=>false,'multiple'=>false,'upsert'=>false)
);
I've got such logs:
Wed Jul 25 11:08:36 [conn7002912] update mob.stat_pages query: { _id: BinData } update: { $inc: { hits: 1 }, $set: { times.gen: 1343203715.684896, http.code: 200 } } nscanned:1 nupdated:1 keyUpdates:0 locks(micros) w:342973 342ms
In logs as you can see I don't have any "fastmod" flags. There is no "moved" flag, because I set fields 'times.gen' and 'http.code' on insert, so padding factor is 1.0.
Am I doing something wrong, or I misunderstood meaning of fastmod?

You are correct that "fastmod" in the logs means an in-place update. Some possible reasons for the omission of logged fastmod/in-place operations:
You are actually setting or incrementing a field that doesn't exist, so it must be added, not an in place operation
The logs only show slow queries (default >100ms), so the in-place ones are probably happening too fast to be logged
You seem to be using 2.1 or 2.2 judging by the log - did the messages disappear if/when you switched to the new version?
In terms of looking into this further:
Have a look at the profiler, try with different settings, note: profiling adds load - so use carefully.
You can also try setting the slowms value lower, either on start up or:
> db.setProfilingLevel(0,20) // slow threshold=20ms

Related

logstash-input-mongodb loop on a "restarting error" - Timestamp

I try to use the mongodb plugin as input for logstash.
Here is my simple configuration:
input {
mongodb {
uri => 'mongodb://localhost:27017/testDB'
placeholder_db_dir => '/Users/TEST/Documents/WORK/ELK_Stack/LogStash/data/'
collection => 'logCollection_ALL'
batch_size => 50
}
}
filter {}
output { stdout {} }
But I'm facing a "loop issue" probably due to a field "timestamp" but I don't know what to do.
[2018-04-25T12:01:35,998][WARN ][logstash.inputs.mongodb ] MongoDB Input threw an exception, restarting {:exception=>#TypeError: wrong argument type String (expected LogStash::Timestamp)>}
With also a DEBUG log:
[2018-04-25T12:01:34.893000 #2900] DEBUG -- : MONGODB | QUERY | namespace=testDB.logCollection_ALL selector={:_id=>{:$gt=>BSON::ObjectId('5ae04f5917e7979b0a000001')}} flags=[:slave_ok] limit=50 skip=0 project=nil |
runtime: 39.0000ms
How can I parametrize my logstash config to get my output in the stdout console ?
It's because of field #timestamp that has ISODate data type.
You must remove this field from all documents.
db.getCollection('collection1').update({}, {$unset: {"#timestamp": 1}}, {multi: true})

Insert of data into MongoDB failed: localhost:27017: cannot use 'j' option

Insert of data into MongoDB failed: localhost:27017: cannot use 'j' option when a host does not have journaling enabled in codeigniter with mongodb
Controller
maincontroller.php
function createUser() {
$this->load->library('mongo_db');
$user = array("name" => "tutorialspoint3");
$options = array(
"w" => 1,
"j" => true,
);
$this->mongo_db->insert('tutorialspoint', $user,$options);
}
Had the same problem, you are probably using a 32 bit mongod which has journaling disabled by default.
Just start the database with --journal or add "journal=true" to your mongod.conf (on debian you can find it in /etc/mongod/) to start the database with journaling enabled

User Assertion: 1: Update query failed -- RUNNER_DEAD

We are using MongoDB (v2.6.4) to process some data and everything works great except, once in a while, we get a weird RUNNER_DEAD exception...
MongoDB.Driver.WriteConcernException: WriteConcern detected an error ' Update query failed -- RUNNER_DEAD'. (Response was { "lastOp" : { "$timestamp" : NumberLong("6073471510486450182") }, "connectionId" : 49, "err" : " Update query failed -- RUNNER_DEAD", "code" : 1, "n" : 0, "ok" : 1.0 }).
This is the method that causes the exception:
private void UpdateEntityClassName(EntityClassName myEntity) {
var dateTimeNow = DateTime.UtcNow;
var update = Update<EntityClassName>.Set(p => p.Data, myEntity.Data)
...some more Sets...
.Set(p => p.MetaData.LastModifiedDateTime, dateTimeNow);
var result = _myCollection.Update(Query.EQ("_id", myEntity.Identifier), update, UpdateFlags.Upsert);
}
Exception in MongoDB log:
2014-10-23T13:51:29.989-0500 [conn45] update Database.Table query: { _id: "SameID" } update: { $set: { Data: BinData(0, SomeData...), ...more fields... MetaData.LastModifiedDateTime: new Date(1414090294910) } } nmoved:1 nMatched:1 nModified:1 keyUpdates:0 numYields:0 locks(micros) w:2344 2ms
2014-10-23T13:51:29.989-0500 [conn49] User Assertion: 1: Update query failed -- RUNNER_DEAD
2014-10-23T13:51:29.989-0500 [conn46] update Database.Table query: { _id: "SameID" } update: { $set: { Data: BinData(0, SomeData...), ...more fields... MetaData.LastModifiedDateTime: new Date(1414090294926) } } nMatched:1 nModified:1 fastmod:1 keyUpdates:0 numYields:0 locks(micros) w:249 0ms
2014-10-23T13:51:29.989-0500 [conn49] update Database.Table query: { _id: "SameID" } update: { $set: { Data: BinData(0, SomeData...), ...more fields... MetaData.LastModifiedDateTime: new Date(1414090294864) } } nModified:0 keyUpdates:0 exception: Update query failed -- RUNNER_DEAD code:1 numYields:1 locks(micros) w:285 8ms
I found very little documentation about this exception so any help appreciated.
We are running this in a 3 machine replica set if that changes anything.
We've been running this code for a while and we didn't have that issue before (in our original tests) so we went back to MongoDB 2.4.9 (the one we first tested on) and we don't get this exception anymore. Any ideas as to what might have changed that causes this exception?
Why you couldn't use a regular "update" using capped arrays to
limit the size of the array of queries rather than using some custom
logic).
If you have multiple threads that are doing the same thing, your code
doesn't appear thread-safe - let's say that two threads try to update
the same object with _id XYZ but with different changes. Both fetch
the object, both add a new attribute/value to the array and now both
call save - the first one saves, but the second one's save overwrites
the first one.
But that's not likely to be related to your error with RUNNER_DEAD
error - that's more likely a case where either something is killing
the operation or dropping the collection you're writing to (or the
index being used).
Source: #Asya Kamsky's post.

mongodb status of index creation job

I'm using MongoDB and have a collection with roughly 75 million records.
I have added a compound index on two "fields" by using the following command:
db.my_collection.ensureIndex({"data.items.text":1, "created_at":1},{background:true}).
Two days later I'm trying to see the status of the index creation. Running db.currentOp() returns {}, however when I try to create another index I get this error message:
cannot add index with a background operation in progress.
Is there a way to check the status/progress of the index creation job?
One thing to add - I am using mongodb version 2.0.6. Thanks!
At the mongo shell, type below command to see the current progress:
rs0:PRIMARY> db.currentOp(true).inprog.forEach(function(op){ if(op.msg!==undefined) print(op.msg) })
Index Build (background) Index Build (background): 1431577/55212209 2%
To do a real-time running status log:
> while (true) { db.currentOp(true).inprog.forEach(function(op){ if(op.msg!==undefined) print(op.msg) }); sleep(1000); }
Index Build: scanning collection Index Build: scanning collection: 43687948/47760207 91%
Index Build: scanning collection Index Build: scanning collection: 43861991/47760228 91%
Index Build: scanning collection Index Build: scanning collection: 44993874/47760246 94%
Index Build: scanning collection Index Build: scanning collection: 45968152/47760259 96%
You could use currentOp with a true argument which returns a more verbose output, including idle connections and system operations.
db.currentOp(true)
... and then you could use db.killOp() to Kill the desired operation.
The following should print out index progress:
db
.currentOp({"command.createIndexes": { $exists : true } })
.inprog
.forEach(function(op){ print(op.msg) })
outputs:
Index Build (background) Index Build (background): 5311727/27231147 19%
Unfortunately, DR9885 answer didn't work for me, it has spaces in the code (syntax error) and even if the spaces are removed, it returns nothing.
This works as of Mongo Shell v3.6.0
db.currentOp().inprog.forEach(function(op){ if(op.msg) print(op.msg) })
Didn't read Bajal answer until after I posted mine, but it's almost exactly the same except that it's slightly shorter code and also works.
I like:
db.currentOp({
'msg' :{ $exists: true },
'command': { $exists: true },
$or: [
{ 'command.createIndexes': { $exists: true } },
{ 'command.reIndex': { $exists: true } }
]
}).inprog.forEach(function(op) {
print(op.msg);
});
Output example:
Index Build Index Build: 84826/335739 25%
Documentation suggests:
db.adminCommand(
{
currentOp: true,
$or: [
{ op: "command", "command.createIndexes": { $exists: true } },
{ op: "none", "msg" : /^Index Build/ }
]
}
)
Active Indexing Operations example.
Simple one to just check progress of a single index going on:
db.currentOp({"msg":/Index/}).inprog[0].progress;
outputs:
{ "done" : 86007212, "total" : 96868386 }
Find progress of index jobs, nice one liner:
> db.currentOp().inprog.map(a => a.msg)
[
undefined,
undefined,
undefined,
undefined,
undefined,
undefined,
"Index Build: scanning collection Index Build: scanning collection: 16448156/54469342 30%",
undefined,
undefined
]

mongo-erlang distinct command does not work

I'm trying to get all the distinct values of some collection, I can get the result by executing db.$cmd.findOne({distinct: collection_name, key: some_key}) from within the shell. However, when I do:
mongo:do(safe, master, DbConn, some_db,
fun() ->
mongo:command(
{
distinct, some_collection,
key, some_key
}
)
end
)
I always get a blank list. I'm working on Ubuntu 12.04 + MongoDb 2.2.1 + Erlang R15B02.
Thanks!
Found the solution. It's the problem of bson-erlang as symbol is deprecated according to the bson specs.
See HERE for more info.
mongo:do(safe, master, DbConn, some_db,
fun() ->
mongo:command(
{
'distinct'=>'some_collection',
'key'=> 'some_key'
}
)
end
)
I am using the command in my Lithium project. It gives me the result.
Users::connection()->connection->command(array(
'distinct' => 'users',
'key' => 'status',
));
So the above should work for you too...