I have an index which i need to modify.
{
"v" : 1,
"key" : {
"expectedDateTime" : 1
},
"name" : "expectedDateTime_1",
"ns" : "expectation.expectation_data",
"expireAfterSeconds" : 43200
}
The expireAfterSeconds is incorrect and needs to be changed to 432000.
When I dropped the index it seemed fine
db.expectation_data.dropIndex({"expectedDateTime":1})
{ "nIndexesWas" : 4, "ok" : 1 }
The getIndexes() shows that the index does not exist.
Then when i try to recreate the index i get this error
db.expectation_data.createIndex({"expectedDateTime":1},
{expireAfterSeconds:432000,name:"expectedDateTime"});
{
"ok" : 0,
"errmsg" : "Index with name: expectedDateTime already exists with different options",
"code" : 85
}
Now on running getIndexes(), i see that the index seems to have recreated with the old TTL. I tried repeating this process multiple times, but ran into the same issue again and again.
I cannot find any documentation which says that i cannot recreate an index of the same name. If i use a different name it works fine
db.expectation_data.createIndex({"expectedDateTime":1}, {expireAfterSeconds:432000});
.
.
>db.expectation_data.getIndexes()
.
.
{
"v" : 1,
"key" : {
"expectedDateTime" : 1
},
"name" : "expectedDateTime_1",
"ns" : "expectation.expectation_data",
"expireAfterSeconds" : 432000
}
Is there any restriction on recreating indexes with the same name ?
This looks like the index is recreated automatically after deletion. Make sure that no applications using ensureIndex or #Index-Annotations are connecting to the database.
As it turned out. This was due to an #Index annotation used in the entity with the old timeout. The application was still running when i made the index changes.
When i stopped the application, i was able to create the index as i originally expected
Related
I recently saw this error in a Mongo 2.6 replicaset:
WARNING: the collection 'mydatabase.somecollection' lacks a unique index on _id. This index is needed for replication to function properly.
I assumed the _id index would be unique by default. But I am trying to check / set it. getIndexes shows there is no unique option set.
> db.somecollection.getIndexes()[0]
{
"v" : 1,
"key" : {
"_id" : 1
},
"ns" : "mydatabase.somecollection",
"name" : "_id_"
}
> db.somecollection.ensureIndex({"_id":1},{unique:true})
> { "numIndexesBefore" : 3, "note" : "all indexes already exist", "ok" : 1 }
> db.somecollection.getIndexes()[0]
{
"v" : 1,
"key" : {
"_id" : 1
},
"ns" : "mydatabase.somecollection",
"name" : "_id_"
}
I have tried .validate(true):
...
"valid" : true,
"errors" : [ ],
"ok" : 1
}
and also .reIndex() that runs without error. I am unable to remove the _id index to recreate it - how can I set the index to unique or what should I do to ensure data consistency in the RS? Note the RS was upgraded as per upgrade instructions from 2.2 --> 2.4 --> 2.6. I have found this MongoDB - Collection lacks a unique index on _id but there is nothing that resolves my issue in there.
I have seen this in the past when a new member to the replica set was added with a different Compatibility Version. Run db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } ) on all of your nodes in the replica set and if one is different, stop the replication on that node, change the CompatibilityVersion and then ready it to the replica.
So it turns out that the error came up when a new member was added to the existing replica set, and was only shown on that member. If I connect to the database and try add a duplicate _id, I get the usual E11000 duplicate key error index: ... even though getIndexes() doesn't indicate the unique constraint on the index (assuming it is implicit).
I am trying to remove a shard from a Mongo shard (v3.2) - I cannot see any database names that I need to remove but the draining process is not ending which I am guessing is because its waiting for user action to fix something. Based on the error message, can anyone figure out how to further troubelshoot this issue?
db.runCommand( { removeShard : "shard0001" } )
{
"msg" : "draining ongoing",
"state" : "ongoing",
"remaining" : {
"chunks" : NumberLong(1),
"dbs" : NumberLong(0)
},
"note" : "you need to drop or movePrimary these databases",
"dbsToMove" : [ ],
"ok" : 1
}
Deleting all test databases/collections that I did not need resolved the issue. Doing that in a non-test environment would not be the best approach. Still looking for a better solution.
I'm toying with auto-expiring documents from a collection. The java application creates an index per the Mongo TTL docs.
coll.createIndex(new Document("Expires", 1).append("expireAfterSeconds", 0));
When inserting my document, I set the Expires field to a future Date. For this testing I've been setting it 1 minute in the future.
I've verified the date exists properly, the index appears to be correct, and I've waited 10+ minutes (even though the ttl runner operates every sixty seconds) but the document remains.
{
"_id" : ObjectId("569847baf7794c44b8f2f17b"),
// my data
"Created" : ISODate("2016-01-15T02:02:30.116Z"),
"Expires" : ISODate("2016-01-15T02:03:30.922Z")
}
What else could I have missed? Here are the indexes:
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "prism.prismEventRecord"
},
{
"v" : 1,
"key" : {
"Location.X" : 1,
"Location.Z" : 1,
"Location.Y" : 1,
"Created" : -1
},
"name" : "Location.X_1_Location.Z_1_Location.Y_1_Created_-1",
"ns" : "prism.prismEventRecord"
},
{
"v" : 1,
"key" : {
"Created" : -1,
"EventName" : 1
},
"name" : "Created_-1_EventName_1",
"ns" : "prism.prismEventRecord"
},
{
"v" : 1,
"key" : {
"Expires" : 1,
"expireAfterSeconds" : 0
},
"name" : "Expires_1_expireAfterSeconds_0",
"ns" : "prism.prismEventRecord"
}
]
I wonder if it makes sense to take the java mongo client out of the pic for a minute.
I have created a similar collection, and made the following call in the shell.
db.weblog.createIndex({"expireAt":1},{expireAfterSeconds:0})
When I do, and then I call db.weblog.getIndexes(), this is what the expiring index looks like:
{
"v" : 1,
"key" : {
"expireAt" : 1
},
"name" : "expireAt_1",
"ns" : "logs.weblog",
"expireAfterSeconds" : 0
}
I think your java call may be "appending" a new column to your index (not setting the property you were hoping to set). Take a look... your index def looks like this:
{
"v" : 1,
"key" : {
"Expires" : 1,
"expireAfterSeconds" : 0
},
"name" : "Expires_1_expireAfterSeconds_0",
"ns" : "prism.prismEventRecord"
}
See what I mean? "expireAfterSeconds is a key, not a property. Now -- how do you do THAT with the java shell? Ummm ... don't yell at me, but Im a c# guy ... I found a post or two that punt on the question of ttl indexes from the java client, but they're old-ish.
Maybe the java client has gotten better and now supports options? Hopefully, knowing what the problem is gives a guy with your stellar coding skills enough to take it from here ;-)
EDIT: Stack java driver code (untested):
IndexOptions options = new IndexOptions()
.name('whocareswhatwecallthisindex')
.expireAfter(1, TimeUnit.DAYS);
coll.createIndex(new Document("Expires", 1), options);
EDIT2: C# driver code to create the same index:
var optionsIdx = new CreateIndexOptions() { ExpireAfter = new TimeSpan(0)};
await coll.Indexes.CreateOneAsync(Builders<MyObject>.IndexKeys.Ascending("expiresAt"), optionsIdx);
In case you run into this question as a Golang user, you have two choices:
1 Use structures: This works when you know the payload structure, and is documented extensively
2 Introduce an actual date object into your JSON payload: Only use this if for some reason your payload structure absolutely can't be known ahead of time for some reason.
In my case, the source data comes from a system whose structure is a black-box. I first tried to introduce an ISO-compliant TTL index whose format matched Mongo's but it was still read as text. This led me to deduct that it was due to the driver not being instructed to format it properly.
I believe a deep dive into Mongo's Golang driver's specifics to manipulate this process could only give us a short lived solution (since the implementation details are subject to change). So instead, I suggest introducing a real date property into your payload and let the driver adapt it for Mongo (adapting the principle in this snippet to your own code structure)
err = json.Unmarshal([]byte (objectJsonString.String()), &objectJson)
if err == nil && objectJson != nil {
//Must introduce as native or they are considered text
objectJson["createdAt"] = time.Now()
//Your other code
insertResult, err := collection.InsertOne(context.TODO(), objectJson)
}
So basically, create your JSON or BSON object normally with the rest of the data, and then introduce your TTL index using real date values rather than attempt to have your JSON parser do that work for you.
I look forward to corrections, just please be civil and I'll make sure to update and improve this answer with any observations made.
When I call ensureIndex from the mongo shell on a collection for a compound index an _id field of type ObjectId is auto-generated in the index object.
> db.system.indexes.find();
{ "name" : "_id_", "ns" : "database.coll", "key" : { "_id" : 1 } }
{ "_id" : ObjectId("4ea78d66413e9b6a64c3e941"), "ns" : "database.coll", "key" : { "a.b" : 1, "a.c" : 1 }, "name" : "a.b_1_a.c_1" }
This makes intuitive sense as all documents in a collection need an _id field (even system.indexes, right?), but when I check the indexes generated by morphia's ensureIndex call for the same collection *there is no _id property*.
Looking at morphia's source code, it's clear that it's calling the same code that the shell uses, but for some reason (whether it's the fact that I'm creating a compound index or indexing an Embedded document or both) they produce different results. Can anyone explain this behavior to me?
Not exactly sure how you managed to get an _id field in the indexes collection but both shell and Morphia originated ensureIndex calls for compound indexes do not put an _id field in the index object :
> db.test.ensureIndex({'a.b':1, 'a.c':1})
> db.system.indexes.find({})
{ "v" : 1, "key" : { "_id" : 1 }, "ns" : "test.test", "name" : "_id_" }
{ "v" : 1, "key" : { "a.b" : 1, "a.c" : 1 }, "ns" : "test.test", "name" : "a.b_1_a.c_1" }
>
Upgrade to 2.x if you're running an older version to avoid running into now resolved issues. And judging from your output you are running 1.8 or earlier.
I am using MongoDB 1.6.3, to store a big collection (300k+ records). I added a composite index.
db['collection_name'].getIndexes()
[
{
"name" : "_id_",
"ns" : "db_name.event_logs",
"key" : {
"_id" : 1
}
},
{
"key" : {
"updated_at.t" : -1,
"community_id" : 1
},
"ns" : "db_name.event_logs",
"background" : true,
"name" : "updated_at.t_-1_community_id_1"
}
]
However, when I try to run this code:
db['collection_name']
.find({:community_id => 1})
.sort(['updated_at.t', -1])
.skip(#skip)
.limit(#limit)
I am getting:
Mongo::OperationFailure (too much data
for sort() with no index. add an
index or specify a smaller limit)
What am I doing wrong?
Try adding {community_id: 1, 'updated_at.t': -1} index. It needs to search by community_id first and then sort.
So it "feels" like you're using the index, but the index is actually a composite index. I'm not sure that the sort is "smart enough" to use only the partial index.
So two problems:
Based on your query, I would put community_id as the first part of the index, not the second. updated_at.t sounds like a field on which you'll do range queries. Indexes work better if the range query is the second bit.
How many entries are going to come back from community_id => 1? If the number is not big, you may be able to get away with just sorting without an index.
So you may have to switch the index around and you may have to change the sort to use both community_id and updated_at.t. I know it seems redundant, but start there and check the Google Groups if it's still not working.
Even with an index, I think you can still get that error if your result set exceeds 4MB.
You can see the size by going into the mongodb console and doing this:
show dbs
# pick yours (e.g., production)
use db-production
db.articles.stats()
I ended up with results like this:
{
"ns" : "mdalert-production.encounters",
"count" : 89077,
"size" : 62974416,
"avgObjSize" : 706.9660630690302,
"storageSize" : 85170176,
"numExtents" : 8,
"nindexes" : 6,
"lastExtentSize" : 25819648,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 18808832,
"indexSizes" : {
"_id_" : 3719168,
"patient_num_1" : 3440640,
"msg_timestamp_1" : 2981888,
"practice_id_1" : 2342912,
"patient_id_1" : 3342336,
"msg_timestamp_-1" : 2981888
},
"ok" : 1
}
Having a cursor batch size that is too large will cause this error. Setting the batch size does not limit the amount of data you can process, it just limits how much data is brought back from the database. When you iterate through and hit the batch limit, the process will make another trip to the database.