MongoDB How to copy index definitions from one collection to another? - mongodb

I know there's a way to do db.collection.getIndexes() Which will list all the indexes defined for a collection. Is there a way to copy and create those index definitions to another collection?
There's a lot of them and I don't want to do them one by one.
regarding the duplicated question comment: I do not wish to copy a collection. I wish to export indexes in a format that I can apply to another collection.

For example I have one existing user collection with indexes _id_, name_1, email_1 and website_1
Then I have another collection called usertest, I want to copy indexes from user collection to usertest collection. The following commands works for this scenario:
Copy both index key and index options
var indexes = db.user.getIndexes();
indexes.forEach(function(index){
delete index.v;
delete index.ns;
var key = index.key;
delete index.key
var options = [];
for (var option in index) {
options.push(index[option]);
}
db.usertest.createIndex(key, options);
});
Copy index key only (batch processing)
var indexKeys = db.user.getIndexKeys();
db.usertest.createIndexes(indexKeys);
Hope this will be helpful. Here's the doc: createIndexes

To do this directly in MongoDB do the following,
The following command will generate mongo DB queries for existing indexes of all collections,
db.getCollectionNames().forEach(function(col) {
var indexes = db[col].getIndexes();
indexes.forEach(function (c) {
var fields = '', result = '', options = {};
for (var i in c) {
if (i == 'key') {
fields = c[i];
} else if (i == 'name' && c[i] == '_id_') {
return;
} else if (i != 'name' && i != 'v' && i != 'ns') {
options[i] = c[i];
}
}
var fields = JSON.stringify(fields);
var options = JSON.stringify(options);
if (options == '{}') {
result = "db." + col + ".createIndex(" + fields + "); ";
} else {
result = "db." + col + ".createIndex(" + fields + ", " + options + "); ";
}
result = result
.replace(/{"floatApprox":-1,"top":-1,"bottom":-1}/ig, '-1')
.replace(/{"floatApprox":(-?\d+)}/ig, '$1')
.replace(/\{"\$numberLong":"(-?\d+)"\}/ig, '$1');
print(result);
});
});
The above command will output something like the following, based on the amount of collection you have
db.User.createIndex({"createdAt":-1}, {"background":true});
db.User.createIndex({"updatedAt":-1}, {"background":true});
db.Login.createIndex({"loginDate":-1}, {"background":true});
So after executing this, copy the MongoDB queries that are generated above to create the indexes to the new collection, Change the collection name in that then execute it.
For eg: to copy all indexes belonging to the User collection to the UserNew collection, I will rename the query's old collection name to new like the following and execute it, that is it, now you have all the indexes copied to a new collection from the old one.
db.UserNew.createIndex({"createdAt":-1}, {"background":true});
db.UserNew.createIndex({"updatedAt":-1}, {"background":true});
Credits: http://aleksandrmaiorov.com/2019/04/29/mongo-how-to-copy-indexes-from-one-database-to-another/

Thank you for the answer from Rocky and Bloke which helped me a lot
here is the consolidated version as suggested by Bloke.
and in PRODUCTION. we would like to make sure the background: true
is used to avoid slave halt query when indexes creation replicated.
var indexes = db.user.getIndexes();
// we skipped the __id__ indexes and set the default background: true option
indexes.forEach(function(index){
if(index.name =='_id_'){
print("we are skip the _id_ index")
}else{
delete index.v;
delete index.ns;
var key = index.key;
delete index.key
var options = {};
for (var option in index) {
options[option] = index[option]
}
options['background'] = true;
printjson(key);
printjson(options);
db.usertest.createIndex(key, options);
}
});

Rocky Li's answer was helpful but did not create the index options properly at the time of writing (It gathered the option values but not the keys). The following modification worked for me:
var indexes = db.user.getIndexes();
indexes.forEach(function(index){
delete index.v;
delete index.ns;
var key = index.key;
delete index.key
// uncomment if you want to ensure creation is in background
//if(!('background' in index))
//index['background'] = true;
db.user.createIndex(key, index);
});

Copy all indexes from one database another database
use firstDbName;
var indexKeyArray = [];
db.getCollectionNames().forEach(function(collection) {
var indexKeys = db[collection].getIndexKeys();
var base = {};
base["name"] = collection;
base["indices"] = indexKeys
indexKeyArray.push(base);
});
#printjson(indexKeyArray);
use destinationDbName;
indexKeyArray.forEach(function(data) {
db[data.name].createIndexes(data.indices);
});

Related

MongoDB : How to find multiple documents and update at the same time?

I have mongo DB and I am using C#.Net to interact with mongo db. C# API has methods for finding a single document and updating it at the same time. For example FindOneAndUpdateAsync.
However I couldn't find any method to find multiple documents and update them at the same time asynchronously.
The code below finding and processing each document asynchronously. How do I also update that document at the same time?
public async Task<IList<IDictionary<string, string>>> DoWork()
{
var collection = _mongoDatabase.GetCollection<BsonDocument>("units");
var filterBuilder = Builders<BsonDocument>.Filter;
var filter = filterBuilder.Ne<string>("status", "INIT") &
(filterBuilder.Exists("isDone", false) |
filterBuilder.Eq<bool>("isDone", false));
// I want to pass this update filter to update the document. But not sure how
var update = Builders<BsonDocument>.Update
.CurrentDate("startTime");
var sort = Builders<BsonDocument>.Sort.Ascending("startTime");
var projection = Builders<BsonDocument>.Projection
.Include("_id")
.Include("fileName"); // for bravity i have removed other projection elements
var output = new List<IDictionary<string, string>>();
// How do i pass update filter and update the document at the same time??
await collection
.Find(filter)
.Sort(sort)
.Project(projection)
.ForEachAsync((unit) =>
{
var dictionary = new Dictionary<string, string>();
Recurse(unit, dictionary);
output.Add(dictionary);
});
return output.Count > 0 ? output : null;
}
That doesn't exist in the mongo .Net api see here.
Just use a combination of Find and UpdateManyAsync.

Checking if an Index exists in mongodb

Is there a command that i can use via javascript in mongo shell that can be used to check if the particular index exists in my mongodb. I am building a script file that would create indexes. I would like that if I run this file multiple number of times then the indexes that already exists are not recreated.
I can use db.collection.getIndexes() to get the collection of all the indexes in my db and then build a logic to ignore the ones that already exists but i was wondering if there is command to get an index and then ignore a script that creates the index. Something like:
If !exists(db.collection.exists("indexname"))
{
create db.collectionName.CreateIndex("IndexName")
}
Creating indexes in MongoDB is an idempotent operation. So running db.names.createIndex({name:1}) would create the index only if it didn't already exist.
The deprecated (as of MongoDB 3.0) alias for createIndex() is ensureIndex() which is a bit clearer on what createIndex() actually does.
Edit:
Thanks to ZitRo for clarifying in comments that calling createIndex() with the same name but different options than an existing index will throw an error MongoError: Index with name: **indexName** already exists with different options as explained in this question.
If you have other reasons for checking, then you can access current index data one of two ways:
As of v3.0, we can use db.names.getIndexes() where names is the name of the collection. Docs here.
Before v3.0, you can access the system.indexes collection and do a find as bri describes below.
Use db.system.indexes and search on it.
If, for example, you have an index called 'indexname', you can search for it like this:
db.system.indexes.find({'name':'indexname'});
If you need to search for that index on a specific collection,then you need to use the ns property (and, it would be helpful to have the db name).
db.system.indexes.find({'name':'indexname', 'ns':'dbname.collection'});
Or, if you absolutely hate including the db name...
db.system.indexes.find({'name':'indexname', 'ns': {$regex:'.collection$'}});
Pulling that together...
So, you're finished check would be:
if(db.system.indexes.find({name:'indexname',ns:{$regex:'.collection$'}}).count()==0) {
db.collection.createIndex({blah:1},{name:'indexname'})
}
Using nodeJS MongoDB driver version 2.2:
const MongoClient = require('mongodb').MongoClient;
exports.dropOldIndexIfExist = dropOldIndexIfExist;
async function dropOldIndexIfExist() {
try {
const mongoConnection = MongoClient.connect('mongodb://localhost:27017/test');
const indexName = 'name_1';
const isIndexExist = await mongoConnection.indexExists(indexName);
if (isIndexExist === true) {
await mongoConnection.dropIndex(indexName);
}
} catch (err) {
console.error('dropOldIndexIfExist', err.message);
throw err;
}
}
I've created a custom method in c# to check if the index exists, using mongo driver:
public bool IndexExists<TDocument>(
IMongoCollection<TDocument> collection, string name)
{
var indexes = collection.Indexes.List().ToList();
var indexNames = indexes
.SelectMany(index => index.Elements)
.Where(element => element.Name == "name")
.Select(name => name.Value.ToString());
return indexNames.Contains(name);
}
maybe we can use something like https://docs.mongodb.com/v3.2/reference/method/db.collection.getIndexes/#db.collection.getIndexes to check if the collection have an index equal to something ?
if yes then drop and add the new one or add the new one directly
In my case i did as follows.
DBCollection yourcollectionName = mt.getCollection("your_collection");
if (yourcollectionName.getIndexInfo() == null || yourcollectionName.getIndexInfo().isEmpty()) {
DBObject indexOptions = new BasicDBObject();
indexOptions.put("pro1", 1);
indexOptions.put("pro2", 1);
yourcollectionName.createIndex(indexOptions, "name_of_your_index", true);
}
Here is a Python 3.5+ and pymongo >= 4.1 (type hints) function that I wrote which checks to see if the index name exists (other details about the index are omitted).
from pymongo import MongoClient
from pymongo.collection import Collection
def check_collection_indexes(db: MongoClient, collection_name: str, index_name: str) -> bool:
coll: Collection = db[collection_name]
indexes: dict = coll.index_information()
# assume false
found = False
# get length of the index name for substring
l = len(index_name)
for k in indexes.keys():
# Substring the keys and check for match
if k[:l] == index_name:
found = True
break
else:
found = False
return found
If the index exists it will return True, otherwise you can use the False output to call another function that creates/recreates the indexes.

Query without condition in MongoDB + C#

I'm trying to use the collection.FindAndModify and give it a IMongoQuery which selects all the documents. But I can not find how to create a query without any conditions!
Can anyone tell me how to do this? I'm using MongoDB C# Driver v1.8.3.
Here's my code:
var query = ???;
var sortBy = SortBy.Ascending(new string[] { "last_update" });
var update = Update<Entity>.Set(e => e.last_update, DateTime.Now);
var fields = Fields.Include(new string[] { "counter", "_id" });
var m = collection.FindAndModify(query, sortBy, update, fields, false, false);
I wonder what should I write in place of ??? to select all the documents!?
Use an empty QueryDocument:
var query = new QueryDocument();
But keep in mind that FindAndModify will only modify the first matching document.

MongoDB c# driver slow find on index

Got a 50+ million of documents and non-unique index on field "Base.UserID"
Two mongo servers in replicaset and connection string :
<add name="MongoConnectionString" connectionString="mongodb://mango1,mango2:27017" />
Index insured:
var eventCollection = Collection<EventMongo>();
eventCollection.EnsureIndex(IndexKeys.Ascending("Base.UserID"), IndexOptions.SetName("Event.Base.UserID"));
Then I do find
var _Set = new SortedSet<Int64>();
using (var db = new BaseDataAccess())
{
var col = db.Collection<EventMongo>();
var counter = 0;
var query = Query.And(
Query.EQ("Base.UserID", UserID),
Query.EQ("Base.Visible", 1)
);
var _docs = col.Find(query);
_docs.SetFields(new[] {"SQLId"});
_docs.SetSortOrder(SortBy.Descending("SQLId"));
_docs.SetLimit(HowMany);
int i = 0;
foreach (var doc in _docs)
{
var _EventID = doc.SQLId;
_Set.Add(_EventID);
if (++counter >= HowMany) break;
}
}
return _Set;
Same documents contains a parrallel MS SQL db, and i mention that the first query for read to MongoDB takes more time (up to 5 seconds) than MS SQL !
(The second hit on the same UserID is faster)
Your index is not being used. If you are querying by "Base.UserID" and "Base.Visible", then you need a compound index with both fields.
"scanAndOrder" : true, you want this to be false if possible.
I believe you might want to add your sort to the index.
MongoBlog on Index Ordering
eventCollection.EnsureIndex(IndexKeys.Ascending("Base.UserID"), IndexKeys.Descending("SQLId"), IndexOptions.SetName("Event.Base.UserID"));

Performance question about Mongo database

today I have tested the Mongo database, but I got a performance issue.
After I insert 1.800.00, I tried to make a sum of all values but it too 57s.
Then I tried the same thing in MSSQL and took 0s!!
Can you give any tips what I'm doing wrong?
Is this a Mango limitation?
static void Main(string[] args)
{
//Create a default mongo object. This handles our connections to the database.
//By default, this will connect to localhost, port 27017 which we already have running from earlier.
var connStr = new MongoConnectionStringBuilder();
connStr.ConnectTimeout = new TimeSpan(1, 0, 0);
connStr.SocketTimeout = new TimeSpan(1, 0, 0);
connStr.Server = new MongoServerAddress("localhost");
var mongo = MongoServer.Create(connStr);
//Get the blog database. If it doesn't exist, that's ok because MongoDB will create it
//for us when we first use it. Awesome!!!
var db = mongo.GetDatabase("blog");
var sw = new Stopwatch();
sw.Start();
//Get the Post collection. By default, we'll use the name of the class as the collection name. Again,
//if it doesn't exist, MongoDB will create it when we first use it.
var collection = db.GetCollection<Post>("Post");
Console.WriteLine(collection.Count());
sw.Stop();
Console.WriteLine("Time: " + sw.Elapsed.TotalSeconds);
sw.Reset();
sw.Start();
var starting = collection.Count();
var batch = new List<Post>();
for (int i = starting; i < starting + 200000; i++)
{
var post = new Post
{
Body = i.ToString(),
Title = "title " + i.ToString(),
CharCount = i.ToString().Length,
CreatedBy = "user",
ModifiedBy = "user",
ModifiedOn = DateTime.Now,
CreatedOn = DateTime.Now
};
//collection.Insert<Post>(post);
batch.Add(post);
}
collection.InsertBatch(batch);
Console.WriteLine(collection.Count());
sw.Stop();
Console.WriteLine("Time to insert 100.000 records: " + sw.Elapsed.TotalSeconds);
//var q = collection.Find(Query.LT("Body", "30000")).ToList();
//Console.WriteLine(q.Count());
sw.Reset();
sw.Start();
var q2 = collection.AsQueryable<Post>();
var sum = q2.Sum(p => p.CharCount);
Console.WriteLine(sum);
sw.Stop();
Console.WriteLine("Time to sum '" + q2.Count() + "' Post records: " + sw.Elapsed.TotalSeconds); //PROBLEM: take 57 to SUM 1.000.000 records
}
}
Performance issue in the following row:
var q2 = collection.AsQueryable<Post>();
In row above you loading all posts from the posts collection into memory, because of driver does not support linq. In MSSQL it's taking less than second because of linq and calculating will go through the database. Here i guess almost all 57 second need to load data into memory.
In mongodb to achieve best performance you need to create extra fields (de normalize data) and calculate any sums,counters, etc whenever it possible. If it not possible you need to use map/reduce or available aggregate functions, like group (good fit for your example of sum calculation).