today I have tested the Mongo database, but I got a performance issue.
After I insert 1.800.00, I tried to make a sum of all values but it too 57s.
Then I tried the same thing in MSSQL and took 0s!!
Can you give any tips what I'm doing wrong?
Is this a Mango limitation?
static void Main(string[] args)
{
//Create a default mongo object. This handles our connections to the database.
//By default, this will connect to localhost, port 27017 which we already have running from earlier.
var connStr = new MongoConnectionStringBuilder();
connStr.ConnectTimeout = new TimeSpan(1, 0, 0);
connStr.SocketTimeout = new TimeSpan(1, 0, 0);
connStr.Server = new MongoServerAddress("localhost");
var mongo = MongoServer.Create(connStr);
//Get the blog database. If it doesn't exist, that's ok because MongoDB will create it
//for us when we first use it. Awesome!!!
var db = mongo.GetDatabase("blog");
var sw = new Stopwatch();
sw.Start();
//Get the Post collection. By default, we'll use the name of the class as the collection name. Again,
//if it doesn't exist, MongoDB will create it when we first use it.
var collection = db.GetCollection<Post>("Post");
Console.WriteLine(collection.Count());
sw.Stop();
Console.WriteLine("Time: " + sw.Elapsed.TotalSeconds);
sw.Reset();
sw.Start();
var starting = collection.Count();
var batch = new List<Post>();
for (int i = starting; i < starting + 200000; i++)
{
var post = new Post
{
Body = i.ToString(),
Title = "title " + i.ToString(),
CharCount = i.ToString().Length,
CreatedBy = "user",
ModifiedBy = "user",
ModifiedOn = DateTime.Now,
CreatedOn = DateTime.Now
};
//collection.Insert<Post>(post);
batch.Add(post);
}
collection.InsertBatch(batch);
Console.WriteLine(collection.Count());
sw.Stop();
Console.WriteLine("Time to insert 100.000 records: " + sw.Elapsed.TotalSeconds);
//var q = collection.Find(Query.LT("Body", "30000")).ToList();
//Console.WriteLine(q.Count());
sw.Reset();
sw.Start();
var q2 = collection.AsQueryable<Post>();
var sum = q2.Sum(p => p.CharCount);
Console.WriteLine(sum);
sw.Stop();
Console.WriteLine("Time to sum '" + q2.Count() + "' Post records: " + sw.Elapsed.TotalSeconds); //PROBLEM: take 57 to SUM 1.000.000 records
}
}
Performance issue in the following row:
var q2 = collection.AsQueryable<Post>();
In row above you loading all posts from the posts collection into memory, because of driver does not support linq. In MSSQL it's taking less than second because of linq and calculating will go through the database. Here i guess almost all 57 second need to load data into memory.
In mongodb to achieve best performance you need to create extra fields (de normalize data) and calculate any sums,counters, etc whenever it possible. If it not possible you need to use map/reduce or available aggregate functions, like group (good fit for your example of sum calculation).
Related
I have read the following two questions but did not see a satisfactory solution or suggestion:
Performance degrade with Mongo when using bulkwrite with upsert
Mongodb/Mongoose bulkwrite(upsert) performance issues
I have an API that will be receiving ~80k objects every 5 - 6 seconds.
These objects will need to be inserted if they do not exist and updated if they do exist.
The collection has a unique index (of type asc 1), which I use in the filter expression below.
Upserting 100k documents using the code below takes more than a minute, which is too long for my requirement (instantiating the objects takes less than a second).
MongoClient _mongoClient = new MongoClient("connection string");
IMongoDatabase _hotsauceOdds = _mongoClient.GetDatabase("DatabaseName");
var collection = _hotsauceOdds.GetCollection<Person>("TestMongo");
List<Person> peeps = new List<Person>();
for (int i = 0; i < 100000; i++)
{
peeps.Add(new Person { Age = i, Name = Guid.NewGuid().ToString() });
}
var models = peeps.Select(item => new ReplaceOneModel<Person>(new ExpressionFilterDefinition<Person>(doc => doc.Name == item.Name), item) { IsUpsert = true });
await collection.BulkWriteAsync(models);
I am testing this on the free tier of mongo atlas.
Is it possible to make this operation run more quickly, or is this as good as it gets for mongodb?
I know there's a way to do db.collection.getIndexes() Which will list all the indexes defined for a collection. Is there a way to copy and create those index definitions to another collection?
There's a lot of them and I don't want to do them one by one.
regarding the duplicated question comment: I do not wish to copy a collection. I wish to export indexes in a format that I can apply to another collection.
For example I have one existing user collection with indexes _id_, name_1, email_1 and website_1
Then I have another collection called usertest, I want to copy indexes from user collection to usertest collection. The following commands works for this scenario:
Copy both index key and index options
var indexes = db.user.getIndexes();
indexes.forEach(function(index){
delete index.v;
delete index.ns;
var key = index.key;
delete index.key
var options = [];
for (var option in index) {
options.push(index[option]);
}
db.usertest.createIndex(key, options);
});
Copy index key only (batch processing)
var indexKeys = db.user.getIndexKeys();
db.usertest.createIndexes(indexKeys);
Hope this will be helpful. Here's the doc: createIndexes
To do this directly in MongoDB do the following,
The following command will generate mongo DB queries for existing indexes of all collections,
db.getCollectionNames().forEach(function(col) {
var indexes = db[col].getIndexes();
indexes.forEach(function (c) {
var fields = '', result = '', options = {};
for (var i in c) {
if (i == 'key') {
fields = c[i];
} else if (i == 'name' && c[i] == '_id_') {
return;
} else if (i != 'name' && i != 'v' && i != 'ns') {
options[i] = c[i];
}
}
var fields = JSON.stringify(fields);
var options = JSON.stringify(options);
if (options == '{}') {
result = "db." + col + ".createIndex(" + fields + "); ";
} else {
result = "db." + col + ".createIndex(" + fields + ", " + options + "); ";
}
result = result
.replace(/{"floatApprox":-1,"top":-1,"bottom":-1}/ig, '-1')
.replace(/{"floatApprox":(-?\d+)}/ig, '$1')
.replace(/\{"\$numberLong":"(-?\d+)"\}/ig, '$1');
print(result);
});
});
The above command will output something like the following, based on the amount of collection you have
db.User.createIndex({"createdAt":-1}, {"background":true});
db.User.createIndex({"updatedAt":-1}, {"background":true});
db.Login.createIndex({"loginDate":-1}, {"background":true});
So after executing this, copy the MongoDB queries that are generated above to create the indexes to the new collection, Change the collection name in that then execute it.
For eg: to copy all indexes belonging to the User collection to the UserNew collection, I will rename the query's old collection name to new like the following and execute it, that is it, now you have all the indexes copied to a new collection from the old one.
db.UserNew.createIndex({"createdAt":-1}, {"background":true});
db.UserNew.createIndex({"updatedAt":-1}, {"background":true});
Credits: http://aleksandrmaiorov.com/2019/04/29/mongo-how-to-copy-indexes-from-one-database-to-another/
Thank you for the answer from Rocky and Bloke which helped me a lot
here is the consolidated version as suggested by Bloke.
and in PRODUCTION. we would like to make sure the background: true
is used to avoid slave halt query when indexes creation replicated.
var indexes = db.user.getIndexes();
// we skipped the __id__ indexes and set the default background: true option
indexes.forEach(function(index){
if(index.name =='_id_'){
print("we are skip the _id_ index")
}else{
delete index.v;
delete index.ns;
var key = index.key;
delete index.key
var options = {};
for (var option in index) {
options[option] = index[option]
}
options['background'] = true;
printjson(key);
printjson(options);
db.usertest.createIndex(key, options);
}
});
Rocky Li's answer was helpful but did not create the index options properly at the time of writing (It gathered the option values but not the keys). The following modification worked for me:
var indexes = db.user.getIndexes();
indexes.forEach(function(index){
delete index.v;
delete index.ns;
var key = index.key;
delete index.key
// uncomment if you want to ensure creation is in background
//if(!('background' in index))
//index['background'] = true;
db.user.createIndex(key, index);
});
Copy all indexes from one database another database
use firstDbName;
var indexKeyArray = [];
db.getCollectionNames().forEach(function(collection) {
var indexKeys = db[collection].getIndexKeys();
var base = {};
base["name"] = collection;
base["indices"] = indexKeys
indexKeyArray.push(base);
});
#printjson(indexKeyArray);
use destinationDbName;
indexKeyArray.forEach(function(data) {
db[data.name].createIndexes(data.indices);
});
I am retrieving data from mongoDB using C# driver, It is taking a lot of time when i do to list Please help me
My Mongoquery is
var documentReportIds = new BsonValue[] { LatestReportIds };
var documentChennelIds = new BsonValue[] { Cid };
var documentPropertyIds = new BsonValue[] { Pid };
IMongoQuery query = new QueryDocument();
query = Query.And(Query.GTE("CheckInDate", startdate.Date.AddMinutes(330)),
Query.LTE("CheckInDate", endDate.Date.AddMinutes(330)));
query = Query.And(query, Query.EQ("SubscriberPropertyId", reportFilter.SubscriberPropertyId));
query = Query.And(query, Query.EQ("LengthOfStay", reportFilter.LOS));
query = Query.And(query, Query.In("ReportId", documentReportIds));
query = Query.And(query, Query.In("ChannelId", documentChennelIds));
query = Query.And(query, Query.In("PropertyId", documentPropertyIds));
MongoDBEntities<ScheduleOptimizationReportDetails> _obj = new MongoDBEntities<ScheduleOptimizationReportDetails>();
var list= _obj.GetSchedularOptimizationJoin(query);
Class from where it perform data retrieving
public class MongoDBEntities<T>
{
MongoDatabase db = MongoDBInstance.GetMongoDatabase;
public List GetSchedularOptimizationJoin(IMongoQuery query)
{
MongoCollection MCollection = db.GetCollection(“Subscription_OptimisedReports”);
MongoCursor cursor = MCollection.FindAs(query).SetFields(Fields.Include(“ScheduleLogId”, “SubscriberPropertyId”, “CheckInDate”, “ReportId”, “CreatedDate”));
List entities = cursor.ToList();
return entities ;
}
}
what is another option to select data in C#, I have also applied indexing on column.
Please help me how to solve it.
You can use the MongoDB.Driver.Linq package to help you create your querys with Linq expressions.
I'm trying to use the collection.FindAndModify and give it a IMongoQuery which selects all the documents. But I can not find how to create a query without any conditions!
Can anyone tell me how to do this? I'm using MongoDB C# Driver v1.8.3.
Here's my code:
var query = ???;
var sortBy = SortBy.Ascending(new string[] { "last_update" });
var update = Update<Entity>.Set(e => e.last_update, DateTime.Now);
var fields = Fields.Include(new string[] { "counter", "_id" });
var m = collection.FindAndModify(query, sortBy, update, fields, false, false);
I wonder what should I write in place of ??? to select all the documents!?
Use an empty QueryDocument:
var query = new QueryDocument();
But keep in mind that FindAndModify will only modify the first matching document.
Got a 50+ million of documents and non-unique index on field "Base.UserID"
Two mongo servers in replicaset and connection string :
<add name="MongoConnectionString" connectionString="mongodb://mango1,mango2:27017" />
Index insured:
var eventCollection = Collection<EventMongo>();
eventCollection.EnsureIndex(IndexKeys.Ascending("Base.UserID"), IndexOptions.SetName("Event.Base.UserID"));
Then I do find
var _Set = new SortedSet<Int64>();
using (var db = new BaseDataAccess())
{
var col = db.Collection<EventMongo>();
var counter = 0;
var query = Query.And(
Query.EQ("Base.UserID", UserID),
Query.EQ("Base.Visible", 1)
);
var _docs = col.Find(query);
_docs.SetFields(new[] {"SQLId"});
_docs.SetSortOrder(SortBy.Descending("SQLId"));
_docs.SetLimit(HowMany);
int i = 0;
foreach (var doc in _docs)
{
var _EventID = doc.SQLId;
_Set.Add(_EventID);
if (++counter >= HowMany) break;
}
}
return _Set;
Same documents contains a parrallel MS SQL db, and i mention that the first query for read to MongoDB takes more time (up to 5 seconds) than MS SQL !
(The second hit on the same UserID is faster)
Your index is not being used. If you are querying by "Base.UserID" and "Base.Visible", then you need a compound index with both fields.
"scanAndOrder" : true, you want this to be false if possible.
I believe you might want to add your sort to the index.
MongoBlog on Index Ordering
eventCollection.EnsureIndex(IndexKeys.Ascending("Base.UserID"), IndexKeys.Descending("SQLId"), IndexOptions.SetName("Event.Base.UserID"));