I am trying to move my web blog-like app from Flask to Quart which aparently could significantly improve the performance.
However, I am not able to replicate flask_mongoengine behaviour. So far I tried AsyncIOMotorClient and quart-motor.
If I reduce my code to the core issue, it seems that the issue is here:
from motor.motor_asyncio import AsyncIOMotorClient
client = AsyncIOMotorClient("mongodb://localhost:27017")
db= client['pets-api']
print(db)
doc = db.pet.find_one({})
print(doc)
returns:
AsyncIOMotorDatabase(Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=False, driver=DriverInfo(name='Motor', version='2.3.1', platform='asyncio')), 'pets-api'))
<Future pending cb=[_chain_future.<locals>._call_check_cancel() at C:\Python39\lib\asyncio\futures.py:384]>
It doesn't throw an error, but I am not able to query any documents from my collections. Does connect=False indicate some sort of issue?
In pymongo this code works perfectly fine:
from pymongo import MongoClient
client = MongoClient('localhost', 27017)
db = client['pets-api']
print(list(db.pet.find({})))
What am I missing?
This has finally worked:
async def db_connection():
client = motor.motor_asyncio.AsyncIOMotorClient('mongodb://127.0.0.1:27017')
db = client['pets-api']
return db
async def do_find_one():
db = await db_connection()
document = await db.user.find_one()
pprint.pprint(document)
loop = asyncio.get_event_loop()
loop.run_until_complete(do_find_one())
Related
I've been trying out Javascript for my new backend, but I noticed something odd. When I use Python's PyMongo library, fetching all my data runs twice as fast (33.5s -> 16.44s) as when I use the official Javascript MongoDB module. The test setups are as follows:
mongo = MongoClient(URI) # initializing client
rn = time() # starting timer
for ID in LIST_OF_IDS:
results = mongo["current_day_info"][ID].find() # fetches all the documents in the collection named <ID>
results = [[result["time"].timestamp() / 10, result["buy_price"], result["sell_price"], result["buy_volume"], result["sell_volume"], result["week_buy_amount"], result["week_sell_amount"]] for result in results] # expands list of documents into array of their values
print("mongodb", time()-rn) # ending timer
const client = new MongoClient(URI); // initializing client
async function main() {
await client.connect(); // connecting to client
const database = client.db("current_day_info"); // goes to proper database
console.time("mongodb"); // starting timer
for (let ID of LIST_OF_IDS) {
let results = [];
const documents = database.collection(ID).find({}); // fetches all the documents in the collection named <ID>
documents.forEach(item => results.push([new Date(item.time).getTime(), item.buy_price, item.sell_price, item.buy_volume, item.sell_volume, item.week_buy_amount, item.week_sell_amount])); // expands list of documents into array of their values
}
console.timeEnd("mongodb"); // ending timer
}
main();
My best guess is that PyMongo has some parts written in C, but I wouldn't think that results in such a drastic increase?
Versions:
Python 3.8.10, PyMongo 4.0.2, MongoDB version 5.0.6, MongoDB Node Driver version 4.4.1, NodeJS version 17.7.1
Copied from comments below:
Example document:
{
"time":"ISODate(""2022-03-12T23:23:45.000Z"")",
"_id":"ObjectId(""622d2b8c83d4c06e792895cb"")",
"buy_volume":2079625,
"sell_price":5.5,
"sell_volume":10210328,
"week_sell_amount":68184205,
"week_buy_amount":10783950,
"buy_price":7.4
}
Also, tried the same in MongoSH and it took significantly longer (7 minutes) I assume I messed something up there:
async function main() {
console.time("mongodb");
for (let ID of LIST_OF_IDS) {
let results = [];
const collection = db[ID];
await collection.find({}).forEach((item) => results.push([cut for charlimit]));
}
console.timeEnd("mongodb");
}
I'm currently encountring a performance issue with play slick hitting a Postgresql database on AWS RDS. The app was running quite well 2 days ago but now some queries are freezing the app.
A join query with a result of only 8 rows would take up to 2300ms and one resulting with 30 row is taking more than 4000ms.
Here is the query performed as an example:
protected def jobOfferQuery = for {
(((mission, client), process), clientCompany) <- jobOffers
.joinLeft(users).on(_.clientId === _.userId)
.joinLeft(processes).on(_._1.offerId === _.jobOfferId)
.joinLeft(companies)
} yield (mission, client, process, clientCompany)
Here is my slick configuration for my play app:
database {
dataSourceClass = org.postgresql.ds.PGSimpleDataSource
properties = {
serverName = ${app.postgres.host}
portNumber = ${app.postgres.port}
databaseName = ${app.postgres.db}
user = ${app.postgres.user}
password = ${app.postgres.password}
}
poolName = ${app.postgres.poolName}
numThreads=5
driver="slick.driver.PostgresDriver$"
}
How can it be so slow ? Should I add more hikarycp configuration ?
Thank you so much for helping me out, much appreciated ..
I am trying to set the maxPoolSize via connection string in MongoDB following this piece of documentation. Here is my connection string:
mongodb://localhost:27017/databaseName?maxPoolSize=200
However, instead of having the database databaseName with the maxPoolSize equals to 200, I'm getting a database called databaseName?maxPoolSize=200. This is, Mongo is getting everything (name + options) as the database name.
Some info:
Mongo version: 3.2.10
Connecting using Morphia 1.1.0
I will be happy to provide any further information.
if you are doing
MongoClient client = new MongoClient(
"mongodb://localhost:27017/databaseName?maxPoolSize=200");
then dont do that, instead do as following,
MongoClient client = new MongoClient(
new MongoClientURI(
"mongodb://localhost:27017/databaseName?maxPoolSize=200"));
because you need to tell mongo that you are passing some options along the connection string.
if you think i misunderstood your question. please post the piece of code where you are trying to get a connection.
You can try something like this.
MongoClientURI uri = new MongoClientURI("mongodb://localhost:27017/databaseName?maxPoolSize=200");
MongoClient mongoClient = new MongoClient(uri);
Morphia morphia = new Morphia();
Datastore datastore = morphia.createDatastore(mongoClient, "dbname");
Alternatively
MongoClientOptions.Builder options = new MongoClientOptions.Builder();
//set your connection option here.
options.connectionsPerHost(200); //max pool size
MongoClient mongoClient = new MongoClient(new ServerAddress("localhost", 27017), options.build());
Morphia morphia = new Morphia();
Datastore datastore = morphia.createDatastore(mongoClient, "dbname");
I'm having a problem with lots of connections being opened to the mongo db.
The readme on the Github page for the C# driver gives the following code:
using MongoDB.Bson;
using MongoDB.Driver;
var client = new MongoClient("mongodb://localhost:27017");
var server = client.GetServer();
var database = server.GetDatabase("foo");
var collection = database.GetCollection("bar");
collection.Insert(new BsonDocument("Name", "Jack"));
foreach(var document in collection.FindAll())
{
Console.WriteLine(document["Name"]);
}
At what point does the driver open the connection to the server? Is it at the GetServer() method or is it the Insert() method?
I know that we should have a static object for the client, but should we also have a static object for the server and database as well?
Late answer... but the server connection is created at this point:
var client = new MongoClient("mongodb://localhost:27017");
Everything else is just getting references for various objects.
See: http://docs.mongodb.org/ecosystem/tutorial/getting-started-with-csharp-driver/
While using the latest MongoDB drivers for C#, the connection happens at the actual database operation. For eg. db.Collection.Find() or at db.collection.InsertOne().
{
//code for initialization
//for localhost connection there is no need to specify the db server url and port.
var client = new MongoClient("mongodb://localhost:27017/");
var db = client.GetDatabase("TestDb");
Collection = db.GetCollection<T>("testCollection");
}
//Code for db operations
{
//The connection happens here.
var collection = db.Collection;
//Your find operation
var model = collection.Find(Builders<Model>.Filter.Empty).ToList();
//Your insert operation
collection.InsertOne(Model);
}
I found this out after I stopped my mongod server and debugged the code with breakpoint. Initialization happened smoothly but error was thrown at db operation.
Hope this helps.
Is there a possibility to check if a mongo database allready exists?
Yes, you can get the list of existing databases. From the Java driver you could do something like this to get the database names on a mongod server running on localhost
Mongo mongo = new Mongo( "127.0.0.1", 27017 );
List<String> databaseNames = mongo.getDatabaseNames();
This is equivalent to the mongo shell "show dbs" command. I am sure similar methods exist in all of the drivers.
From the shell, if you want to explicitely check that a DB exists:
db.getMongo().getDBNames().indexOf("mydb");
Will return '-1' if "mydb" does not exist.
To use this from the shell:
if [ $(mongo localhost:27017 --eval 'db.getMongo().getDBNames().indexOf("mydb")' --quiet) -lt 0 ]; then
echo "mydb does not exist"
else
echo "mydb exists"
fi
For anyone who comes here because the method getDatabaseNames(); is depreciated / not available, here is the new way to get the list of existing databases:
MongoClient mongoClient = new MongoClient();
MongoCursor<String> dbsCursor = mongoClient.listDatabaseNames().iterator();
while(dbsCursor.hasNext()) {
System.out.println(dbsCursor.next());
}
Here is a method that validates if the database is found:
public Boolean databaseFound(String databaseName){
MongoClient mongoClient = new MongoClient(); //Maybe replace it with an already existing client
MongoCursor<String> dbsCursor = mongoClient.listDatabaseNames().iterator();
while(dbsCursor.hasNext()) {
if(dbsCursor.next().equals(databaseName))
return true;
}
return false;
}
in python using Pymongo
from pymongo import MongoClient
db_name = "foo"
conn = MongoClient('mongodb://localhost,localhost:27017')
db = self.conn[str(db_name)]
if bool(db_name in conn.database_names()):
collection.drop()
using MongoDb c# Driver 2.4
private bool DatabaseExists(string database)
{
// _client is IMongoClient
var dbList = _client.ListDatabases().ToList().Select(db => db.GetValue("name").AsString);
return dbList.Contains(database);
}
usage:
if (!DatabaseExists("FooDb")
{
// create and seed db
}
I'd like to add a C# version. I'm using the MongoDB.Driver 2.2.2.
static bool DatabaseExists(string connectionString)
{
var mongoUri = new MongoUrl(connectionString);
var client = new MongoClient(mongoUri);
var dbList = Enumerate(client.ListDatabases()).Select(db => db.GetValue("name").AsString);
return dbList.Contains(mongoUri.DatabaseName);
}
static IEnumerable<BsonDocument> Enumerate(IAsyncCursor<BsonDocument> docs)
{
while (docs.MoveNext())
{
foreach (var item in docs.Current)
{
yield return item;
}
}
}
Try this, it worked for me (on Mac OSx)
MongoClient mongoClient = new MongoClient("localhost");
/** **/
boolean dbExist =
mongoClient.listDatabaseNames().
into(new ArrayList<String>()).contains("TEST");
System.out.print(dbExist);
The PyMongo example above didn't work for me, so I rewrote it using the more standard list_databases() method to the MongoClient library:
from pymongo import MongoClient
db_name = "foo"
conn = MongoClient('mongodb://localhost,localhost:27017')
if bool(db_name in conn.list_databases()):
print true # or return true here
else:
print false # or return false here
In my case, I could not use listDatabaseNames, because my user did not have the rights to call this function. Instead, I just assume that it exists and call a method on this database, which will fail if it does not exist or if rights are missing.
Demo C# code:
/// <summary>
/// Tests the connection to the MongoDB server, and if the database already exists.
/// If not or an error is detected, an exception is thrown.
/// </summary>
public static void TestConnection(string mongoUrl, string mongoDatabase) {
var client = new MongoClient(mongoUrl);
var database = client.GetDatabase(mongoDatabase);
try {
// Try to perform an action on this database; will fail if it does not exist
database.ListCollections();
}
catch {
throw new Exception("Connection established, " +
"but database does not exist (or missing rights): " + mongoDatabase);
}
}
I searched for how to list database names in golang and accidentially found this page. Looks like no one has provided ways to check if specific database name exists.
Assume that you are using MongoDB Go Driver, here is an approach
// client type is *mongo.Client
dbNames, err := client.ListDatabaseNames(context.Background(), bson.D{{Key: "name", Value: "YOUR-DB-NAME"}})
"dbNames" contains list of all databases in mongo server.
Second parameter is a filter, in this case, it only allows database with name == "YOUR-DB-NAME". Thus, dbNames will be empty if "YOUR-DB-NAME" is not exist.