Flask -- Reloading database which was loaded before first request - mongodb

I have an API run on flask which is connected to MongoDB and it uses this DB for reading only.
I connect to db before first request:
#app.before_first_request
def load_dicti():
c = MongoClient('mongodb://' + app.config['MONGO_DSN'], connect=False)
db = c.my_name
app.first, app.second = dictionary_compilation(db.my_base, another_dictionary)
However, this mongodb may be updated from time to time. My API doesn't know about it because this db was already loaded before first request.
What's the most efficient way to cope with it? I'd be grateful for explanations and code examples.

I don't quite figure out what you are going to do, but Application Context may be best practice. Just like demo in Flask docs, you could do:
def get_db():
"""Opens a new database connection if there is none yet for the
current application context.
"""
if not hasattr(g, 'db'):
c = MongoClient('mongodb://' + app.config['MONGO_DSN'], connect=False)
g.db = c.my_name
return g.db
Then, you could use get_db() directly in your view function, mongdb will be conntected once only when there is no db attr in g.
If your connection is not that stable that you need to change it everytime, you could connect every request or every session.

Related

Flask SQl-Alchemy multiple DB transactions in single request

I am trying to make a single route Flask web app in which the user submits a URL to the client, some data from the URL is extracted at the server-side and appended to a Postgres DB. Next, further processing is done on the data extracted from the URL and the data entry of the URL in the DB is updated. All this processing is done as background tasks using celery. The work flow looks something like this:
#app.route
def app_route(url):
chain(task1(url) | task2(url))
#celery.task
def task1(url):
out1 = some_long_task(url)
append url,out1 to db
#celery.task
def task2(url):
out2 = some_other_long_task(url)
update url row in db with out2
The reason we do this is that, these two tasks are long tasks, and the user should be able to see the status of the task in the client. Hence, we update the out1 in the DB first so the user can know the status and then with out2 so the user can know the final status. At any point, the user can visit the home page which displays the URLs in the DB currently with their data.
This throws me an error: psycopg2.OperationalError) SSL error: decryption failed or bad record mac
The url with out1 is appended to the DB correctly without issue. But the second time when we try to update the row we appended in the previous task, it throws the above error. I am guessing flask sqlalchemy only allows a single session to a DB in one request, hence the error. What can I do to solve this issue?
I solved the error. Switched from flask-sqlalchemy to sqlalchemy to connect to the database since it offers more visibility. One should dispose the engine object by calling engine.dispose() after connecting the app to the database and the error disappears.

How can I performance test a Postgres Database with Gatling?

I'm using Gatling to performance test an application, but the application has a Postgres database, which relies on jdbc to connect. Is it possible to run a scenario to test the database performance, and if so, how?
I found the jdbcFeeder, but I do not know how to execute the scenario I've set up, since the exec doesn't accept the url I provide it... jdbc:postgresql://host:port/database?user=<user>&password=<password>. It returns a java.lang.IllegalArgumentException: could not be parsed into a proper Uri, missing host.
Example:
val sqlQueryFeeder: RecordSeqFeederBuilder[Any] =
jdbcFeeder(connectionUrl, user, password, s"SELECT * FROM $schema.$table")
val scn: ScenarioBuilder = scenario("Test 1")
.feed(sqlQueryFeeder)
.exec(http.baseUrl(jdbc:postgresql://host:port/database?user=<user>&password=<password>))
)
setUp(scn.inject(atOnceUsers(100)))
Not out of the box.
jdbcFeeder is about driving performance tests with data read from a database (i.e. it's about "feeding" a test with data, not about testing a database).
The gatling-jdbc project appears to be intended to do what you want, but it didn't work when I tried it out (Gatling 3.0-RC4). I'm pretty new to Gatling and Scala, the problem might be me, or it might even by the Gatling release candidate - you might have better luck if you try it out.
There's also gatling-sql but I've not tried it out myself and it hasn't been touched in over a year.

Azure SQL Server (elastic pool) connect slow

We ar running a webapplication in Azure Web Apps using a database per customer (multiple accounts per customer). When logging in we connect the user to the correct customer database. This database is also hosted in azure (an elastic pool). It is hosted in the same region (West Europe) as the Web App.
Once the connection is pooled, request times are fast, but the first time a user log's in, the connection still needs to be created an this takes (quiet) a long time.
The connectionstring is build up using a SqlConnectionStringBuilder.
var csb = new System.Data.SqlClient.SqlConnectionStringBuilder();
csb.DataSource = "tcp:******.database.windows.net,1433";
csb.InitialCatalog = "***-***-***";
csb.UserID = "**-**";
csb.Password = "**********";
csb.MultipleActiveResultSets = true;
csb.Encrypt = true;
csb.PersistSecurityInfo = false;
csb.TrustServerCertificate = false;
csb.ConnectTimeout = 30;
_connectionString = csb.ConnectionString;
// Data Source=tcp:******.database.windows.net,1433;Initial Catalog=***-***-***;Persist Security Info=False;User ID=**-***;Password=******;MultipleActiveResultSets=True;Connect Timeout=30;Encrypt=True;TrustServerCertificate=False
Am I doing anything wrong? Or are there some settings in azure to speed up the connect process?
The above request shows the first request to the application of a customer. It therefor includes the EF Migration Seed resulting in the first 2 queries not actually going to the database itself and quite a lot of queries (not all shown here) to the database.
Well, I solved my problem eventualy. Seems i was matching wrong queries within Applications Insights. I installed Stackify and this gives just the little bit more information I needed.
Seem's Entity Framework does some things with the 'master' database. As the user in the connectionstring did not have access to the 'master' database it throws an error. Well, handling that error take's up quite some time on the app-service used and therefor returning slow. It just doesn't fail.
What EF tries to do is determine if the database exist by querying the master database wich is faster then connecting to a non existing database. If it fails because it can not connect to the master database, EF just tries to connect to the database itself. If connection to the database works, it continues normal execution like the seed method.

How to perform an "I can reach my database" healthcheck?

I have a classic spray+slick http server which is my database access layer, and I'd like to be able to have an healthcheck route to ensure my server is still able to reach my DB.
I could do it by doing a generic sql query, but I was wondering if there was a better way to just check the connection is alive and usable without actually adding load on the database (or at least the minimum possible load).
So pretty much :
val db = Database.forConfig("app.mydb")
[...]
db.???? // Do the check here
Why do you want to avoid executing a query against the database?
I think the best health check is to actually use the database as your application would (actually connecting and running a query). With that in mind, you can perform a SELECT 1 against your DB, and verify that it responds accordingly.

Simultaneous identical requests = double in database

My controller handles a POST request, inserting a object in PostgreSQL. It works like this :
Check if the object does not exist in DB
Save the object in DB
But sometimes, 2 identical requests come too close and I guess the second one does not find the object of the first one so both are written in DB.
I am on heroku and I must take scalability into account also : the two requests can come on different dynos ( static variable wont work )
I did not find anything about database locking in Play
Any idea ?