I am trying to make a single route Flask web app in which the user submits a URL to the client, some data from the URL is extracted at the server-side and appended to a Postgres DB. Next, further processing is done on the data extracted from the URL and the data entry of the URL in the DB is updated. All this processing is done as background tasks using celery. The work flow looks something like this:
#app.route
def app_route(url):
chain(task1(url) | task2(url))
#celery.task
def task1(url):
out1 = some_long_task(url)
append url,out1 to db
#celery.task
def task2(url):
out2 = some_other_long_task(url)
update url row in db with out2
The reason we do this is that, these two tasks are long tasks, and the user should be able to see the status of the task in the client. Hence, we update the out1 in the DB first so the user can know the status and then with out2 so the user can know the final status. At any point, the user can visit the home page which displays the URLs in the DB currently with their data.
This throws me an error: psycopg2.OperationalError) SSL error: decryption failed or bad record mac
The url with out1 is appended to the DB correctly without issue. But the second time when we try to update the row we appended in the previous task, it throws the above error. I am guessing flask sqlalchemy only allows a single session to a DB in one request, hence the error. What can I do to solve this issue?
I solved the error. Switched from flask-sqlalchemy to sqlalchemy to connect to the database since it offers more visibility. One should dispose the engine object by calling engine.dispose() after connecting the app to the database and the error disappears.
Related
We ar running a webapplication in Azure Web Apps using a database per customer (multiple accounts per customer). When logging in we connect the user to the correct customer database. This database is also hosted in azure (an elastic pool). It is hosted in the same region (West Europe) as the Web App.
Once the connection is pooled, request times are fast, but the first time a user log's in, the connection still needs to be created an this takes (quiet) a long time.
The connectionstring is build up using a SqlConnectionStringBuilder.
var csb = new System.Data.SqlClient.SqlConnectionStringBuilder();
csb.DataSource = "tcp:******.database.windows.net,1433";
csb.InitialCatalog = "***-***-***";
csb.UserID = "**-**";
csb.Password = "**********";
csb.MultipleActiveResultSets = true;
csb.Encrypt = true;
csb.PersistSecurityInfo = false;
csb.TrustServerCertificate = false;
csb.ConnectTimeout = 30;
_connectionString = csb.ConnectionString;
// Data Source=tcp:******.database.windows.net,1433;Initial Catalog=***-***-***;Persist Security Info=False;User ID=**-***;Password=******;MultipleActiveResultSets=True;Connect Timeout=30;Encrypt=True;TrustServerCertificate=False
Am I doing anything wrong? Or are there some settings in azure to speed up the connect process?
The above request shows the first request to the application of a customer. It therefor includes the EF Migration Seed resulting in the first 2 queries not actually going to the database itself and quite a lot of queries (not all shown here) to the database.
Well, I solved my problem eventualy. Seems i was matching wrong queries within Applications Insights. I installed Stackify and this gives just the little bit more information I needed.
Seem's Entity Framework does some things with the 'master' database. As the user in the connectionstring did not have access to the 'master' database it throws an error. Well, handling that error take's up quite some time on the app-service used and therefor returning slow. It just doesn't fail.
What EF tries to do is determine if the database exist by querying the master database wich is faster then connecting to a non existing database. If it fails because it can not connect to the master database, EF just tries to connect to the database itself. If connection to the database works, it continues normal execution like the seed method.
I have an API run on flask which is connected to MongoDB and it uses this DB for reading only.
I connect to db before first request:
#app.before_first_request
def load_dicti():
c = MongoClient('mongodb://' + app.config['MONGO_DSN'], connect=False)
db = c.my_name
app.first, app.second = dictionary_compilation(db.my_base, another_dictionary)
However, this mongodb may be updated from time to time. My API doesn't know about it because this db was already loaded before first request.
What's the most efficient way to cope with it? I'd be grateful for explanations and code examples.
I don't quite figure out what you are going to do, but Application Context may be best practice. Just like demo in Flask docs, you could do:
def get_db():
"""Opens a new database connection if there is none yet for the
current application context.
"""
if not hasattr(g, 'db'):
c = MongoClient('mongodb://' + app.config['MONGO_DSN'], connect=False)
g.db = c.my_name
return g.db
Then, you could use get_db() directly in your view function, mongdb will be conntected once only when there is no db attr in g.
If your connection is not that stable that you need to change it everytime, you could connect every request or every session.
I have a multi-tenant application built in Laravel 5.1. It uses one main database connection for storing users, roles, permissions, as well as jobs and failed_jobs. Additionally, every user has his own database.
I use a Job class for sending mail, and when it is executed, the following exception occurs:
[PDOException] SQLSTATE[3D000]: Invalid catalog name: 1046 No database selected
The class uses tables from two db connections (the main connection, and the one associated with the current user).
Thanks in advance, any help is appreciated.
Ok, this was easy. For anyone who is interested, I totally forgot to set the database in the second connection.
In fact, the database field in the second db connection is dynamically filled, depending on the authenticated user. So, each time the job class is executed, there should be initialization of the database field:
Config::set('database.connections.second_connection.database', 'user_' . $user_id);
// $user_id is in fact auth()->user()->id, passed as parameter
That solves the problem.
I'm using Pyramid and SQLAlchemy for my REST server. For the logging purposes I need some way how to determine user name in postgresql after update trigger function. One way is setting some runtime configuration parameter before update execution and using this value in trigger function.
In psql terminal it can be done using
set myapp.user_name = 'User Name';
This parameter will be subsequently used in postgresql trigger.
Is it possible to set this parameter in Pyramid / SQLAlchemy application?
I suppose I can use SQLAlchemy Events. But I'm not sure which event is correct for this case.
You can register a Pyramid event handler which would set the parameter on any new request:
from pyramid import events
def on_new_request(event):
"""
you can access request as event.request
and the current registry settings as event.request.registry.settings
"""
session = DBSession()
session.execute("SET blah TO 'foo'")
def app(global_config, **settings):
config = Configurator(...)
config.add_subscriber(on_new_request, events.NewRequest)
See Deployment Settings and events.NewRequest for more details.
One thing to watch out is to make sure you're always using the same session object - SQLAlchemy maintains a pool of connections and, if you commit your session, all subsequent operations will use a brand-new session which is not pre-configured with your settings. In other words, you should not do session.commit(), session.rollback(), transaction.commit() etc. in your code and instead rely on ZopeTransactionExtension committing/rolling back the transaction at the end of the request/response cycle. Which is a recommended practice anyway.
I'm not sure what the exact command you're running with set myapp.user = "User Name".
When you say "In psql this can be done using..." I'm assuming you are running some sort of SQL command, although the command you listed is not a valid command, at least not typically.
In any event, arbitrary SQL can be executed in SQLAlchemy using the .execute() method.
session = Session()
session.execute("select 1;")
http://docs.sqlalchemy.org/en/rel_0_9/orm/session.html?highlight=execute#sqlalchemy.orm.session.Session.execute
My controller handles a POST request, inserting a object in PostgreSQL. It works like this :
Check if the object does not exist in DB
Save the object in DB
But sometimes, 2 identical requests come too close and I guess the second one does not find the object of the first one so both are written in DB.
I am on heroku and I must take scalability into account also : the two requests can come on different dynos ( static variable wont work )
I did not find anything about database locking in Play
Any idea ?