In the following code:
val users = TableQuery[Users]
def getUserById(id:Int) = db(users.filter(_.id === id).result)
From what I understand, getUserById would create a prepared statement everytime the getUserById is executed and then discarded. Is there a way to cache the prepared statement so it is created only once and called many times.
The documentation for Slick indicates that you need to enable prepared statement caching on the connection pool configuration.
Here is quite a good article on it also.
The summary is that Slick seems to cache the strings that are used to prepare the actual statements, but delegates the caching of the actual prepared statements to the underlying connection pool implementation.
Related
I'm struggling to write py.test fixtures for managing my app's database that maximize speed, supports pytest-xdist parallelization of tests, and isolates the tests from each other.
I'm using Flask-SQLAlchemy 2.1 against a PostgreSQL 9.4 database.
Here's the general outline of what I'm trying to accomplish:
$ py.test -n 3 spins up three test sessions for running tests.
Within each session, a py.test fixture runs once to setup a transaction, create the database tables, and then at the end of the session it rolls back the transaction. Creating the database tables needs to happen within a PostgreSQL transaction that's only visible to that particular test-session, otherwise the parallelized test sessions created by pytest-xdist cause conflicts with each other.
A second py.test fixture that runs for every test connects to the existing transaction in order to see the created tables, creates a nested savepoint, runs the test, then rolls back to the nested savepoint.
Ideally, these pytest fixtures support tests that call db.session.rollback(). There's a potential recipe for accomplishing this at the bottom of this SQLAlchemy doc.
Ideally the pytest fixtures should yield the db object, not just the session so that
folks can write tests without having to remember to use a session that's
different than the standard db.session they use throughout the app.
Here's what I have so far:
import pytest
# create_app() is my Flask application factory
# db is just 'db = SQLAlchemy()' + 'db.init_app(app)' within the create_app() function
from app import create_app, db as _db
#pytest.yield_fixture(scope='session', autouse=True)
def app():
'''Session-wide test application'''
a = create_app('testing')
with a.app_context():
yield a
#pytest.yield_fixture(scope='session')
def db_tables(app):
'''Session-wide test database'''
connection = _db.engine.connect()
trans = connection.begin() # begin a non-ORM transaction
# Theoretically this creates the tables within the transaction
_db.create_all()
yield _db
trans.rollback()
connection.close()
#pytest.yield_fixture(scope='function')
def db(db_tables):
'''db session that is joined to existing transaction'''
# I am quite sure this is broken, but it's the general idea
# bind an individual Session to the existing transaction
db_tables.session = db_tables.Session(bind=db_tables.connection)
# start the session in a SAVEPOINT...
db_tables.session.begin_nested()
# yield the db object, not just the session so that tests
# can be written transparently using the db object
# without requiring someone to understand the intricacies of these
# py.test fixtures or having to remember when to use a session that's
# different than db.session
yield db_tables
# rollback to the savepoint before the test ran
db_tables.session.rollback()
db_tables.session.remove() # not sure this is needed
Here's the most useful references that I've found while googling:
http://docs.sqlalchemy.org/en/latest/orm/session_transaction.html#joining-a-session-into-an-external-transaction-such-as-for-test-suites
http://koo.fi/blog/2015/10/22/flask-sqlalchemy-and-postgresql-unit-testing-with-transaction-savepoints/
https://github.com/mitsuhiko/flask-sqlalchemy/pull/249
I'm a couple years late here, but you might be interested in pytest-flask-sqlalchemy, a plugin I wrote to help address this exact problem.
The plugin provides two fixtures, db_session and db_engine, which you can use like regular Session and Engine objects to run updates that will get rolled back at the end of the test. It also exposes a few configuration directives (mocked-engines and mocked-sessions) that will mock out connectables in your app and replace them with these fixtures so that you can run methods and be sure that any state changes will get cleaned up when the test exits.
The plugin should work with a variety of databases, but it's been tested most heavily against Postgres 9.6 and is in production in the test suite for https://dedupe.io. You can find some examples in the documentation that should help you get started, but if you're willing to provide some code I'd be happy to demonstrate how to use the plugin, too.
I had similar issue trying to combine yield fixtures. Unfortunately according to the doc you are not able to combine more than one yield level.
But you might be able to find a work around using request.finalizer:
#pytest.fixture(scope='session', autouse=True)
def app():
'''Session-wide test application'''
a = create_app('testing')
with a.app_context():
return a
#pytest.fixture(scope='session')
def db_tables(request, app):
'''Session-wide test database'''
connection = _db.engine.connect()
trans = connection.begin() # begin a non-ORM transaction
# Theoretically this creates the tables within the transaction
_db.create_all()
def close_db_session():
trans.rollback()
connection.close()
request.addfinalizer(close_db_session)
return _db
I am trying to query a MySQL database asynchronously using Slick. The following code template, which I use to query about 90k rows in a for comprehension, seems to be working initially, but the program consumes several gigabytes of RAM and fails without warning after around 200 queries.
import scala.slick.jdbc.{StaticQuery => Q}
def doQuery(): Future[List[String]] = future {
val q = "select name from person"
db withSession {
Q.query[String](q).list
}
}
I have tried setting up connections both using the fromURL method and also using a c3p0 connection pool. My question is: Is this the way to do asynchronous calls to the database?
Async is still an open issue for Slick.
You could try using Iterables and stream data instead of storing it in memory with a solution similar to this: Treating an SQL ResultSet like a Scala Stream
Although please omit the .toStream call at the end. It will cache the data in memory, while Iterable will not.
If you want an async version of iterable you could look into Observables.
It turns out that this is a non issue (actually a bug in my code, which opened a new database connection for each query). In my experience, you can wrap DB queries in Futures as shown above and compose them later with Scala Async or Rx, as shown here. All is required for good performance is a large thread pool (x2 the CPUs in my case) and an equally large connection pool.
Slick 3 (Reactive Slick) looks like it might address this.
I'm very new to Neo4j. I'm using version 2.0.0-M05.
Why does GraphDatabaseService.index() throw exception if it's called outside of transaction ? Can I make it work without transaction ?
Here is my sample code
val graphDb = new GraphDatabaseFactory().
newEmbeddedDatabaseBuilder("test_db").
setConfig(GraphDatabaseSettings.node_keys_indexable, "nodeProp1,nodeProp2").
setConfig(GraphDatabaseSettings.relationship_keys_indexable, "relProp1,relProp2").
setConfig(GraphDatabaseSettings.node_auto_indexing, "true").
setConfig(GraphDatabaseSettings.relationship_auto_indexing, "true").
newGraphDatabase()
val userIndex = graphDb.index().forNodes("user")
Causes following exception
Exception in thread "main" org.neo4j.graphdb.NotInTransactionException
at org.neo4j.kernel.impl.transaction.AbstractTransactionManager.assertInTransaction(AbstractTransactionManager.java:108)
at org.neo4j.kernel.IndexManagerImpl.assertInTransaction(IndexManagerImpl.java:465)
at org.neo4j.kernel.IndexManagerImpl.forNodes(IndexManagerImpl.java:300)
at org.neo4j.kernel.IndexManagerImpl.forNodes(IndexManagerImpl.java:293)
at Test$.main(Test.scala:77)
at Test.main(Test.scala)
PS. It doesn't happen in 1.9.RC1.
Looks like it's by design.
Previously it was considered good practice to wrap read operations inside a transaction. The 2.0.0-M04 release makes this recommendation mandatory. Any attempt to execute operations on the database outside of a transaction will now throw a NotInTransactionException. Enforcing this idiom allows Neo4j to reclaim resources much more efficiently leading to a database that will handle much more load. Cypher and the REST-API open their own transactions, so this will only affect users of the embedded Java-API.
I need to do small (but frequent) operations on my database, from one of my api methods. When I try wrapping them into "withSession" each time, I get terrible performance.
db withSession {
SomeTable.insert(a,b)
}
Running the above example 100 times takes 22 seconds. Running them all in a single session is instantaneous.
Is there a way to re-use the session in subsequent function invocations?
Do you have some type of connection pooling (see JDBC Connection Pooling: Connection Reuse?)? If not you'll be using a new connection for every withSession(...) and that is a very slow approach. See http://groups.google.com/group/scalaquery/browse_thread/thread/9c32a2211aa8cea9 for a description of how to use C3PO with ScalaQuery.
If you use a managed resource from an application server you'll usually get this for "free", but in stand-alone servers (for example jetty) you'll have to configure this yourself.
I'm probably stating the way too obvious, but you could just put more calls inside the withSession block like:
db withSession {
SomeTable.insert(a,b)
SomeOtherTable.insert(a,b)
}
Alternately you can create an implicit session, do your business, then close it when you're done:
implicit val session = db.createSession
SomeTable.insert(a,b)
SomeOtherTable.insert(a,b)
session.close
Can anybody please tell me how to handle a transaction rollback in squeryl explicitly?
And also how can we add or remove columns in squeryl dynamically?
Thanx...
Just to elaborate a bit on the response from #didierd. There is one Session/Connection bound to each transaction. You can access the current Session, and thereby the Connection with code like:
Session.currentSession.connection
Or, if you're not sure if you're within a transaction
Session.currentSessionOption map {_.connection}
If you do roll back the transaction this way it will be your responsibility to start a new one or make sure there is no further use of the connection, so use with care.
You have an access to the JDBC's java.sql.Connection (connection in Session), so if you really cannot use transaction / inTransaction, you can call rollback there.
With access to the connection, you can also execute arbitrary SQL requests and so change the database schema, but be mindful that your squeryl-using code has a static, compile time known schema.