Following the recommended transaction setup for Squeryl, in my Boot.scala:
import net.liftweb.squerylrecord.SquerylRecord
import org.squeryl.Session
import org.squeryl.adapters.H2Adapter
SquerylRecord.initWithSquerylSession(Session.create(
DriverManager.getConnection("jdbc:h2:lift_proto.db;DB_CLOSE_DELAY=-1", "sa", ""),
new H2Adapter
))
The first startup works fine. I can connect via H2's web-interface and if I use my app, it updates the database appropriately. However if I restart jetty without restarting the JVM, I get:
java.sql.SQLException: No suitable driver found for jdbc:h2:lift_proto.db;DB_CLOSE_DELAY=-1
The same result is had if I replace "DB_CLOSE_DELAY=-1" with "AUTO_SERVER=TRUE", or remove it entirely.
Following the recommendations on the Squeryl list, I tried C3P0:
import com.mchange.v2.c3p0.ComboPooledDataSource
val cpds = new ComboPooledDataSource
cpds.setDriverClass("org.h2.Driver")
cpds.setJdbcUrl("jdbc:h2:lift_proto")
cpds.setUser("sa")
cpds.setPassword("")
org.squeryl.SessionFactory.concreteFactory =
Some(() => Session.create(
cpds.getConnection, new H2Adapter())
)
This produces similar behavior:
WARNING: A C3P0Registry mbean is already registered. This probably means that an application using c3p0 was undeployed, but not all PooledDataSources were closed prior to undeployment. This may lead to resource leaks over time. Please take care to close all PooledDataSources.
To be sure it wasn't anything I was doing which was causing this, I started and stopped the server without calling a transaction { } block. No exceptions were thrown. I then added to my Boot.scala:
transaction { /* Do nothing */ }
And the exception was once again thrown (I'm assuming because connections are lazy). So I moved the db initialization code to its own file away from Lift:
SessionFactory.concreteFactory = Some(()=>
Session.create(
java.sql.DriverManager.getConnection("jdbc:h2:mem:test", "sa", ""),
new H2Adapter
))
transaction {}
Results were unchanged. What am I doing wrong? I cannot find any mention of needing to explicitly close connections or sessions in the Squeryl documentation, and this is my first time using JDBC.
I found mention of the same issue here on the Lift google group, but no resolution.
Thanks for any help.
When you say you are restarting Jetty, I think what you're actually doing is reloading your webapp within Jetty. Neither the h2 database or C3P0 will automatically shut down when your app reloads, which explains the errors you are receiving when Lift tries to initialize them a second time. You don't see the error when you don't create a transaction block because both h2 and C3P0 are initialized when the first DB connection is retrieved.
I tend to use BoneCP as a connection pool myself. You can configure the minimum number of pooled connections to be > 1, which will stop h2 from shutting down without the need for DB_CLOSE_DELAY=-1. Then you can use:
LiftRules.unloadHooks append { () =>
yourPool.close() //should destroy the pool and it's associated threads
}
That will close all of the connections when Lift is shutdown, which should properly shutdown the h2 database as well.
Related
I have a ASP.NET Core application with npgsql that is setup to retry when a database connection fails:
services.AddDbContext<DatabaseContext>(
opt => opt.UseNpgsql(Configuration.GetConnectionString("DefaultConnection"),
(options) =>
{
options.EnableRetryOnFailure(10);
})
);
However, when i restart the database during a request, it will first throw an exception
Exception thrown: 'Npgsql.PostgresException' in System.Private.CoreLib.dll: '57P01: terminating connection due to administrator command'
That propogates into my code and causes an exception without any retries.
Looking into the documentation for npgsql it seems like some exceptions are marked as IsTransient, and that means they will be retried, but this one is not. I assume there's a reason for that, but I'd still like to be able to restart the postgres database without interrupting requests by simply having them retry after a while. I could create try catches around all the requests and check for this particular exception, but that seems like a clunky solution.
How do I handle this situation gracefully?
npgsql package used:
<PackageReference Include="Npgsql.EntityFrameworkCore.PostgreSQL" Version="5.0.10" />
Managed to find an answer myself, you can add the error code in the EnableRetryOnFailure
services.AddDbContext<DatabaseContext>(
opt => opt.UseNpgsql(Configuration.GetConnectionString("DefaultConnection"),
(options) =>
{
options.EnableRetryOnFailure(10, TimeSpan.FromSeconds(30), new string[] { "57P01" });
})
);
Now the only question that remains is if that is a good idea or not.
Context
I'm creating Flask app connected to mongodb using MongoEngine via flask-mongoengine extension. I create my app using application factory pattern as specified in configuration instructions.
Problem
While running test(s), I specified testing database named datazilla_test which is passed to mongo instance via mongo.init_app(app). Even though my app.config['MONGODB_DB'] and mongo.app.config['MONGODB_DB'] instance has correct value (datazilla_test), this value is not reflected in mongo instance. Thus, when I run assertion assert mongo.get_db().name == mongo.app.config['MONGODB_DB'] this error is triggered AssertionError: assert 'datazzilla' == 'datazzilla_test'
Question
What am I doing wrong? Why database connection persist with default database datazzilla rather, than datazilla_test? How to fix it?
Source Code
# __init__.py
from flask_mongoengine import MongoEngine
mongo = MongoEngine()
def create_app(config=None):
app = Flask(__name__)
app.config['MONGODB_HOST'] = 'localhost'
app.config['MONGODB_PORT'] = '27017'
app.config['MONGODB_DB'] = 'datazzilla'
# override default config
if config is not None:
app.config.from_mapping(config)
mongo.init_app(app)
return app
# conftest.py
import pytest
from app import mongo
from app import create_app
#pytest.fixture
def app():
app = create_app({
'MONGODB_DB': 'datazzilla_test',
})
assert mongo.get_db().name == mongo.app.config['MONGODB_DB']
# AssertionError: assert 'datazzilla' == 'datazzilla_test'
return app
Mongoengine is already connected when your fixture is called, when you call app = create_app from your fixture, it tries to re-establish the connection but fails silently (as it sees that there is an existing default connection established).
This got reworked in the development version of mongoengine (see https://github.com/MongoEngine/mongoengine/pull/2038) but wasn't released yet (As of 04-JUN-2019). When that version gets out, you'll be able to disconnect any existing mongoengine connection by calling disconnect_all
In the meantime, you can either:
- check where the existing connection is created and prevent it
- try to disconnect the existing connection by using the following:
from mongoengine.connection import disconnect, _connection_settings
#pytest.fixture
def app():
disconnect()
del _connection_settings['default']
app = create_app(...)
...
But it may have other side effects
Context
Coincidentally, I figure-out fix for this problem. #bagerard answer is correct! It works for MongoClient where client's connect is set to True -this is/should be default value.
MongoClient(host=['mongo:27017'], document_class=dict, tz_aware=False, connect=False, read_preference=Primary())
If that is the case, then you have to disconnect database and delete connection settings as #bagerard explains.
Solution
However, if you change MongoClient connection to False, then you don't have to disconnect database and delete connection settings. At the end solution that worked for me was this solution.
def create_app(config=None):
...
app.config['MONGODB_CONNECT'] = False
...
Notes
As I wrote earlier. I found this solution coincidentally, I was trying to solve this problem MongoClient opened before fork. Create MongoClient only after forking. It turned out that it fixes both problems :)
P.S If there are any side effects I'm not aware of them at this point! If you find some then please share them in comments section.
In an existing Grails 3.1.15 application, recently upgraded to Hibernate 5.1, an odd issue with afterInsert (or other) hooks started to appear. After some hours of testing, I could track this down to Hibernate 5.1 + PostgreSQL combination - issue is not reproducible with H2. To reproduce, create a simple application consisting of 2 domain objects - an Audit trail and a User, as shown here:
class Audit {
Date dateCreated
String auditData
}
class User {
String name
String email
def afterInsert() {
new Audit(auditData: "User created: $this").save()
}
}
The code above works OK with Hibernate 4, however, if the application is upgraded to Hibernate5 plugin + Hibernate 5.1.x (tested with 5.1.0.Final and 5.1.5.Final) the above scenario will always lead to a ConcurrentModificationException when you attempt to save a new User instance. You can just use a scaffold controller to reproduce. Note this only happens with PostgreSQL as the data source - with H2 it would still work OK.
Now, according to GORM Docs (see chapter 8.1.3) one should use a new session when attempting to save other objects in beforeUpdate or afterInsert hooks anyway:
def afterInsert() {
Audit.withNewSession() {
new Audit(auditData: "User created: $this").save()
/* no exception logged, but Audit instance not preserved */
}
}
But this wouldn't really resolve the issue with PSQL. The exception is gone, the User instance is persisted, but the Audit instance is not saved. Again, this code would work OK with H2. To really avoid the issue with PSQL, you would have to manually flush the session in afterInsert:
def afterInsert() {
Audit.withNewSession() { session ->
new Audit(auditData: "User created: $this").save()
session.flush()
/* finally no exceptions, both User and Audit saved */
}
}
Question: is this a bug, or is this expected? I find it a bit suspicious that the manual flush is required in order for the Audit instance to be persisted - and even more so when I see it works without a flush with H2 and only seems to affect PostgreSQL. But I couldn't really find any reports - any pointers are appreciated!
For the sake of completeness, I tested with the following JDBC driver versions for PostgreSQL:
runtime 'org.postgresql:postgresql:9.3-1101-jdbc41'
runtime 'org.postgresql:postgresql:9.4.1208.jre7'
runtime 'org.postgresql:postgresql:42.0.0'
And for the upgrade to Hibernate 5.1, the following dependencies were used:
classpath "org.grails.plugins:hibernate5:5.0.13"
...
compile "org.grails.plugins:hibernate5:5.0.13"
compile "org.hibernate:hibernate-core:5.1.5.Final"
compile "org.hibernate:hibernate-ehcache:5.1.5.Final"
TLDR:
Lots of TCP connections in OPEN_WAIT status shutting down server
Setup:
riak_1.2.0-1_amd64.deb installed on Ubuntu12
Spring MVC 3.2.5
riak-client-1.1.0.jar
Tomcat7.0.51 hosted on Windows Server 2008 R2
JRE6_45
Full Description:
How do I ensure that the Java RiakClient is properly cleaning up it's connections to that I'm not left with an abundance of CLOSE_WAIT tcp connections?
I have a Spring MVC application which uses the Riak java client to connect to the remote instance/cluster.
We are seeing a lot of TCP Connections on the server hosting the Spring MVC application, which continue to build up until the server can no longer connect to anything because there are no ports available.
Restarting the Riak cluster does not clean the connections up.
Restarting the webapp does clean up the extra connections.
We are using the HTTPClientAdapter and REST api.
When connecting to a relational database, I would normally clean up connections by either explicitly calling close on the connection, or by registering the datasource with a pool and transaction manager and then Annotating my Services with #Transactional.
But since using the HTTPClientAdapter, I would have expected this to be more like an HttpClient.
With an HttpClient, I would consume the Response entity, with EntityUtils.consume(...), to ensure that the everything is properly cleaned up.
HTTPClientAdapter does have a shutdown method, and I see it being called in the online examples.
When I traced the method call through to the actual RiakClient, the method is empty.
Also, when I dig through the source code, nowhere in it does it ever close the Stream on the HttpResponse or consume any response entity (as with the standard Apache EntityUtils example).
Here is an example of how the calls are being made.
private RawClient getRiakClientFromUrl(String riakUrl) {
return new HTTPClientAdapter(riakUrl);
}
public IRiakObject fetchRiakObject(String bucket, String key, boolean useCache) {
try {
MethodTimer timer = MethodTimer.start("Fetch Riak Object Operation");
//logger.debug("Fetching Riak Object {}/{}", bucket, key);
RiakResponse riakResponse;
riakResponse = riak.fetch(bucket, key);
if(!riakResponse.hasValue()) {
//logger.debug("Object {}/{} not found in riak data store", bucket, key);
return null;
}
IRiakObject[] riakObjects = riakResponse.getRiakObjects();
if(riakObjects.length > 1) {
String error = "Got multiple riak objects for " + bucket + "/" + key;
logger.error(error);
throw new RuntimeException(error);
}
//logger.debug("{}", timer);
return riakObjects[0];
}
catch(Exception e) {
logger.error("Error fetching " + bucket + "/" + key, e);
throw new RuntimeException(e);
}
}
The only option I can think of, is to create the RiakClient separately from the adapter so I can access the HttpClient and then the ConnectionManager.
I am currently working on switching over to the PBClientAdapter to see if that might help, but for the purposes of this question (and because the rest of the team may not like me switching for whatever reason), let's assume that I must continue to connect over HTTP.
So it's been almost a year, so I thought I would go ahead and post how I solved this problem.
The solution was to change the client implementation we were using to the HTTPClientAdapter provided by the java client, passing in the configuration to implement pools and max connections. Here's some code example of how to do it.
First, we are on an older version of RIAK, so here's the amven dependency:
<dependency>
<groupId>com.basho.riak</groupId>
<artifactId>riak-client</artifactId>
<version>1.1.4</version>
</dependency>
And here's the example:
public RawClient riakClient(){
RiakConfig config = new RiakConfig(riakUrl);
//httpConnectionsTimetolive is in seconds, but timeout is in milliseconds
config.setTimeout(30000);
config.setUrl("http://myriakurl/);
config.setMaxConnections(100);//Or whatever value you need
RiakClient client = new RiakClient(riakConfig);
return new HTTPClientAdapter(client);
}
I actually broke that up a bit in my implementation and used Spring to inject values; I just wanted to show a simplified example for it.
By setting the timeout to something less than the standard five minutes, the system will not hang to the connections for too long (so, 5 minutes + whatever you set the timeout to) which causes the connectiosn to enter the close_wait status sooner.
And of course setting the max connections in the pool prevents the application from opening up 10's of thousands of connections.
For the relevant part of our server stack, we're running:
NGINX 1.2.3
PHP-FPM 5.3.10 with PECL mongo 1.2.12
MongoDB 2.0.7
CentOS 6.2
We're getting some strange, but predictable behavior when the MongoDB server goes away (crashes, gets killed, etc). Even with a try/catch block around the connection code, i.e:
try
{
$mdb = new Mongo('mongodb://localhost:27017');
}
catch (MongoConnectionException $e)
{
die( $e->getMessage() );
}
$db = $mdb->selectDB('collection_name');
Depending on which PHP-FPM workers have connected to mongo already, the connection state is cached, causing further exceptions to go unhandled, because the $mdb connection handler can't be used. The troubling thing is that the try does not consistently fail for a considerable amount of time, up to 15 minutes later, when -- I assume -- the php-fpm processes die/respawn.
Essentially, the behavior is that when you hit a worker that hasn't connected to mongo yet, you get the die message above, and when you connect to a worker that has, you get an unhandled exception from $mdb->selectDB('collection_name'); because catch does not run.
When PHP is a single process, i.e. via Apache with mod_php, this behavior does not occur. Just for posterity, going back to Apache/mod_php is not an option for us at this time.
Is there a way to fix this behavior? I don't want the connection state to be inconsistent between different php-fpm processes.
Edit:
While I wait for the driver to be fixed in this regard, my current workaround is to do a quick polling to determine if the driver can handle requests and then load or not load the MongoDB library/run queries if it can't connect/query:
try
{
// connect
$mongo = new Mongo("mongodb://localhost:27017");
// try to do anything with connection handle
try
{
$mongo->YOUR_DB->YOUR_COLLECTION->findOne();
$mongo->close();
define('MONGO_STATE', TRUE);
}
catch(MongoCursorException $e)
{
$mongo->close();
error_log('Error connecting to MongoDB: ' . $e->getMessage() );
define('MONGO_STATE', FALSE);
}
}
catch(MongoConnectionException $e)
{
error_log('Error connecting to MongoDB: ' . $e->getMessage() );
define('MONGO_STATE', FALSE);
}
The PHP mongo driver connectivity code is getting a big overhaul in the 1.3 release, currently in beta2 as of writing this. Based on your description, your issues may be resolved by the fixes for:
https://jira.mongodb.org/browse/PHP-158
https://jira.mongodb.org/browse/PHP-465
Once it is released you will be able to see the full list of fixes here:
https://jira.mongodb.org/browse/PHP/fixforversion/10499
Or, alternatively on the PECL site. If you can test 1.3 and confirm that your issues are still present then I'm sure the driver devs would love to hear from you before the 1.3.0 release, especially if it is easily reproducible.