I'm trying to follow the steps listed at http://dev.mysql.com/doc/connector-j/en/connector-j-master-slave-replication-connection.html which states
To enable this functionality, use the com.mysql.jdbc.ReplicationDriver
class when configuring your application server's connection pool
From https://github.com/brettwooldridge/HikariCP - it says
HikariCP will attempt to resolve a driver through the DriverManager
based solely on the jdbcUrl
So is this configuration all thats needed?
db.default.url=jdbc:mysql:replication ...
Squeryl has has a number of db Adapters; but my understanding is these are unrelated?
http://squeryl.org/api/index.html#org.squeryl.adapters.MySQLInnoDBAdapter
Sorry for the key word loading - I'm just not too sure where I need to focus
Thanks
Brent
For people hitting this in 2020, Hikari uses
com.mysql.jdbc.jdbc2.optional.MysqlDataSource
as a data source. If I look at the code of the above class. It has a method named connect which returns Connection instance.
protected Connection getConnection(Properties props) throws SQLException {
String jdbcUrlToUse = null;
if (!this.explicitUrl) {
StringBuffer jdbcUrl = new StringBuffer("jdbc:mysql://");
if (this.hostName != null) {
jdbcUrl.append(this.hostName);
}
jdbcUrl.append(":");
jdbcUrl.append(this.port);
jdbcUrl.append("/");
if (this.databaseName != null) {
jdbcUrl.append(this.databaseName);
}
jdbcUrlToUse = jdbcUrl.toString();
} else {
jdbcUrlToUse = this.url;
}
Properties urlProps = mysqlDriver.parseURL(jdbcUrlToUse, (Properties)null);
urlProps.remove("DBNAME");
urlProps.remove("HOST");
urlProps.remove("PORT");
Iterator keys = urlProps.keySet().iterator();
while(keys.hasNext()) {
String key = (String)keys.next();
props.setProperty(key, urlProps.getProperty(key));
}
return mysqlDriver.connect(jdbcUrlToUse, props);
}
where mysqlDriver is an instance of
protected static final NonRegisteringDriver mysqlDriver;
if i check the connect method of NonRegisteringDriver class. It looks like this
public Connection connect(String url, Properties info) throws SQLException {
if (url != null) {
if (StringUtils.startsWithIgnoreCase(url, "jdbc:mysql:loadbalance://")) {
return this.connectLoadBalanced(url, info);
}
if (StringUtils.startsWithIgnoreCase(url, "jdbc:mysql:replication://")) {
return this.connectReplicationConnection(url, info);
}
}
Properties props = null;
if ((props = this.parseURL(url, info)) == null) {
return null;
} else if (!"1".equals(props.getProperty("NUM_HOSTS"))) {
return this.connectFailover(url, info);
} else {
try {
com.mysql.jdbc.Connection newConn = ConnectionImpl.getInstance(this.host(props), this.port(props), props, this.database(props), url);
return newConn;
} catch (SQLException var6) {
throw var6;
} catch (Exception var7) {
SQLException sqlEx = SQLError.createSQLException(Messages.getString("NonRegisteringDriver.17") + var7.toString() + Messages.getString("NonRegisteringDriver.18"), "08001", (ExceptionInterceptor)null);
sqlEx.initCause(var7);
throw sqlEx;
}
}
}
After looking at the code, it looks like it supports. I haven't tried it till now. Will try and let you know from personal experience. From code, it looks directly feasible.
Squeryl offers different MySQL adapters because innodb supports referential keys, while myisam does not. It seems like what your'e doing should be handled at the connection pool level, so I don't think your Squeryl configuration will have an affect.
I've never configured Hikari for replicated MySQL, but if it requires an alternative JDBC driver I'd be surprised if you can provide a JDBC URL and everything just works. I'm guessing that Hikari's default functionality is to pick the plain vanilla MySQL JDBC driver unless you tell it otherwise. Luckily, Hikari has quite a few config options including the ability to set a specific driverClassName.
Replication allows for a different URL:
jdbc:mysql:replication://[server1],[server2],[server2]/[database]
I've never tried it, but I assume this will resolve to the ReplicationDriver.
And I find myself back here - please note, hikari doesn't support the Replication driver.
https://github.com/brettwooldridge/HikariCP/issues/625#issuecomment-251613688
MySQL Replication Driver simply does NOT work together with HikariCP.
And
https://groups.google.com/forum/#!msg/hikari-cp/KtKgzR8COrE/higEHoPkAwAJ
... nobody running anything resembling a mission critical application takes MySQL's driver-level replication support seriously.
Related
We have two .net services (.Net core console applications) which are accessing a postgres db table.
Service 1 inserts some 500 rows every 1 minute. It runs as a background thread.
Service 2 reads data from the same table continuously. There is an MQTT publisher which keeps reading data from this table when any new data is requested. This also happens very frequently i.e atleast 4/5 times a minute.
We are getting "FATAL: sorry, too many clients already " error.
What I am assuming is since write and read is happening simultaneously too frequently, the connection is not getting dispose properly.
Is there a way to avoid read whenever a write is happening.
EDITED
Thanks for the reply.. I know some connection pooling is happening but not sure where.. so my question was how to avoid concurrent access of postgres db..
Was not sure what part of code I can post to make the question clear
I am having using clause on dbcontext and also disposed like the below..
This is retrieval section
using (PlatinumDBContext platinumDBContext = new PlatinumDBContext())
{
try
{
var data = platinumDBContext.TrendPoints.Where(x => ids.Contains(x.TrendPointID) && x.TimeStamp >= DateTime.Now.AddHours(-timeinHours));
result = data.Select(x => new Last24hours
{
Label = x.TrendPointID.ToString(),
Value = (double)x.TrendPointValue,
time = x.TimeStamp.ToString("MM/dd/yyyy HH:mm:ss")
}).ToList();
}
catch (Exception oE)
{
}
finally {
platinumDBContext.Dispose();
}
}
This is the insertion section
using (PlatinumDBContext platinumDBContext = new PlatinumDBContext())
{
try
{
foreach (var point in trendPoints)
{
if (point != null)
{
TrendPoint item = new TrendPoint();
item.CreatedDate = DateTime.Now;
item.ObjectState = ObjectState.Added;
item.TrendPointID = point.TrendID;
item.TrendPointValue = double.IsNaN(point.Value) ? decimal.MinValue : (decimal)point.Value;
item.TimeStamp = new DateTime(point.TimeStamp);
platinumDBContext.Add(item);
}
}
platinumDBContext.SaveChanges();
}
catch (Exception ex)
{
}
finally
{
platinumDBContext.Dispose();
}
}
Regards,
Geervani
In SqlSessionTemplate,getSqlSession and closeSqlSession method all are operate SqlSessionHolder,but i think is not must to do this,Spring manager the transaction and provide ConnectionHolder,transaction is linked about connection.
SqlSessionHolder holder = (SqlSessionHolder) getResource(sessionFactory);
if (holder != null && holder.isSynchronizedWithTransaction()) {
if (holder.getExecutorType() != executorType) {
throw new TransientDataAccessResourceException("Cannot change the ExecutorType when there is an existing transaction");
}
holder.requested();
if (logger.isDebugEnabled()) {
logger.debug("Fetched SqlSession [" + holder.getSqlSession() + "] from current transaction");
}
return holder.getSqlSession();
}
if (logger.isDebugEnabled()) {
logger.debug("Creating a new SqlSession");
}
SqlSession session = sessionFactory.openSession(executorType);
why we need SqlSessionHolder?
Some session implementations are not stateless in mybatis for performance reasons, they contains local cache.
So while theoretically it is possible to implement session as a thin wrapper around connection and create new session instance (what would get a connection that is bound to current transaction from spring) every time it is needed that would have a performance penalty in scenarios where cache hits are observed.
I get the error com.mongodb.MongoWaitQueueFullException: Too many threads are already waiting for a connection. Max number of threads (maxWaitQueueSize) of 500 has been exceeded. while doing a stress test on my application.
So I am thinking of configuring the maxWaitQueueSize property via configuration.
I am using spring boot to configure mongodb connection. I am using #EnableAutoConfiguration in my Application and I have declared only spring.data.mongodb.uri=mongodb://user:password#ip:27017 in the application.properties file.
How do I configure the maxWaitQueueSize property with spring boot?
How do I decide a good value for the maxWaitQueueSize?
If you're using MongoDB 3.0+, you can set waitQueueMultiple in your mongouri :
spring.data.mongodb.uri=mongodb://user:password#ip:27017/?waitQueueMultiple=10
waitQueueMultiple is a number that the driver multiples the maxPoolSize value to, to provide the maximum number of threads allowed to wait for a connection to become available from the pool.
How do I decide a good value for the maxWaitQueueSize?
It's not directly related to MongoDB but you can read more about Pool Sizing in Hikari github wiki.
In com.mongodb.MongoClientURI, you can find the parameters which can be used in MongoClientOption.
if (key.equals("maxpoolsize")) {
builder.connectionsPerHost(Integer.parseInt(value));
} else if (key.equals("minpoolsize")) {
builder.minConnectionsPerHost(Integer.parseInt(value));
} else if (key.equals("maxidletimems")) {
builder.maxConnectionIdleTime(Integer.parseInt(value));
} else if (key.equals("maxlifetimems")) {
builder.maxConnectionLifeTime(Integer.parseInt(value));
} else if (key.equals("waitqueuemultiple")) {
builder.threadsAllowedToBlockForConnectionMultiplier(Integer.parseInt(value));
} else if (key.equals("waitqueuetimeoutms")) {
builder.maxWaitTime(Integer.parseInt(value));
} else if (key.equals("connecttimeoutms")) {
builder.connectTimeout(Integer.parseInt(value));
} else if (key.equals("sockettimeoutms")) {
builder.socketTimeout(Integer.parseInt(value));
} else if (key.equals("autoconnectretry")) {
builder.autoConnectRetry(_parseBoolean(value));
} else if (key.equals("replicaset")) {
builder.requiredReplicaSetName(value);
} else if (key.equals("ssl")) {
if (_parseBoolean(value)) {
builder.socketFactory(SSLSocketFactory.getDefault());
}
}
I am using spring boot starter webflux. This issue also happens.
I tried to add MongoClientFactoryBean. It doesn't work.
The whole application is located in https://github.com/yigubigu/webfluxbenchmark. I tried to test performance benchmark of webflux and original mvc.
#Bean
public MongoClientFactoryBean mongoClientFactoryBean() {
MongoClientFactoryBean factoryBean = new MongoClientFactoryBean();
factoryBean.setHost("localhost");
factoryBean.setPort(27017);
factoryBean.setSingleton(true);
MongoClientOptions options = MongoClientOptions.builder()
.connectionsPerHost(1000)
.minConnectionsPerHost(500)
.threadsAllowedToBlockForConnectionMultiplier(10)
.build();
factoryBean.setMongoClientOptions(options);
return factoryBean;
}
you can achieve this by injecting an object of MongoOptions to your MongoTemplate.
This maxQueueSize limit is computed here in the Java client source code :
https://github.com/mongodb/mongo-java-driver/blob/3.10.x/driver-core/src/main/com/mongodb/connection/ConnectionPoolSettings.java#L273
It is the product of maxConnectionPoolSize and threadsAllowedToBlockForConnectionMultiplier and hence can be modified through ?maxPoolSize= and ?waitQueueMultiple= in the connection URI.
I am using EntityFramework Core commands to migration database. The command I am using is like the docs suggests: dnx . ef migration apply. The problem is when specifying AttachDbFileName in connection string, the following error appear: Unable to Attach database file as database xxxxxxx. This is the connection string I am using:
Data Source=(LocalDB)\mssqllocaldb;Integrated Security=True;Initial Catalog=EfGetStarted2;AttachDbFileName=D:\EfGetStarted2.mdf
Please help how to attach the db file to another location.
Thanks
EF core seem to have troubles with AttachDbFileName or doesn't handle it at all.
EnsureDeleted changes the database name to master but keeps any AttachDbFileName value, which leads to an error since we cannot attach the master database to another file.
EnsureCreated opens a connection using the provided AttachDbFileName value, which leads to an error since the file of the database we want to create does not yet exist.
EF6 has some logic to handle these use cases, see SqlProviderServices.DbCreateDatabase, so everything worked quite fine.
As a workaround I wrote some hacky code to handle these scenarios:
public static void EnsureDatabase(this DbContext context, bool reset = false)
{
if (context == null)
throw new ArgumentNullException(nameof(context));
if (reset)
{
try
{
context.Database.EnsureDeleted();
}
catch (SqlException ex) when (ex.Number == 1801)
{
// HACK: EF doesn't interpret error 1801 as already existing database
ExecuteStatement(context, BuildDropStatement);
}
catch (SqlException ex) when (ex.Number == 1832)
{
// nothing to do here (see below)
}
}
try
{
context.Database.EnsureCreated();
}
catch (SqlException ex) when (ex.Number == 1832)
{
// HACK: EF doesn't interpret error 1832 as non existing database
ExecuteStatement(context, BuildCreateStatement);
// this takes some time (?)
WaitDatabaseCreated(context);
// re-ensure create for tables and stuff
context.Database.EnsureCreated();
}
}
private static void WaitDatabaseCreated(DbContext context)
{
var timeout = DateTime.UtcNow + TimeSpan.FromMinutes(1);
while (true)
{
try
{
context.Database.OpenConnection();
context.Database.CloseConnection();
}
catch (SqlException)
{
if (DateTime.UtcNow > timeout)
throw;
continue;
}
break;
}
}
private static void ExecuteStatement(DbContext context, Func<SqlConnectionStringBuilder, string> statement)
{
var builder = new SqlConnectionStringBuilder(context.Database.GetDbConnection().ConnectionString);
using (var connection = new SqlConnection($"Data Source={builder.DataSource}"))
{
connection.Open();
using (var command = connection.CreateCommand())
{
command.CommandText = statement(builder);
command.ExecuteNonQuery();
}
}
}
private static string BuildDropStatement(SqlConnectionStringBuilder builder)
{
var database = builder.InitialCatalog;
return $"drop database [{database}]";
}
private static string BuildCreateStatement(SqlConnectionStringBuilder builder)
{
var database = builder.InitialCatalog;
var datafile = builder.AttachDBFilename;
var dataname = Path.GetFileNameWithoutExtension(datafile);
var logfile = Path.ChangeExtension(datafile, ".ldf");
var logname = dataname + "_log";
return $"create database [{database}] on primary (name = '{dataname}', filename = '{datafile}') log on (name = '{logname}', filename = '{logfile}')";
}
It's far from nice, but I'm using it for integration testing anyway. For "real world" scenarios using EF migrations should be the way to go, but maybe the root cause of this issue is the same...
Update
The next version will include support for AttachDBFilename.
There may be a different *.mdf file already attached to a database named EfGetStarted2... Try dropping/detaching that database then try again.
You might also be running into problems if the user LocalDB is running as doesn't have correct permissions to the path.
We require programmatic access to a SQL Server Express service as part of our application. Depending on what the user is trying to do, we may have to attach a database, detach a database, back one up, etc. Sometimes the service might not be started before we attempt these operations. So we need to ensure the service is started. Here is where we are running into problems. Apparently the ServiceController.WaitForStatus(ServiceControllerStatus.Running) returns prematurely for SQL Server Express. What is really puzzling is that the master database seems to be immediately available, but not other databases. Here is a console application to demonstrate what I am talking about:
namespace ServiceTest
{
using System;
using System.Data.SqlClient;
using System.Diagnostics;
using System.ServiceProcess;
using System.Threading;
class Program
{
private static readonly ServiceController controller = new ServiceController("MSSQL$SQLEXPRESS");
private static readonly Stopwatch stopWatch = new Stopwatch();
static void Main(string[] args)
{
stopWatch.Start();
EnsureStop();
Start();
OpenAndClose("master");
EnsureStop();
Start();
OpenAndClose("AdventureWorksLT");
Console.ReadLine();
}
private static void EnsureStop()
{
Console.WriteLine("EnsureStop enter, {0:N0}", stopWatch.ElapsedMilliseconds);
if (controller.Status != ServiceControllerStatus.Stopped)
{
controller.Stop();
controller.WaitForStatus(ServiceControllerStatus.Stopped);
Thread.Sleep(5000); // really, really make sure it stopped ... this has a problem too.
}
Console.WriteLine("EnsureStop exit, {0:N0}", stopWatch.ElapsedMilliseconds);
}
private static void Start()
{
Console.WriteLine("Start enter, {0:N0}", stopWatch.ElapsedMilliseconds);
controller.Start();
controller.WaitForStatus(ServiceControllerStatus.Running);
// Thread.Sleep(5000);
Console.WriteLine("Start exit, {0:N0}", stopWatch.ElapsedMilliseconds);
}
private static void OpenAndClose(string database)
{
Console.WriteLine("OpenAndClose enter, {0:N0}", stopWatch.ElapsedMilliseconds);
var connection = new SqlConnection(string.Format(#"Data Source=.\SQLEXPRESS;initial catalog={0};integrated security=SSPI", database));
connection.Open();
connection.Close();
Console.WriteLine("OpenAndClose exit, {0:N0}", stopWatch.ElapsedMilliseconds);
}
}
}
On my machine, this will consistently fail as written. Notice that the connection to "master" has no problems; only the connection to the other database. (You can reverse the order of the connections to verify this.) If you uncomment the Thread.Sleep in the Start() method, it will work fine.
Obviously I want to avoid an arbitrary Thread.Sleep(). Besides the rank code smell, what arbitary value would I put there? The only thing we can think of is to put some dummy connections to our target database in a while loop, catching the SqlException thrown and trying again until it works. But I'm thinking there must be a more elegant solution out there to know when the service is really ready to be used. Any ideas?
EDIT: Based on feedback provided below, I added a check on the status of the database. However, it is still failing. It looks like even the state is not reliable. Here is the function I am calling before OpenAndClose(string):
private static void WaitForOnline(string database)
{
Console.WriteLine("WaitForOnline start, {0:N0}", stopWatch.ElapsedMilliseconds);
using (var connection = new SqlConnection(string.Format(#"Data Source=.\SQLEXPRESS;initial catal
using (var command = connection.CreateCommand())
{
connection.Open();
try
{
command.CommandText = "SELECT [state] FROM sys.databases WHERE [name] = #DatabaseName";
command.Parameters.AddWithValue("#DatabaseName", database);
byte databaseState = (byte)command.ExecuteScalar();
Console.WriteLine("databaseState = {0}", databaseState);
while (databaseState != OnlineState)
{
Thread.Sleep(500);
databaseState = (byte)command.ExecuteScalar();
Console.WriteLine("databaseState = {0}", databaseState);
}
}
finally
{
connection.Close();
}
}
Console.WriteLine("WaitForOnline exit, {0:N0}", stopWatch.ElapsedMilliseconds);
}
I found another discussion dealing with a similar problem. Apparently the solution is to check the sys.database_files of the database in question. But that, of course, is a chicken-and-egg problem. Any other ideas?
Service start != database start.
Service is started when the SQL Server process is running and responded to the SCM that is 'alive'. After that the server will start putting user databases online. As part of this process, it runs the recovery process on each database, to ensure transactional consistency. Recovery of a database can last anywhere from microseconds to whole days, it depends on the ammount of log to be redone and the speed of the disk(s).
After the SCM returns that the service is running, you should connect to 'master' and check your database status in sys.databases. Only when the status is ONLINE can you proceed to open it.