In SqlSessionTemplate,getSqlSession and closeSqlSession method all are operate SqlSessionHolder,but i think is not must to do this,Spring manager the transaction and provide ConnectionHolder,transaction is linked about connection.
SqlSessionHolder holder = (SqlSessionHolder) getResource(sessionFactory);
if (holder != null && holder.isSynchronizedWithTransaction()) {
if (holder.getExecutorType() != executorType) {
throw new TransientDataAccessResourceException("Cannot change the ExecutorType when there is an existing transaction");
}
holder.requested();
if (logger.isDebugEnabled()) {
logger.debug("Fetched SqlSession [" + holder.getSqlSession() + "] from current transaction");
}
return holder.getSqlSession();
}
if (logger.isDebugEnabled()) {
logger.debug("Creating a new SqlSession");
}
SqlSession session = sessionFactory.openSession(executorType);
why we need SqlSessionHolder?
Some session implementations are not stateless in mybatis for performance reasons, they contains local cache.
So while theoretically it is possible to implement session as a thin wrapper around connection and create new session instance (what would get a connection that is bound to current transaction from spring) every time it is needed that would have a performance penalty in scenarios where cache hits are observed.
Related
We have two .net services (.Net core console applications) which are accessing a postgres db table.
Service 1 inserts some 500 rows every 1 minute. It runs as a background thread.
Service 2 reads data from the same table continuously. There is an MQTT publisher which keeps reading data from this table when any new data is requested. This also happens very frequently i.e atleast 4/5 times a minute.
We are getting "FATAL: sorry, too many clients already " error.
What I am assuming is since write and read is happening simultaneously too frequently, the connection is not getting dispose properly.
Is there a way to avoid read whenever a write is happening.
EDITED
Thanks for the reply.. I know some connection pooling is happening but not sure where.. so my question was how to avoid concurrent access of postgres db..
Was not sure what part of code I can post to make the question clear
I am having using clause on dbcontext and also disposed like the below..
This is retrieval section
using (PlatinumDBContext platinumDBContext = new PlatinumDBContext())
{
try
{
var data = platinumDBContext.TrendPoints.Where(x => ids.Contains(x.TrendPointID) && x.TimeStamp >= DateTime.Now.AddHours(-timeinHours));
result = data.Select(x => new Last24hours
{
Label = x.TrendPointID.ToString(),
Value = (double)x.TrendPointValue,
time = x.TimeStamp.ToString("MM/dd/yyyy HH:mm:ss")
}).ToList();
}
catch (Exception oE)
{
}
finally {
platinumDBContext.Dispose();
}
}
This is the insertion section
using (PlatinumDBContext platinumDBContext = new PlatinumDBContext())
{
try
{
foreach (var point in trendPoints)
{
if (point != null)
{
TrendPoint item = new TrendPoint();
item.CreatedDate = DateTime.Now;
item.ObjectState = ObjectState.Added;
item.TrendPointID = point.TrendID;
item.TrendPointValue = double.IsNaN(point.Value) ? decimal.MinValue : (decimal)point.Value;
item.TimeStamp = new DateTime(point.TimeStamp);
platinumDBContext.Add(item);
}
}
platinumDBContext.SaveChanges();
}
catch (Exception ex)
{
}
finally
{
platinumDBContext.Dispose();
}
}
Regards,
Geervani
I'm trying to follow the steps listed at http://dev.mysql.com/doc/connector-j/en/connector-j-master-slave-replication-connection.html which states
To enable this functionality, use the com.mysql.jdbc.ReplicationDriver
class when configuring your application server's connection pool
From https://github.com/brettwooldridge/HikariCP - it says
HikariCP will attempt to resolve a driver through the DriverManager
based solely on the jdbcUrl
So is this configuration all thats needed?
db.default.url=jdbc:mysql:replication ...
Squeryl has has a number of db Adapters; but my understanding is these are unrelated?
http://squeryl.org/api/index.html#org.squeryl.adapters.MySQLInnoDBAdapter
Sorry for the key word loading - I'm just not too sure where I need to focus
Thanks
Brent
For people hitting this in 2020, Hikari uses
com.mysql.jdbc.jdbc2.optional.MysqlDataSource
as a data source. If I look at the code of the above class. It has a method named connect which returns Connection instance.
protected Connection getConnection(Properties props) throws SQLException {
String jdbcUrlToUse = null;
if (!this.explicitUrl) {
StringBuffer jdbcUrl = new StringBuffer("jdbc:mysql://");
if (this.hostName != null) {
jdbcUrl.append(this.hostName);
}
jdbcUrl.append(":");
jdbcUrl.append(this.port);
jdbcUrl.append("/");
if (this.databaseName != null) {
jdbcUrl.append(this.databaseName);
}
jdbcUrlToUse = jdbcUrl.toString();
} else {
jdbcUrlToUse = this.url;
}
Properties urlProps = mysqlDriver.parseURL(jdbcUrlToUse, (Properties)null);
urlProps.remove("DBNAME");
urlProps.remove("HOST");
urlProps.remove("PORT");
Iterator keys = urlProps.keySet().iterator();
while(keys.hasNext()) {
String key = (String)keys.next();
props.setProperty(key, urlProps.getProperty(key));
}
return mysqlDriver.connect(jdbcUrlToUse, props);
}
where mysqlDriver is an instance of
protected static final NonRegisteringDriver mysqlDriver;
if i check the connect method of NonRegisteringDriver class. It looks like this
public Connection connect(String url, Properties info) throws SQLException {
if (url != null) {
if (StringUtils.startsWithIgnoreCase(url, "jdbc:mysql:loadbalance://")) {
return this.connectLoadBalanced(url, info);
}
if (StringUtils.startsWithIgnoreCase(url, "jdbc:mysql:replication://")) {
return this.connectReplicationConnection(url, info);
}
}
Properties props = null;
if ((props = this.parseURL(url, info)) == null) {
return null;
} else if (!"1".equals(props.getProperty("NUM_HOSTS"))) {
return this.connectFailover(url, info);
} else {
try {
com.mysql.jdbc.Connection newConn = ConnectionImpl.getInstance(this.host(props), this.port(props), props, this.database(props), url);
return newConn;
} catch (SQLException var6) {
throw var6;
} catch (Exception var7) {
SQLException sqlEx = SQLError.createSQLException(Messages.getString("NonRegisteringDriver.17") + var7.toString() + Messages.getString("NonRegisteringDriver.18"), "08001", (ExceptionInterceptor)null);
sqlEx.initCause(var7);
throw sqlEx;
}
}
}
After looking at the code, it looks like it supports. I haven't tried it till now. Will try and let you know from personal experience. From code, it looks directly feasible.
Squeryl offers different MySQL adapters because innodb supports referential keys, while myisam does not. It seems like what your'e doing should be handled at the connection pool level, so I don't think your Squeryl configuration will have an affect.
I've never configured Hikari for replicated MySQL, but if it requires an alternative JDBC driver I'd be surprised if you can provide a JDBC URL and everything just works. I'm guessing that Hikari's default functionality is to pick the plain vanilla MySQL JDBC driver unless you tell it otherwise. Luckily, Hikari has quite a few config options including the ability to set a specific driverClassName.
Replication allows for a different URL:
jdbc:mysql:replication://[server1],[server2],[server2]/[database]
I've never tried it, but I assume this will resolve to the ReplicationDriver.
And I find myself back here - please note, hikari doesn't support the Replication driver.
https://github.com/brettwooldridge/HikariCP/issues/625#issuecomment-251613688
MySQL Replication Driver simply does NOT work together with HikariCP.
And
https://groups.google.com/forum/#!msg/hikari-cp/KtKgzR8COrE/higEHoPkAwAJ
... nobody running anything resembling a mission critical application takes MySQL's driver-level replication support seriously.
I have a EF code first context which represents a queue of jobs which a processing application can retrieve and run. These processing applications can be running on different machines but pointing at the same database.
The context provides a method that returns a QueueItem if there is any work to do, or null, called CollectQueueItem.
To ensure no two applications can pick up the same job, the collection takes place in a transaction with an ISOLATION LEVEL of REPEATABLE READ. This means that if there are two attempts to pick up the same job at the same time, one will be chosen as the deadlock victim and be rolled back. We can handle this by catching the DbUpdateException and return null.
Here is the code for the CollectQueueItem method:
public QueueItem CollectQueueItem()
{
using (var transaction = new TransactionScope(TransactionScopeOption.Required, new TransactionOptions { IsolationLevel = IsolationLevel.RepeatableRead }))
{
try
{
var queueItem = this.QueueItems.FirstOrDefault(qi => !qi.IsLocked);
if (queueItem != null)
{
queueItem.DateCollected = DateTime.UtcNow;
queueItem.IsLocked = true;
this.SaveChanges();
transaction.Complete();
return queueItem;
}
}
catch (DbUpdateException) //we might have been the deadlock victim. No matter.
{ }
return null;
}
}
I ran a test in LinqPad to check that this is working as expected. Here is the test below:
var ids = Enumerable.Range(0, 8).AsParallel().SelectMany(i =>
Enumerable.Range(0, 100).Select(j => {
using (var context = new QueueContext())
{
var queueItem = context.CollectQueueItem();
return queueItem == null ? -1 : queueItem.OperationId;
}
})
);
var sw = Stopwatch.StartNew();
var results = ids.GroupBy(i => i).ToDictionary(g => g.Key, g => g.Count());
sw.Stop();
Console.WriteLine("Elapsed time: {0}", sw.Elapsed);
Console.WriteLine("Deadlocked: {0}", results.Where(r => r.Key == -1).Select(r => r.Value).SingleOrDefault());
Console.WriteLine("Duplicates: {0}", results.Count(r => r.Key > -1 && r.Value > 1));
//IsolationLevel = IsolationLevel.RepeatableRead:
//Elapsed time: 00:00:26.9198440
//Deadlocked: 634
//Duplicates: 0
//IsolationLevel = IsolationLevel.ReadUncommitted:
//Elapsed time: 00:00:00.8457558
//Deadlocked: 0
//Duplicates: 234
I ran the test a few times. Without the REPEATABLE READ isolation level, the same job is retrieved by different theads (seen in the 234 duplicates). With REPEATABLE READ, jobs are only retrieved once but performance suffers and there are 634 deadlocked transactions.
My question is: is there a way to get this behaviour in EF without the risk of deadlocks or conflicts? I know in real life there will be less contention as the processors won't be continually hitting the database, but nonetheless, is there a way to do this safely without having to handle the DbUpdateException? Can I get performance closer to that of the version without the REPEATABLE READ isolation level? Or are Deadlocks not that bad in fact and I can safely ignore the exception and let the processor retry after a few millis and accept that the performance will be OK if the not all the transactions are happening at the same time?
Thanks in advance!
Id recommend a different approach.
a) sp_getapplock
Use an SQL SP that provides an Application lock feature
So you can have unique app behaviour, which might involve read from the DB or what ever else activity you need to control. It also lets you use EF in a normal way.
OR
b) Optimistic concurrency
http://msdn.microsoft.com/en-us/data/jj592904
//Object Property:
public byte[] RowVersion { get; set; }
//Object Configuration:
Property(p => p.RowVersion).IsRowVersion().IsConcurrencyToken();
a logical extension to the APP lock or used just by itself is the rowversion concurrency field on DB. Allow the dirty read. BUT when someone goes to update the record As collected, it fails if someone beat them to it. Out of the box EF optimistic locking.
You can delete "collected" job records later easily.
This might be better approach unless you expect high levels of concurrency.
As suggested by Phil, I used optimistic concurrency to ensure the job could not be processed more than once. I realised that rather than having to add a dedicated rowversion column I could use the IsLocked bit column as the ConcurrencyToken. Semantically, if this value has changed since we retrieved the row, the update should fail since only one processor should ever be able to lock it. I used the fluent API as below to configure this, although I could also have used the ConcurrencyCheck data annotation.
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
modelBuilder.Entity<QueueItem>()
.Property(p => p.IsLocked)
.IsConcurrencyToken();
}
I was then able to simple the CollectQueueItem method, losing the TransactionScope entirely and catching the more DbUpdateConcurrencyException.
public OperationQueueItem CollectQueueItem()
{
try
{
var queueItem = this.QueueItems.FirstOrDefault(qi => !qi.IsLocked);
if (queueItem != null)
{
queueItem.DateCollected = DateTime.UtcNow;
queueItem.IsLocked = true;
this.SaveChanges();
return queueItem;
}
}
catch (DbUpdateConcurrencyException) //someone else grabbed the job.
{ }
return null;
}
I reran the tests, you can see it's a great compromise. No duplicates, nearly 100x faster than with REPEATABLE READ, and no DEADLOCKS so the DBAs won't be on my case. Awesome!
//Optimistic Concurrency:
//Elapsed time: 00:00:00.5065586
//Deadlocked: 624
//Duplicates: 0
We have two different query strategies that we'd ideally like to operate in conjunction on our site without opening redundant connections. One strategy uses the enterprise library to pull Database objects and Execute_____(DbCommand)s on the Database, without directly selecting any sort of connection. Effectively like this:
Database db = DatabaseFactory.CreateDatabase();
DbCommand q = db.GetStoredProcCommand("SomeProc");
using (IDataReader r = db.ExecuteReader(q))
{
List<RecordType> rv = new List<RecordType>();
while (r.Read())
{
rv.Add(RecordType.CreateFromReader(r));
}
return rv;
}
The other, newer strategy, uses a library that asks for an IDbConnection, which it Close()es immediately after execution. So, we do something like this:
DbConnection c = DatabaseFactory.CreateDatabase().CreateConnection();
using (QueryBuilder qb = new QueryBuilder(c))
{
return qb.Find<RecordType>(ConditionCollection);
}
But, the connection returned by CreateConnection() isn't the same one used by the Database.ExecuteReader(), which is apparently left open between queries. So, when we call a data access method using the new strategy after one using the old strategy inside a TransactionScope, it causes unnecessary promotion -- promotion that I'm not sure we have the ability to configure for (we don't have administrative access to the SQL Server).
Before we go down the path of modifying the query-builder-library to work with the Enterprise Library's Database objects ... Is there a way to retrieve, if existent, the open connection last used by one of the Database.Execute_______() methods?
Yes, you can get the connection associated with a transaction. Enterprise Library internally manages a collection of transactions and the associated database connections so if you are in a transaction you can retrieve the connection associated with a database using the static TransactionScopeConnections.GetConnection method:
using (var scope = new TransactionScope())
{
IEnumerable<RecordType> records = GetRecordTypes();
Database db = DatabaseFactory.CreateDatabase();
DbConnection connection = TransactionScopeConnections.GetConnection(db).Connection;
}
public static IEnumerable<RecordType> GetRecordTypes()
{
Database db = DatabaseFactory.CreateDatabase();
DbCommand q = db.GetStoredProcCommand("GetLogEntries");
using (IDataReader r = db.ExecuteReader(q))
{
List<RecordType> rv = new List<RecordType>();
while (r.Read())
{
rv.Add(RecordType.CreateFromReader(r));
}
return rv;
}
}
I am looking for the best way to handle concurrency while using Entity Framework. The simplest and most recommended (also on stack) solution is described here:
http://msdn.microsoft.com/en-us/library/bb399228.aspx
And it looks like:
try
{
// Try to save changes, which may cause a conflict.
int num = context.SaveChanges();
Console.WriteLine("No conflicts. " +
num.ToString() + " updates saved.");
}
catch (OptimisticConcurrencyException)
{
// Resolve the concurrency conflict by refreshing the
// object context before re-saving changes.
context.Refresh(RefreshMode.ClientWins, orders);
// Save changes.
context.SaveChanges();
Console.WriteLine("OptimisticConcurrencyException "
+ "handled and changes saved");
}
But is it enough? What if something changes between Refresh() and the second SaveChanges()? There will be uncaught OptimisticConcurrencyException?
EDIT 2:
I think this would be the final solution:
int savesCounter = 100;
Boolean saveSuccess = false;
while (!saveSuccess && savesCounter > 0)
{
savesCounter--;
try
{
// Try to save changes, which may cause a conflict.
int num = context.SaveChanges();
saveSuccess = true;
Console.WriteLine("Save success. " + num.ToString() + " updates saved.");
}
catch (OptimisticConcurrencyException)
{
// Resolve the concurrency conflict by refreshing the
// object context before re-saving changes.
Console.WriteLine("OptimisticConcurrencyException, refreshing context.");
context.Refresh(RefreshMode.ClientWins, orders);
}
}
I am not sure if Iunderstand how the Refresh() works. Does it refresh whole context? If yes, why does it take additional arguments (entities objects)? Or does it refreshes only objects specified?
For example in this situation what should be passed as Refresh() second argument:
Order dbOrder = dbContext.Orders.Where(x => x.ID == orderID);
dbOrder.Name = "new name";
//here whole the code written above to save changes
should it be dbOrder?
Yes, even the second save may cause an OptimisticConcurrencyException if - as you say - something changes between Refresh() and SaveChanges().
The example given is just a very simple retry logic, if you need to retry more than once or resolve the conflict in a more complex way, you're better off creating a loop that will retry n times than nesting try/catch more than this single level.