I'm working on an ASP.NET MVC app. This app is relying on data stored in a SQL Azure database. When I logged into the SQL Azure management interface, I notice that I had 17 active connections to my database. I vaguely remember the concept of database connection pooling from long ago. For some reason, I thought to use connection pooling, you needed to add a setting to your connection string in your web.config file. For the life of me though, I can't remember or find documentation on a setting.
For reference sake, I'm using System.Data.SqlClient as the provide in my connection string settings. Can someone please tell me how to use connection pooling in SQL Azure? Considering I'm the only one hitting the database, 17 active connections seemed high. I figured if connection pooling was turned on, only 1 active connection should appear.
Thank you.
Connection pooling is a default behavior that cannot be configured by the client for Sql Azure. Your app should be getting the benefits of connection pooling by default. Ensure that your connections strings are unique as a new pool will be created for connections with different strings. This article in MSDN specifies:
When a new connection is opened, if the connection string is not an
exact match to an existing pool, a new pool is created. Connections
are pooled per process, per application domain, per connection string
and when integrated security is used, per Windows identity. Connection
strings must also be an exact match; keywords supplied in a different
order for the same connection will be pooled separately.
Now with regards to a setting that you don't remember. You may have been talking about MARS (Multiple Active Result Sets). This feature is now available on Sql Azure.
I would suggest you to read the following article which explains how to correctly design a connection to SQL Azure and it also have a C# base sample code to show you how:
http://blogs.msdn.com/b/sqlazure/archive/2010/05/11/10011247.aspx
Related
We have mongodb as datastorage, and there is a MongoClient which we are using for connection pooling.
Question is whether to explicitly use the MongoClient.close to shutdown the connection pool or not.
Here's what I have explored on this so far.
The documentation for the close API says
Closes all resources associated with this instance, in particular any open network connections. Once called, this instance and any databases obtained from it can no longer be used.
But when I referred other questions on this topic, it says you can perform your operations and don't need to explicitly manage operations like MongoClient.close, as this object manages connection pooling automatically.
Java MongoDB connection pool
Both of them are contradicting. If I were to follow the second, what will be the downsides of it?
Will the connections in the pool be closed when the mongoclient object is de-referenced from jvm?
or will the connections stay open for a particular period of time and then expire?
I would like to know what are the actual downsides of this approach. Any pointers on this is highly appreciated.
IMO, using close on server shut down seems to be the clean way to do it.
But I would like to get an expert opinion on this.
Update: There is no need to explicitly close the connection pool via API. Mongo driver takes care of it.
I'm looking for more detailed guidance / other people's experience of using Npgsql in production with Pgbouncer.
Basically we have the following setup using GKE and Google Cloud SQL:
Right now - I've got npgsql configured as if pgbouncer wasn't in place, using a local connection pool. I've added pgbouncer as a deployment in my GKE cluster as Google SQL has very low max connection limits - and to be able to scale my application horizontally inside of Kubernetes I need to protect against overwhelming it.
My problem is one of reliability when one of the pgbouncer pods dies (due to a node failure or as I'm scaling up/down).
When that happens (1) all of the existing open connections from the client side connection pools in the application pods don't immediately close (2) - and basically result in exceptions to my application as it tries to execute commands. Not ideal!
As I see it (and looking at the advice at https://www.npgsql.org/doc/compatibility.html) I have three options.
Live with it, and handle retries of SQL commands within my application. Possible, but seems like a lot of effort and creates lots of possible bugs if I get it wrong.
Turn on keep alives and let npgsql itself 'fail out' relatively quickly the bad connections when those fail. I'm not even sure if this will work or if it will cause further problems.
Turn off client side connection pooling entirely. This seems to be the official advice, but I am loathe to do this for performance reasons, it seems very wasteful for Npgsql to have to open a connnection to pgbouncer for each session - and runs counter to all of my experience with other RDBMS like SQL Server.
Am I on the right track with one of those options? Or am I missing something?
You are generally on the right track and your analysis seems accurate. Some comments:
Option 2 (turning out keepalives) will help remove idle connections in Npgsql's pool which have been broken. As you've written your application will still some failures (as some bad idle connections may not be removed in time). There is no particular reason to think this would cause further problems - this should be pretty safe to turn on.
Option 3 is indeed problematic for perf, as a TCP connection to pgbouncer would have to be established every single time a database connection is needed. It will also not provide a 100% fail-proof mechanism, since pgbouncer may still drop out while a connection is in use.
At the end of the day, you're asking about resiliency in the face of arbitrary network/server failure, which isn't an easy thing to achieve. The only 100% reliable way to deal with this is in your application, via a dedicated layer which would retry operations when a transient exception occurs. You may want to look at Polly, and note that Npgsql helps our a bit by exposing an IsTransient exception which can be used as a trigger to retry (Entity Framework Core also includes a similar "retry strategy"). If you do go down this path, note that transactions are particularly difficult to handle correctly.
We have different database as per client but all SP's and tables schema are same for all.
How to connect azure mobile service base on client?
Option:
publish service as per client, so number of client is equal to services.
put all connection string in config file. Read header value and pick connection accordingly.
any other option, do you know.
1st option is not feasible for us. because need publish code on all site for single change.
Please suggest me.
You cannot really use Azure Mobile Services for this. Azure Mobile Services is pretty much designed around a single database per service. I'd suggest switching over to Azure App Service. If you just need database access, you can set up a REST endpoint that provides the necessary access but looking up your connection strings on a per-authenticated user. You might want to use a schema per client instead to reduce the number of connection strings you have.
Short version: Look at the design of your service to reduce the number of SQL connection strings you are using. An ideal number is 1.
As far as I know database connectivity technologies like the entity framework open and close connections automatically to enhance scalability. (Managing Connections and Transactions)
For example a form using asp.net mvc and the entity framework will connect to retrieve a record and them will immediately disconnect and remains disconnected until I modify the data in the controls and save it.
I wonder if the same behavior applies for an access 2013 form linked via odbc to SQL Server. Once a record is retrieved, is the connection closed until my next operation or the connection remains open until I close the form? Is the behavior configurable?
The fact or existence of a connection does not change nor increase scalability for typical applications. So if you have 10, or 1000 connections, and those connections are NOT doing anything, then SQL server not doing any work, and hence no increase in scalability will occur in these typical cases.
And OFTEN there is additional chatter over the network to open the connection pull the data, close the connection.
Then when you write the data back you AGAIN have 3 steps. So you again open the connection, open the table, write the data, and then close the table!
In fact keeping the connection open means you don’t waste network bandwidth opening and closing the connection!
The MAIN reason for disconnected datasets is that such connections work far more reliable in the case when you have a poor or less than ideal connection (such as over the internet or via Wi-Fi at a coffee shop). I these cases, if the open connection command fails, then the connection does NOT occur, and you don’t pull any data. And if a bit of time delay or re-try occurs as the connection is re-attempted, then no big deal. So you grab that data and close the connection.
However, this opening, and then closing often as noted causes additional overhead. However, given how the internet works (as opposed to a typical office network), then this disconnected approach is much the norm for pulling data over the internet, or when using something like Wi-Fi. So the approach is one of expecting that a minor disconnect will and can occur.
The second “common” reason for this 3 step process is other development platforms “promote” the use of disconnected data because the forms are NOT bound to the actual data tables (or bound to a query). The downside of this disconnected approach is you thus in general have to write code to pull the data down to the client, and THEN render the data from the recordset object to the form. The result is a TON of additional work to edit data in a form. So expect the typical asp .net application to cost 5 or even 10 times as much as writing that application in Access.
In the case of Access bound form model, it eliminates the developer having to code the data pull and coding and eliminates the need of the developer to pull that data into some object, and then close that connection. Once Access establishes a connection to the SQL server, then that connection remains open until you shut down the Access application.
The keep the connection open and active has the advantage of rapid application development due to the bound forms model. So you DO NOT need to write code to pull data from the server and THEN transfer that data from some type of object into the form.
So the downside to the Access approach has little (if anything) to do with scalability. The downside is a simple break in the connection is NOT at all well handled by Access.
So if you build a form in Access that is bound to a SQL server table of say 1 million records, and you launch that form with a where clause of InvoiceNumer = 12356, then Access is smart, and ONLY pulls down the ONE record from SQL server. So in terms of scalability and performance, the use of disconnected system as opposed to the bound connected model in Access will not result in a performance difference.
However, because Access keeps that connection open, then any breakage in that connection will result in an ODBC error. Worse is Access is NOT able to recover from such errors when using bound forms – your only recourse is to restart Access.
It is certainly possible to build un-bound forms in Access, but this is NOT how Access was designed. In fact if one is going to adopt a disconnected data model in Access then Access is the wrong tool due to no wizards or “developer aids” for such an approach. So say .net has wizards built around the disconnected system, and Access has tools built around the connected system. And a LOT of functionally that makes Access such a great rapid application development tool is lost if you build un-bound forms in Access. So bound Access forms have MANY additional events that you not find in say .net forms.
So it not scalability or better performance that is lost by the bound forms approach in Access, but simple ease of development is the main feature and gain that the Access development approach results in.
A developer will STILL need and should LIMIT the number of records pulled into a form (by use of the forms “where” clause.
(so .net forms don’t have such indulgences as a where clause).
So the major shortcoming is that Access does not recover from ODBC disconnections since it was designed to keep such connections open.
Our production people are claiming that they are pretty sure that the Apache DBCP connection pooling system is validating a connection on each use, that is, every time before it issues a query via that connection. But the DBCP config at http://commons.apache.org/dbcp/configuration.html does not seem to provide any such option that could be obtained by default. It seems that the only two options are on getting a connection or on returning it.
The team claims that they determined this using a tool called DynaTrace.
Could someone throw some light on this please?
Thanks.
I've seen something similar when using testOnBorrow (which will issue a validation query every time a connection is requested from the pool) and requesting a new connection for every statement. Sharing the exported dynaTrace session (if you have access to it) could help with diagnosing this.