I've been using npgsql for a few years now.
I've never had an issue until recently when the load on my PostgreSQL server increased dramatically.
At the moment I have hundreds of instances of postgres.exe "Running" and "Terminating" on my server every second.
When I switch on pooling, it doesn't make a difference to the amount of instances being created / terminated and after a few minutes I get an error:
The connection pool has been exhausted, either raise MaxPoolSize (currently 100) or Timeout (currently 15 seconds)
I'm now questioning if my method of using npgsql is correct.
I used nuget to add npgsql into my project and created the following class which is used every time that I run a query:
Public Class dlNpgSQL
Dim _sqlCommand As NpgsqlCommand
Dim _sqlDataAdapter As NpgsqlDataAdapter
Dim _dataset As DataSet
Public Sub New()
_sqlConnection = New NpgsqlConnection(ConfigurationManager.
ConnectionStrings("connstring").ConnectionString)
End Sub
Public Function GetDataSet(ByVal strQuery As String) As DataSet
_dataset = New DataSet
_sqlDataAdapter = New NpgsqlDataAdapter(strQuery, OpenConnection)
_sqlDataAdapter.Fill(_dataset)
Return _dataset
End Function
Public Function OpenConnection() As NpgsqlConnection
If _sqlConnection.State = ConnectionState.Closed Then
_sqlConnection.Open()
End If
End Function
Public Sub CloseConnection()
If _sqlConnection.State = ConnectionState.Open Then
_sqlConnection.Close()
End If
End Function
End Class
I run queries as following:
Dim ds As DataSet
Dim objDB As New dlNpgSQL("connstring")
tmpSQL = "SELECT * FROM table1"
Try
ds = objDB.GetDataSet(tmpSQL)
Catch ex As Exception
If (objDB IsNot Nothing) Then
objDB.CloseConnection()
End If
Return Nothing
Finally
If (objDB IsNot Nothing) Then
objDB.CloseConnection()
End If
End Try
The connection string I use is:
<connectionStrings>
<add name = "connstring" connectionString="Server=localhost;Port=5432;User Id=postgres;Password=password;Database=test_db;CommandTimeout=600;Pooling=True;"/>
</connectionStrings>
So my questions are:
Am I using npgsql in the correct way?
If so, why are so many instances of postgres.exe being created?
Why do I get the error about the pool being exhausted?
Is there a way to see what connections are in the pool? When I check pg_stat_activity, there are no open connections.
I've been battling with this for days - any assistance will be greatly appreciated!
Related
I'm using PostgreSQL 9.5 and Npgsql 2.2.5.0 to connect from an ASP.NET web service written with VB.NET to the PostgreSQL server.
The webservice gets around 15 requests a second.
The connection string I use is:
<connectionStrings>
<add name="PostgreConnectionStringPCMRead"
connectionString="Server=xxx.xxx.xxx.xxx;Port=5432;User
Id=the_user;Password=the_password;Database=example;
CommandTimeout=600;Timeout=30;ConnectionLifeTime=30;
Pooling=False;"/>
</connectionStrings>
When I take a look at the Windows Resource Monitor, I see hundreds of Terminated connections.
I realise that I am using Pooling=False, however when I set Pooling=True, I get errors "Timeout while getting a connection from pool."
With every query I use the following Class (I've left in only the essential code):
Public Class dlNpgSQL
Dim _sqlCommand As NpgsqlCommand
Dim _sqlDataAdapter As NpgsqlDataAdapter
Dim _dataset As DataSet
Public Sub New()
_sqlConnection = New NpgsqlConnection(ConfigurationManager.
ConnectionStrings("connstring").ConnectionString)
End Sub
Public Function GetDataSet(ByVal strQuery As String) As DataSet
_dataset = New DataSet
_sqlDataAdapter = New NpgsqlDataAdapter(strQuery, OpenConnection)
_sqlDataAdapter.Fill(_dataset)
Return _dataset
End Function
Public Function OpenConnection() As NpgsqlConnection
If _sqlConnection.State = ConnectionState.Closed Then
_sqlConnection.Open()
End If
End Function
Public Sub CloseConnection()
If _sqlConnection.State = ConnectionState.Open Then
_sqlConnection.Close()
End If
End Function
End Class
And then the actual query:
Dim ds As DataSet
Dim objDBRead As New dlNpgSQL("connstring")
tmpSQL = "SELECT * FROM table1"
ds = objDBRead.GetDataSet(tmpSQL)
If (objDBRead IsNot Nothing) Then
objDBRead.CloseConnection()
End If
So obviously, I'm opening and closing the connection with each query.
Is this the correct way to do things?
Should I be using pooling, and if yes, why am I getting the errors and how can I fix the issue?
Thanks!
PREFACE: I am using SQL Server 2008 R2 BackEnd and MS Access 2007 for the FrontEnd
I have a Class Module that returns any ADO Recordset I want from the SQL Server. I can then assign this to any form RecordSource property.
The problem is that when I try to edit the fields it says "This form is read-only" in the status bar. I want the form to be editable.
I have two forms
FormEntities
FormEntitiesEdit
The FormEntitiesEdit form does NOT use the Class Module. Rather all the code is in the form itself.
The purpose of the class module is avoid redundancy and just be able to use the Class Module to get a recordset from SQL Server easily.
FIRST HERE IS MY GLOBAL MODULE
'Default error message. 'eh' stands for error handler
Public eh As String
'Global variables for universal use
Public conn As ADODB.Connection
Public rs As ADODB.Recordset
Public com As ADODB.Command
SECOND IS THE CLASS MODULE (Name is cADO).
THIS CLASS MODULE USES THE conn CONNECTION OBJECT ABOVE
Option Explicit
Private Const CONST_LockType = 3
Private Const CONST_CursorType = 1
Private Const CONST_CursorLocationServer = 3
Private Const CONST_CursorLocationClient = 2
Private m_Recordset As ADODB.Recordset
'For Public Recordset function
Private cSQL$
'**********************************************************************
Public Function cGetRecordset(ByRef sql) As ADODB.Recordset
Set m_Recordset = New ADODB.Recordset
cSQL = sql
cOpenRecordset
Set cGetRecordset = m_Recordset
End Function
'**********************************************************************
Public Property Set Recordset(Value As ADODB.Recordset)
'Assigns private variable a property
If Not Value Is Nothing Then Set m_Recordset = Value
End Property
'**********************************************************************
Public Property Get Recordset() As ADODB.Recordset
'Reads the recordset from the private variable and assigns to new object variable
Set Recordset = m_Recordset
End Property
'********************************** PRIVATE SECTION **********************************
Private Sub cOpenRecordset()
On Error GoTo eh
'Ensures that if a recordset is opened from previously that it closes before opening a new one
If m_Recordset.State adStateClosed Then m_Recordset.Close
Set m_Recordset.ActiveConnection = conn
With m_Recordset
.LockType = CONST_LockType
.CursorType = CONST_CursorType
.CursorLocation = CONST_CursorLocationClient
.Source = cSQL
.Open .Source
End With
If Not m_Recordset.EOF Then m_Recordset.MoveFirst
Exit Sub
eh:
eh = "Error # " & Str(Err.Number) & " was generated by " & _
Err.Source & Chr(13) & Err.Description
MsgBox eh, vbCritical, "Open Recordset System"
End Sub
'**********************************************************************
Private Sub cCloseRecordset()
m_Recordset.Close
Set m_Recordset = Nothing
End Sub
'**********************************************************************
Private Sub Class_Terminate()
If Not (m_Recordset Is Nothing) Then Set m_Recordset = Nothing
End Sub
THIRD IS THE CODE BEHIND MY FormEntities FORM (USES THE THE cADO CLASS MODULE)
Option Explicit
Dim db As cADO
'**********************************************************************
Private Sub Form_Current()
LoadTab
End Sub
'**********************************************************************
Private Sub Form_Load()
Set db = New cADO
FetchRecordSource
End Sub
'**********************************************************************
Private Sub FetchRecordSource()
db.cGetRecordset ("SELECT * FROM dbo.Entities")
Set Forms("fEntities").Recordset = db.Recordset
End Sub
FOURTH AND FINALLY IS THE CODE BEHIND THE FormEntitiesEdit FORM (THIS FORM DOES NOT USE THE CLASS MODULE AND I CAN EDIT IT)
Option Compare Database
Option Explicit
Dim rsEntity As New ADODB.Recordset
'**********************************************************************
Private Sub Form_Load()
FetchRecordSource
End Sub
'**********************************************************************
Private Sub FetchRecordSource()
Set rsEntity.ActiveConnection = conn
'Sets the record source for the main form
With rsEntity
.LockType = adLockOptimistic
.CursorType = adOpenKeyset
.CursorLocation = adUseClient
.Source = "SELECT * FROM dbo.Entities"
.Open .Source
End With
Set Forms("fEntitiesEdit").Recordset = rsEntity
End Sub
'**********************************************************************
Private Sub CloseConn()
rsEntity.Close
End Sub
If Access Jet SQL could do the SQL I would bind this form to a Linked Table instead. However I am using a hierarchical (recursive) query which Jet SQL cannot do so I have to bypass the idea of bound forms to Linked Tables.
I have .Allow Edits and .AllowAdditions on the form set to true.
I also tried changing the .LockType on the ADO Recordset to
adOpenKeyset and
adOpenDynamic
My LockType is adLockOptimistic
As you can see the properties are set correctly to be able to edit the recordset I return.
I know this is true because when I use the FormEntitiesEdit form with the same properties it lets me edit the field. But when I use the Class Module to return (using the same properties) it says it's read-only.
This is important because as you can see it is a lot simpler to use the Class Module, just need it to return an editable recordset.
Anyone have ideas? suggestions?
The problem is here in the class' cOpenRecordset() method:
.CursorLocation = CONST_CursorLocationClient
Here is where you assign constant values ...
Private Const CONST_CursorLocationServer = 3
Private Const CONST_CursorLocationClient = 2
Unfortunately, you swapped those two values. Here are the ADODB.CursorLocationEnum constants ...
adUseClient = 3
adUseServer = 2
So your class is effectively doing this ...
.CursorLocation = adUseServer
But you need a client-side cursor if you want the recordset to be editable. I think your class approach may work if you simply redefine the constant, or you will expose a different problem ...
Private Const CONST_CursorLocationClient = 3
FormEntitiesEdit is editable because you're using the correct constant there ...
.CursorLocation = adUseClient
Is this a bug?
My components are:
.NET 4.6.1
Entity Framework 6.0
Npgsql 3.0.5
PostgreSQL 9.5
First, I created a table and stored procedure in PostgreSQL 9.5
CREATE TABLE hello
(
msg text
)
WITH (
OIDS=FALSE
);
ALTER TABLE hello
OWNER TO postgres;
CREATE OR REPLACE FUNCTION sayhello()
RETURNS SETOF hello AS
$BODY$
select
*
from version()
$BODY$
LANGUAGE sql VOLATILE
COST 100
ROWS 1000;
ALTER FUNCTION sayhello()
OWNER TO postgres;
Second, I went to the .edmx file (Entity Framework 6.0), choose "update from database", selected the new table "hello" and the new stored procedure, "sayhello".
The Model Browser now shows the new table entity and the imported function.
Third, add a new procedure to the WCF file:
public string SayHello()
{
using (var ctx = new chaosEntities())
{
var x = ctx.sayhello();
return "Hello";
}
}
Set the WCF Service as Startup project and Start Debugging.
The WCF Test Client comes up. Executing SayHello() from the WCF Service leads to:
public virtual ObjectResult<hello> sayhello()
{
return ((IObjectContextAdapter)this).ObjectContext.ExecuteFunction<hello>("sayhello");
}
When this is executed, I get:
An exception of type 'System.Data.Entity.Core.EntityCommandCompilationException'
occurred in EntityFramework.dll but was not handled in user code
Additional information: An error occurred while preparing the command
definition. See the inner exception for details.
Inner Exception is: {"Value does not fall within the expected range."}
As I have several hundred stored procedures, any help on how to fix this is most appreciated.
TIA
Note: I suspect the problem is with NpgsqlServices.TranslateCommandTree, but I'm only guessing.
I could never get it to work the way I hoped (via EntityFramework), so I ended up doing this. I'd sure be open to a better solution!
The below code calls Npgsql directly to avoid the whole EntityFramework thing.
public string SayHello()
{
using (var ctx = new chaosEntities())
{
var b = ctx.Database.Connection.ConnectionString;
using (var conn = new Npgsql.NpgsqlConnection(connectionString: b))
{
conn.Open();
using (var tran = conn.BeginTransaction())
using (var command = conn.CreateCommand())
{
command.CommandText = "sayhello";
command.CommandType = CommandType.StoredProcedure;
var g = (string)command.ExecuteScalar();
return g;
}
}
}
}
I am updating an application. I have replaced most of the queries in the app with calls to ejbs but the code below calls a huge procedure and it would be almost impossible to re-write.
I would like to return an ejb to the result set R but I have not been able to figure this out....
java.sql.ResultSet R = Cmd.executeQuery("SELECT * FROM TableData");
String[] Names = {"id","Project","Resource","Week","Hours"};
out.print(getTableXML(R,Names));
R.close();
My ejb:
public List<Gridmaster> getDisplayGridList() {
return em.createQuery("FROM Gridmaster order by gridid", Gridmaster.class).getResultList();
Is this possible or do I need to create an old style db connection?
Thanks for any help.
I am not sure I am doing any of this correctly. Is it because I am opening two connections? I am closing them regardless of errors. I did try putting in some inner transaction scopes and setting the second one to RequiresNew. The two methods are independent of each other, however, if one fails I need them both to rollback. I may have to modify how I am creating and closing the connections, I feel. Any thoughts? Here is some example code of what I am doing:
Public Sub TransMethod()
Using sTran As New Transactions.TransactionScope
factory1.UpdateMethod(someObject1)
facotry2.insert(someobject2)
sTran.Complete()
End Using
End Sub
Public Class factory1
Public Shared Sub UpdateMethod(obj)
dim someSQLParams....
DataAcces.ExecuteNonQuery(command,someSQLParams)
End Sub
End Class
Public Class factory2
Public Shared Sub Insert(obj)
dim someSQLParams....
DataAcces.ExecuteNonQuery(command,someSQLParams)
End Sub
End Class
Public Function ExecuteNonQuery(ByVal spname As String, _
ByVal ParamArray parameterValues() As Object) As Object
Dim connection As SqlConnection = Nothing
Dim command As SqlCommand = Nothing
Dim res As Object = Nothing
Try
connection = New SqlConnection(_connectionString)
command = New SqlCommand(spname, connection)
command.CommandType = CommandType.StoredProcedure
command.Parameters.AddRange(parameterValues)
connection.Open()
command.ExecuteNonQuery()
res = command.Parameters(command.Parameters.Count - 1).Value
Catch ex As Exception
CreateDataEntry(ex, WriteType.ToFile, spname)
If Not (transaction Is Nothing) Then
transaction.Rollback()
End If
Finally
If Not (connection Is Nothing) AndAlso _
(connection.State = ConnectionState.Open) Then _
connection.Close()
If Not (command Is Nothing) Then command.Dispose()
End Try
Return res
End Function
Whenever you open more than 1 connection using TransactionScope, it changes from a normal transaction against sql server to a distributed transaction, which requires setting up the MSDTC.
That will happen even if the connections have the same connection string, which is one of the "by design" issues. I haven't followed up if it remains the same on .net 3.5+