I am developing a WebAPI on .NETCore accessing data to a POSTGRESQL DB.
I have troubles with the non-MARS support of PostgreSQL. NPGSQL is unable to support multiple connections from the same instance (as described in EntityFramework DbContext lifecycle + Postgres: "An operation is already in progress."). For Asynchronous management, this is blocking.
Unfortunately, I cannot find any solution to this.
At the moment, I inject my DB context with:
services.AddEntityFrameworkNpgsql().AddDbContextPool<DBApiContext>(opt => opt.UseNpgsql('connectionString');
I use EntityFramework.
Just for people who got stuck in this - In my case, it was a simple code change in the end of the method to fix this:
reader.close();
reader is an object of NpgsqlDataReader.
For who may be interested in: my problem was all about service scope and dependency injection.
The requestor was not a transient service, so that for every requests, even parallel, it was trying to access the DB.
Postgresql doesn't support MARS, then the second requests were rejected.
You need to have transient service requesting the access, for every invokation to use a different DB handler.
Related
I´m getting PK Violation Exception when using EF Core 2.1 DbContext in an Azure QueueTrigger function. Guess is due to the nature of DbContext not being thread-safe, and the Azure Function running different instances in parallel. I have read quite a few, but I can´t find a good approach to solve this.
Here is my scenario (producer-consumer pattern):
I have a Scheduled Azure Function that is calling an API to get Projects from different external systems. To get all the required info for a project, I need to run different Queries to other external services, so I´m decoupling this to another Azure function, so the Scheduled function just queues a message per Project, as “Sync Project ID 101”.
Another QueueTrigger Function fires every time a message is queued, so, it means different instances running in parallel. This function must gather all the data of a specific Project, and that means more calls to other external services / APIs, to (some kind of) aggregate all the info about a Project. IMHO it´s good to do it that way, as I can process multiple Projects in parallel, and I can scale the Function if I need it.
Once I have all this Project info, I want to persist it in a SQL DB using EF Core (and here comes the issue)
Project data includes Users in the Project, and each user have a specific GUID as PK (coming from the external system). That means I can have repeated Users IDs in different Function instances, and here is the problem, as when I try to persist User info in a SQL Table, I can get PK Duplication exception, as multiple Function instances can try to Insert the same User at the same time (when the instance A check if user exists, it gets False, but another instance B is actually adding this User, so when instance A tries the Insert, it fails).
Guess I can lock DbContext somehow, but not sure if is good, as I also have a website doing Queries to the SQL DB (read-only queries for now, but could be updates in future too).
Another idea could be to send the entire Project info to another Queue / Blob file, and have another function in Singleton mode that Insert the data into SQL.
I´ve created this project simplifying my scenario, but enough to reproduce the issue and understand the problem.
https://github.com/luismanez/queuetrigger-efcore-multithreading
Any other ideas or recommended approaches? (open to change the architecture if find something better)
Many thanks!
A "more easy" way could be to do some kind of upsert in the database. There is a sample of how to do that with EF Core: https://www.flexlabs.org/2018/02/adding-upsert-support-for-entity-framework-core
I am developing an application in which the database is selected by the end user at runtime. The database can either be on a MS SQL server or an IBM DB2 server. I am currently using IBM DB2 10 Express-c on a windows server for testing. I am developing using Visual Studio 2013 C# and Entity Framework 6. I have installed the EntityFramework.IBM.DB2 Nuget package for the DB2 support. I am using reverse-engineer code-first against an existing SQL server database to generate my base code. The application works fine against a SQL Server database.
I am using System.Data.Common.DbProviderFactories.GetFactory to generate the provider.
System.Data.EntityClient.EntityConnectionStringBuilder connectString = new System.Data.EntityClient.EntityConnectionStringBuilder(a_Connection);
System.Data.Common.DbConnection conn = System.Data.Common.DbProviderFactories.GetFactory(connectString.Provider).CreateConnection();
conn.ConnectionString = connectString.ProviderConnectionString;
LB500Database = new LB402_TestContext(conn, true);
a_Connection is provider=IBM.Data.DB2;provider connection string="Database=LISTBILL;User ID=xxxx;Password=yyyy;Server=db210:50000"
and is being parsed correctly by the EntityConnectionStringBuilder.
I then try to access a table in the database with
LBData500.LB_System oneSystem;
System.Linq.IQueryable<LB_System> allSystem = LB500Database.LB_System.Where(g => g.DatabaseVersion == databaseVersion && g.CompanyID == companyID);
I get an invalid operation exception "Sequence contains no matching element" which means that no elements are returned. If I remove the Where so that all rows are returned (there is one in the table) and try to enumerate the result set using the VS debugger I see the message:
"The context cannot be used while the model is being created. This exception may be thrown if the context is used inside the OnModelCreating method or if the same context instance is accessed by multiple threads concurrently. Note that instance members of DbContext and related classes are not guaranteed to be thread safe."
I am not using multi-threading. I am not inside the OnModelCreating.
Just changing the connect string to point to SQL server works fine, so I think my basic approach is sound. If I were getting some kind of error back from the server I would have something to go on. I can run the query from inside Visual Studio, so I have connectivity.
Any pointers would be appreciated.
UPDATE:
I turns out the EF objects were generated using EF5 and the EF6 runtime was being used. I regenerated the EF objects using EF6 reverse engineer code first. I can now connect to the database and get an error message:
"ERROR [42704] [IBM][DB2/NT64] SQL0204N \"DBO.LB_SYSTEM\" is an undefined name."
The schema in the DB2 database is the same as my userid (in this case, not always). I added the CurrentSchema=xxxx to the provide connection string, but EF is still passing dbo as the schema name.
Now I need a way to change the schema name at run time. I saw a link to codeplex EFModelAdapter (http://efmodeladapter.codeplex.com). So I may give that a try.
Update2 After looking through EFModelAdapter, I decided to take a different route. Since I only need database access and not schema management, I decided to go with Dapper (https://github.com/StackExchange/dapper-dot-net). This works great for what I need and allows me to change the schema name when accessing DB2 databases.
As per my Update 2, Entity Framework was a little overkill for what I needed. I switched to dapper https://github.com/StackExchange/dapper-dot-net and I am working fine against multiple DBMSs.
I have created several custom activities that update tables in my DB (in this case SQL Server Compact), using Entity Framework 4 with POCOs.
If I put more than one of these inside a WF4 TransactionScope activity, I'm running into problems: EF disposes the DB connection after the first activity has finished, and when the next DB activity tries to do a DB update a new connection is built up. At this moment an exception is thrown.
System.Activities.WorkflowApplicationAbortedException : The workflow has been aborted.
----> System.Data.EntityException : The underlying provider failed on Open.
----> System.InvalidOperationException : The connection object can not be enlisted in transaction scope.
Do I have to keep the EF connection open during the whole transaction scope? How can I do that? Create an explicit custom activity for that, or is there a standard way?
My current workaround goes like this: I created a new code activity that creates our ObjectContext and explicitely calls dbContext.Connection.Open(). It returns the ObjectContext, which is then saved in a workflow variable. That one is passed to all the DB related activities as an InArgument<>. Inside my DB activities, I use this ObjectContext if it is passed in, otherwise I create a new one.
This does work, but I'm not satisfied with this solution: It needs the new InArgument for every DB related activity. In the workflow designer, I have to insert that special OpenDatabaseConnection activity inside the transaction scope, and then make sure that the correct variable is passed into all DB activities. This seems to be very inelegant and error prone, especially if other team members have to use these DB activities.
What would be a better way to handle this?
The problem is that when you open a second connection in the same transaction scope, an attempt is made to promote the transaction to a distributed transaction (even though there's nothing distributed about it since you connect to the same database). SQL Server CE doesn't support this scenario.
What I would do is create a custom 'container' activity that opens (and closes) the connection and makes it available to child activities. This is still not optimal but at least you no longer need to pass InArgument's around. You get the following activity tree:
TransactionScope
InitializeConnection
Sequence
CustomDataActivity1
CustomDataActivity2
CustomDataActivity3
InitializeConnection is a NativeActivity that uses NativeActivityContext.Properties to expose the connection (or the ObjectContext) to child activities.
Make sure you implement proper error handling to ensure you close the connection at all times.
NOTE: Distributed transactions are supported by the full SQL Server only through a Windows service called MSDTC (Microsoft Distributed Transaction Coordinator). You can find this one in your 'Local Services'. Since SQL Server CE is a database that should be able to operate completely standalone, it makes sense that it has no dependency on MSDTC. Therefore it has no support for distributed transactions.
I'm using EF code-first to manage my DB connection, with an explicit connection string declared in web.config. I would like to schedule some DB cleaning process (like deleting test transactions every day), so I'm spawning a thread from Role.OnStart() with proper concurrency management among the instances.
But I'm getting DB database exceptions, like the DB not matching my model, whereas I'm sure it does (the same code used from "inside" the app works well). So my guess is that web.config is not used from the thread, so EF probably uses the default connection string.
What would be the best way to use my connection string from there ?
Thanks
The OnStart method doesn't run in the same process as your web application meaning it doesn't make use of the web.config. I suggest you store the connection string in the service configuration and read it from here when initializing your context.
An other advantage is that you change the setting without re-deploying the application.
This problem is not readily reproducible in a simple example here but was wondering if anyone has any experience and tips, here is the issue:
using Entity Framework
have many points in application where (1) data is written to some entity table e.g. Customer, (2) data is written to history table
both of these actions use Entity Framework, HOWEVER, they use different contexts
these actions need to be both in one transaction: i.e. if one fails to write, the other should not write, etc.
I can wrap them with a TransactionScope,
like this:
using (TransactionScope txScope = new TransactionScope()) {
...
}
but this gives me:
Microsoft Distributed Transaction Coordinator (MSDTC) is disabled for
network transactions.
Our database admin has told me that MSDTC is disabled by choice and can not be installed.
Hence I am making changes trying to create my own EntityConnection with a MetadataWorkspace with the idea that each context will use the same EntityConnection. However, this is proving near impossible trying to get it to work, e.g. currently I continue to get the above error even though theoretically both contexts are using EntityConnection. It's difficult to understand where/why Entity Framework is requiring the MSDTC for example.
Has anyone gone down this road before, have experience or code examples to share?
Well, the problem is quite easy.
If you are using sql server 2008 you should not have that problem because you have promotable transaction, and as .NET knows that you are using the same persistence store (the database) it wont promote it to DTC and commit it as local. look into promotable transaction with sql server 2008.
As far as I know Oracle is working in its driver to support promotable transactions, but I do not know the state, MS oracle driver does not support it.
http://www.oracle.com/technology/tech/windows/odpnet/col/odp.net_11.1.0.7.20_twp.pdf
If you are using a driver that do not support promotable transactions it is impossible for .NET to use local transaction doing two connections. You should change your architecture or convince the database admin for installing MSDTC.
I had a similar problem with SQL 2008, Entity Framework.
I had two frameworks defined (EF1, and EF2) but using identical connection strings to a sql 2008 database.
I got the MSDTC error above, when using nested "usings" across both.
eg the code was like this:
using (TransactionScope dbContext = new TransactionScope())
{
using (EF1 context = new EF1())
{
// do some EF1 db call
using (EF2 context2 = new EF2())
{
// do some EF2 db call
}
}
dbContext.Complete();
}
It wasnt as simple as this, because it was split across several methods, but this was the basic structure of "usings".
The fix was to only open one using at a time. No MTDSC error, No need to open distributed transactions on db.
using (TransactionScope dbContext = new TransactionScope())
{
using (EF1 context = new EF1())
{
// do some EF1 db call
}
using (EF2 context2 = new EF2())
{
// do some EF2 db call
}
dbContext.Complete();
}
I think that what you need to do is to force your contexts to share single database connection. You will be able then to perform these two operations against two different contexts in single transaction. You can achieve this by passing one EntityConnection object to both of your context's constructors. Of course this approach will require you to pass this object to methods which update DB.
I have recently blogged about creating database context scope which will make using multiple EF contexts and transactions easier.