I have the following Scenario:
I have 2 web api functions, which delete / insert data into a SQL Server database. The data Access is handled via .net entityframework v6. The insert / delete methods were only called from a local running c# program. I am using HttpClient class to call the web api methods. The web methods works as follows, when I call insert all existing records will be deleted and the new ones will be inserted, so there is no real update process.
Here are my 2 functions:
[HttpDelete()]
public async Task<int> DeleteStartlist(int eventid, int run, int heat, string category)
{
_data.dbsStartlistEntries.RemoveRange(_data.dbsStartlistEntries.Where(s => s.Event.Id == eventid && s.RoundOrder == run && s.HeatOrder == heat && s.Category == category));
return await _data.SaveChangesAsync();
}
[HttpPost()]
public async Task<int> UpdateStartlists(int eventid, List<StartlistEntry> en)
{
try
{
if (en.Count == 0)
return 0;
var xdel = await DeleteStartlist(eventid, en[0].RoundOrder, en[0].HeatOrder, en[0].Category);
var ev = await _data.dbsEvents.FindAsync(eventid);
if (ev != null)
{
en.ForEach(e => e.Event = ev);
_data.dbsStartlistEntries.AddRange(en);
}
return await _data.SaveChangesAsync();
}
catch (System.Exception ex)
{
return 1;
}
}
But now I have the following Problem. For example when I call the Update Method 10 times in a row without waiting between the function calls I receive following exception:
Store update, insert, or delete statement affected an unexpected number of rows (0). Entities may have been modified or deleted since entities were loaded. Refresh ObjectStateManager entries.
For me this sounds like a concurrency Problem, but I do not really know how to solve it.
So here is my question, is there a way to let the api calls wait for each other server side, or are they always running concurrent or is there a way to lock the database?
Related
I am building an API where I get a specific object sent as a JSON and then it gets converted into another object of another type, so we have sentObject and convertedObject. Now I can do this:
using (var dbContext = _dbContextFactory.CreateDbContext())
using (var dbContext2 = _dbContextFactory2.CreateDbContext())
{
await dbContext.AddAsync(sentObject);
await dbContext.SaveChangesAsync();
await dbContext2.AddAsync(convertedObject);
await dbContext2.SaveChangesAsync();
}
Now I had a problem where the first SaveChanges call went ok but the second threw an error with a datefield that was not properly set. The first SaveChanges call happened so the data is inserted in the database while the second SaveChanges failed, which cannot happen in my use-case.
What I want to do is if the second SaveChanges call goes wrong then I basically want to rollback the changes that have been made by the first SaveChanges.
My first thought was to delete cascade but the sentObject has a complex structure and I don't want to run into circular problems with delete cascade.
Is there any tips on how I could somehow rollback my changes if one of the SaveChanges calls fails?
You can call context.Database.BeginTransaction as follows:
using (var dbContextTransaction = context.Database.BeginTransaction())
{
context.Database.ExecuteSqlCommand(
#"UPDATE Blogs SET Rating = 5" +
" WHERE Name LIKE '%Entity Framework%'"
);
var query = context.Posts.Where(p => p.Blog.Rating >= 5);
foreach (var post in query)
{
post.Title += "[Cool Blog]";
}
context.SaveChanges();
dbContextTransaction.Commit();
}
(taken from the docs)
You can therefore begin a transaction for dbContext in your case and if the second command failed, call dbContextTransaction.Rollback();
Alternatively, you can implement the cleanup logic yourself, but it would be messy to maintain that as your code here evolves in the future.
Here is an example code that is working for me, no need for calling the rollback function. Calling the rollback function can fail. If you do it inside the catch block for example then you have a silent exception that gets thrown and you will never know about it. The rollback happens automatically when the transaction object in the using statement gets disposed. You can see this if you go to SSMS and look for the open transactions while debugging. See this for reference: https://github.com/dotnet/EntityFramework.Docs/issues/327
Using Transactions or SaveChanges(false) and AcceptAllChanges()?
using (var transactionApplication = dbContext.Database.BeginTransaction())
{
try
{
await dbContext.AddAsync(toInsertApplication);
await dbContext.SaveChangesAsync();
using (var transactionPROWIN = dbContextPROWIN.Database.BeginTransaction())
{
try
{
await dbContext2.AddAsync(convertedApplication);
await dbContext2.SaveChangesAsync();
transaction2.Commit();
insertOperationResult = ("Insert successfull", false);
}
catch (Exception e)
{
Logger.LogError(e.ToString());
insertOperationResult = ("Insert converted object failed", true);
return;
}
}
transactionApplication.Commit();
}
catch (DbUpdateException dbUpdateEx)
{
Logger.LogError(dbUpdateEx.ToString());
if (dbUpdateEx.InnerException.ToString().ToLower().Contains("overflow"))
{
insertOperationResult = ("DateTime overflow", true);
return;
}
//transactionApplication.Rollback();
insertOperationResult = ("Duplicated UUID", true);
}
catch (Exception e)
{
Logger.LogError(e.ToString());
transactionApplication.Rollback();
insertOperationResult = ("Insert Application: Some other error happened", true);
}
}
i have scenario like:
i have to check the table if entry is available in DB then if available i need to call the same external api n times using webclient, collect all the response and save them in DB. if entry is not available in DB call the old flow.
here is my implementation. need suggestions to improve it. without for-each
public Mono<List<ResponseObject>> getdata(String id, Req obj) {
return isEntryInDB(id) //checking the entry in DB
.flatMap(
x -> {
final List<Mono<ResponseObject>> responseList = new ArrayList<>();
IntStream.range(0, obj.getQuantity()) // quantity decides how many times api call t happen
.forEach(
i -> {
Mono<ResponseObject> responseMono =
webClientCall(
id,
req.getType())
.map(
res ->
MapperForMappingDataToDesriedFormat(res));
responseList.add(responseMono);
});
return saveToDb(responseList);
})
.switchIfEmpty(oldFlow(id, req)); //if DB entry is not there take this existing flow.
need some suggestions to improve it without using foreach.
I would avoid using IntStream and rather use native operator to reactor called Flux in this case.
You can replace, InsStream.range with Flux.range. Something like this:
return isEntryPresent("123")
.flatMapMany(s -> Flux.range(0, obj.getQuantity())
.flatMap(this::callApi))
.collectList()
.flatMap(this::saveToDb)
.switchIfEmpty(Mono.defer(() ->oldFlow(id, req)));
private Mono<Object> saveToDb(List<String> stringList){
return Mono.just("done");
}
private Mono<String> callApi(int id) {
return Mono.just("iterating" + id);
}
private Mono<String> isEntryPresent(String id) {
return Mono.just("string");
}
I have 2 system which can communicate through API each other.
Here is my code
System A:
using (var transaction = new TransactionScope())
{
var myBook = _bookRepository.Table.FirstOrDefault(x => x.Id == request.bookID);
myBook.AssigneeId = null;
_bookRepository.Update(ticket);
var result = await _anotherBApi.ApproveBookAsync(request.bookID);
if (result.ShStatus != ResponseStatus.Success)
{
result.ErrorType = ErrorType.Error;
return result;
}
transaction.Complete();
}
Function ApproveBookAsync(request.bookID) will call to B system's API. After handling, B system call back A system's API to update Book's information (the same the one above).
Above my code. I cannot transaction.Complete(); because when B system call A system's API it will create new transaction.
Expect: I want to handle step by step as:
Update new information for a Book instance (sample ID = 1)
Call to B system's API (after B system also call A system's A to update Book ID = 1)
When call B system fail, I want to rollback all changes before. If success, commit.
When using async/await in TransactionScope block, you need to opt that you need your transaction to flow accross thread continuations like this:
using (var transaction = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
{
// Your code that contains some calls to async method.
transaction.Complete();
}
I have a EF code first context which represents a queue of jobs which a processing application can retrieve and run. These processing applications can be running on different machines but pointing at the same database.
The context provides a method that returns a QueueItem if there is any work to do, or null, called CollectQueueItem.
To ensure no two applications can pick up the same job, the collection takes place in a transaction with an ISOLATION LEVEL of REPEATABLE READ. This means that if there are two attempts to pick up the same job at the same time, one will be chosen as the deadlock victim and be rolled back. We can handle this by catching the DbUpdateException and return null.
Here is the code for the CollectQueueItem method:
public QueueItem CollectQueueItem()
{
using (var transaction = new TransactionScope(TransactionScopeOption.Required, new TransactionOptions { IsolationLevel = IsolationLevel.RepeatableRead }))
{
try
{
var queueItem = this.QueueItems.FirstOrDefault(qi => !qi.IsLocked);
if (queueItem != null)
{
queueItem.DateCollected = DateTime.UtcNow;
queueItem.IsLocked = true;
this.SaveChanges();
transaction.Complete();
return queueItem;
}
}
catch (DbUpdateException) //we might have been the deadlock victim. No matter.
{ }
return null;
}
}
I ran a test in LinqPad to check that this is working as expected. Here is the test below:
var ids = Enumerable.Range(0, 8).AsParallel().SelectMany(i =>
Enumerable.Range(0, 100).Select(j => {
using (var context = new QueueContext())
{
var queueItem = context.CollectQueueItem();
return queueItem == null ? -1 : queueItem.OperationId;
}
})
);
var sw = Stopwatch.StartNew();
var results = ids.GroupBy(i => i).ToDictionary(g => g.Key, g => g.Count());
sw.Stop();
Console.WriteLine("Elapsed time: {0}", sw.Elapsed);
Console.WriteLine("Deadlocked: {0}", results.Where(r => r.Key == -1).Select(r => r.Value).SingleOrDefault());
Console.WriteLine("Duplicates: {0}", results.Count(r => r.Key > -1 && r.Value > 1));
//IsolationLevel = IsolationLevel.RepeatableRead:
//Elapsed time: 00:00:26.9198440
//Deadlocked: 634
//Duplicates: 0
//IsolationLevel = IsolationLevel.ReadUncommitted:
//Elapsed time: 00:00:00.8457558
//Deadlocked: 0
//Duplicates: 234
I ran the test a few times. Without the REPEATABLE READ isolation level, the same job is retrieved by different theads (seen in the 234 duplicates). With REPEATABLE READ, jobs are only retrieved once but performance suffers and there are 634 deadlocked transactions.
My question is: is there a way to get this behaviour in EF without the risk of deadlocks or conflicts? I know in real life there will be less contention as the processors won't be continually hitting the database, but nonetheless, is there a way to do this safely without having to handle the DbUpdateException? Can I get performance closer to that of the version without the REPEATABLE READ isolation level? Or are Deadlocks not that bad in fact and I can safely ignore the exception and let the processor retry after a few millis and accept that the performance will be OK if the not all the transactions are happening at the same time?
Thanks in advance!
Id recommend a different approach.
a) sp_getapplock
Use an SQL SP that provides an Application lock feature
So you can have unique app behaviour, which might involve read from the DB or what ever else activity you need to control. It also lets you use EF in a normal way.
OR
b) Optimistic concurrency
http://msdn.microsoft.com/en-us/data/jj592904
//Object Property:
public byte[] RowVersion { get; set; }
//Object Configuration:
Property(p => p.RowVersion).IsRowVersion().IsConcurrencyToken();
a logical extension to the APP lock or used just by itself is the rowversion concurrency field on DB. Allow the dirty read. BUT when someone goes to update the record As collected, it fails if someone beat them to it. Out of the box EF optimistic locking.
You can delete "collected" job records later easily.
This might be better approach unless you expect high levels of concurrency.
As suggested by Phil, I used optimistic concurrency to ensure the job could not be processed more than once. I realised that rather than having to add a dedicated rowversion column I could use the IsLocked bit column as the ConcurrencyToken. Semantically, if this value has changed since we retrieved the row, the update should fail since only one processor should ever be able to lock it. I used the fluent API as below to configure this, although I could also have used the ConcurrencyCheck data annotation.
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
modelBuilder.Entity<QueueItem>()
.Property(p => p.IsLocked)
.IsConcurrencyToken();
}
I was then able to simple the CollectQueueItem method, losing the TransactionScope entirely and catching the more DbUpdateConcurrencyException.
public OperationQueueItem CollectQueueItem()
{
try
{
var queueItem = this.QueueItems.FirstOrDefault(qi => !qi.IsLocked);
if (queueItem != null)
{
queueItem.DateCollected = DateTime.UtcNow;
queueItem.IsLocked = true;
this.SaveChanges();
return queueItem;
}
}
catch (DbUpdateConcurrencyException) //someone else grabbed the job.
{ }
return null;
}
I reran the tests, you can see it's a great compromise. No duplicates, nearly 100x faster than with REPEATABLE READ, and no DEADLOCKS so the DBAs won't be on my case. Awesome!
//Optimistic Concurrency:
//Elapsed time: 00:00:00.5065586
//Deadlocked: 624
//Duplicates: 0
We are receiving a file from a client (Silverlight) via WCF and on the serverside I parse this file. Each line in the file is transformed into an object and stored into the database. if the file is very large (10000 entries and more), I get the following error (MSSQLEXPRESS):
The transaction associated with the current connection has completed but has not been disposed. The transaction must be disposed before the connection can be used to execute SQL statements.
I tried a lot (TransactionOptions timeout set and so on), but nothings works. The above exception message is either raised after 3000, sometimes after 6000 objects processed, but I can't succeed in processing all objects.
I append my source, hopefully somebody got an idea and can help me:
public xxxResponse SendLogFile (xxxRequest request
{
const int INTERMEDIATE_SAVE = 100;
using (var context = new EntityFramework.Models.Cubes_ServicesEntities())
{
// start a new transactionscope with the timeout of 0 (unlimited time for developing purposes)
using (var transactionScope = new TransactionScope(TransactionScopeOption.RequiresNew,
new TransactionOptions
{
IsolationLevel = System.Transactions.IsolationLevel.Serializable,
Timeout = TimeSpan.FromSeconds(0)
}))
{
try
{
// open the connection manually to prevent undesired close of DB
// (MSDTC)
context.Connection.Open();
int timeout = context.Connection.ConnectionTimeout;
int Counter = 0;
// read the file submitted from client
using (var reader = new StreamReader(new MemoryStream(request.LogFile)))
{
try
{
while (!reader.EndOfStream)
{
Counter++;
Counter2++;
string line = reader.ReadLine();
if (String.IsNullOrEmpty(line)) continue;
// Create a new object
DomainModel.LogEntry le = CreateLogEntryObject(line);
// an attach it to the context, set its state to added.
context.AttachTo("LogEntry", le);
context.ObjectStateManager.ChangeObjectState(le, EntityState.Added);
// while not 100 objects were attached, go on
if (Counter != INTERMEDIATE_SAVE) continue;
// after 100 objects, make a call to SaveChanges.
context.SaveChanges(SaveOptions.None);
Counter = 0;
}
}
catch (Exception exception)
{
// cleanup
reader.Close();
transactionScope.Dispose();
throw exception;
}
}
// do a final SaveChanges
context.SaveChanges();
transactionScope.Complete();
context.Connection.Close();
}
catch (Exception e)
{
// cleanup
transactionScope.Dispose();
context.Connection.Close();
throw e;
}
}
var response = CreateSuccessResponse<ServiceSendLogEntryFileResponse>("SendLogEntryFile successful!");
return response;
}
}
There is no bulk insert in entity framework. You call SaveChanges after 100 records but it will execute 100 separate inserts with database round trip for each insert.
Setting timeout of the transaction is also dependent on transaction max timeout which is configured on machine level (I think default value is 10 minutes). How lond does it take before your operation fails?
The best way you can do is rewriting your insert logic with common ADO.NET or with bulk insert.
Btw. throw exception and throw e? That is incorrect way to rethrow exceptions.
Important edit:
SaveChanges(SaveOptions.None) !!! means do not accept changes after saving so all records are still in added state. Because of that the first call to SaveChanges will insert first 100 records. The second call will insert first 100 again + next 100, the third call will insert first 200 + next 100, etc.
I had exactly same issue. I did EF code to insert bulk 1000 records each time.
I was working since the beginning, with a little problem with msDTC that I put to allow remot clients and admin , but after that it was ok. I did lot of work with this, but one day it JUST STOP WORKING.
I am getting
The transaction associated with the current connection has completed but has not been disposed. The transaction must be disposed before the connection can be used to execute SQL statements.
VERY WEIRD! Sometimes the error changes. My suspect is the msDTC somehow , strange behaviors.
I am changing now for not using TransactionScope!
I hate when it did work and just stop. I also tried to run this in a vm, another enourmous waste of time...
My code:
private void AddTicks(FileHelperTick[] fhTicks)
{
List<ForexEF.Entities.Tick> Ticks = new List<ForexEF.Entities.Tick>();
var str = LeTicks(ref fhTicks, ref Ticks);
using (TransactionScope scope = new TransactionScope(TransactionScopeOption.Required, new TransactionOptions()
{
IsolationLevel = System.Transactions.IsolationLevel.Serializable,
Timeout = TimeSpan.FromSeconds(180)
}))
{
ForexEF.EUR_TICKSContext contexto = null;
try
{
contexto = new ForexEF.EUR_TICKSContext();
contexto.Configuration.AutoDetectChangesEnabled = false;
int count = 0;
foreach (var tick in Ticks)
{
count++;
contexto = AddToContext(contexto, tick, count, 1000, true);
}
contexto.SaveChanges();
}
finally
{
if (contexto != null)
contexto.Dispose();
}
scope.Complete();
}
}
private ForexEF.EUR_TICKSContext AddToContext(ForexEF.EUR_TICKSContext contexto, ForexEF.Entities.Tick tick, int count, int commitCount, bool recreateContext)
{
contexto.Set<ForexEF.Entities.Tick>().Add(tick);
if (count % commitCount == 0)
{
contexto.SaveChanges();
if (recreateContext)
{
contexto.Dispose();
contexto = new ForexEF.EUR_TICKSContext();
contexto.Configuration.AutoDetectChangesEnabled = false;
}
}
return contexto;
}
It times out due the TransactionScope default Maximum Timeout, check the machine.config for that.
Check out this link:
http://social.msdn.microsoft.com/Forums/en-US/windowstransactionsprogramming/thread/584b8e81-f375-4c76-8cf0-a5310455a394/