I'm using EF Core in a netcoreapp3 app.
(I'm actually using DevArt's latest libs for Postgres but I don't think that's relevant here).
If I begin a transaction (repeatable read in this case):
var pg_conn = this.m_db.GetPGConnection();
pg_conn.Open();
// DevArt override BeginTransaction and allow you to pass in the isolation level
using (var trans = pg_conn.BeginTransaction(this.m_isolation))
{
trans.StateChanged += Trans_StateChanged;
try
{
T result = m_dlg(this.m_db); // This delegate does stuff and calls db.SaveChanges
trans.Commit(); // By the time you get here the transaction has been auto committed and disposed
return result;
}
do some stuff and call SaveChanges the transaction has been committed and disposed.
I attached an event to the transaction's StateChanged event to see where the call was coming from. The stack trace shows that the call is coming from MS EF (not DevArt):
WFDB.dll!WFDB.Contexts.TransactionRepeater<WFDB.Contexts.Workflow_DBContext, object>.Trans_StateChanged(object sender, Devart.Common.TransactionStateChangedEventArgs e) Line 249 C#
Devart.Data.dll!Devart.Common.DbTransactionBase.OnStateChanged(Devart.Common.TransactionAction action, System.Data.Common.DbConnection connection) Unknown
Devart.Data.PostgreSql.dll!Devart.Data.PostgreSql.PgSqlTransaction.Commit() Unknown
Devart.Data.PostgreSql.Entity.EFCore.dll!Devart.Common.Entity.cx.Commit() Unknown
Microsoft.EntityFrameworkCore.Relational.dll!Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.Commit() Unknown (The culprit? How do I stop this call?)
Microsoft.EntityFrameworkCore.Relational.dll!Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.Execute(Microsoft.EntityFrameworkCore.DbContext _, (System.Collections.Generic.IEnumerable<Microsoft.EntityFrameworkCore.Update.ModificationCommandBatch>, Microsoft.EntityFrameworkCore.Storage.IRelationalConnection) parameters) Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Storage.Internal.NoopExecutionStrategy.Execute<System.ValueTuple<System.__Canon, System.__Canon>, int>((System.__Canon, System.__Canon) state, System.Func<Microsoft.EntityFrameworkCore.DbContext, (System.__Canon, System.__Canon), int> operation, System.Func<Microsoft.EntityFrameworkCore.DbContext, (System.__Canon, System.__Canon), Microsoft.EntityFrameworkCore.Storage.ExecutionResult<int>> verifySucceeded) Unknown
Microsoft.EntityFrameworkCore.Relational.dll!Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.Execute(System.Collections.Generic.IEnumerable<Microsoft.EntityFrameworkCore.Update.ModificationCommandBatch> commandBatches, Microsoft.EntityFrameworkCore.Storage.IRelationalConnection connection) Unknown
Microsoft.EntityFrameworkCore.Relational.dll!Microsoft.EntityFrameworkCore.Storage.RelationalDatabase.SaveChanges(System.Collections.Generic.IList<Microsoft.EntityFrameworkCore.Update.IUpdateEntry> entries) Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(System.Collections.Generic.IList<Microsoft.EntityFrameworkCore.Update.IUpdateEntry> entriesToSave) Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(bool acceptAllChangesOnSuccess) Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.DbContext.SaveChanges(bool acceptAllChangesOnSuccess) Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.DbContext.SaveChanges() Unknown
WFDB.dll!WFDB.Contexts.Workflow_DBContext.SaveChanges() Line 107 C#
I don't want the transaction to auto commit. How can it be prevented? There surely must be some setting in EF Core to stop that?
Related
I'm using Spring batch to write a batch process and I'm having issues handling the exceptions.
I have a reader that fetches items from a database with an specific state. The reader passes the item to the processor step that can launch the exception MyException.class. When this exception is thrown I want to skip the item that caused that exception and continue reading the next one.
The issue here is that I need to change the state of that item in the database so it's not fetched again by the reader.
This is what I tried:
return this.stepBuilderFactory.get("name")
.<Input, Output>chunk(1)
.reader(reader())
.processor(processor())
.faultTolerant()
.skipPolicy(skipPolicy())
.writer(writer())
.build();
In my SkipPolicy class I have the next code:
public boolean shouldSkip(Throwable throwable, int skipCount) throws SkipLimitExceededException {
if (throwable instanceof MyException.class) {
// log the issue
// update the item that caused the exception in database so the reader doesn't return it again
return true;
}
return false;
}
With this code the exception is skipped and my reader is called again, however the SkipPolicy didn't commit the change or did a rollback, so the reader fetches the item and tries to process it again.
I also tried with an ExceptionHandler:
return this.stepBuilderFactory.get("name")
.<Input, Output>chunk(1)
.reader(reader())
.processor(processor())
.faultTolerant()
.skip(MyException.class)
.exceptionHandler(myExceptionHandler())
.writer(writer())
.build();
In my ExceptionHandler class I have the next code:
public void handleException(RepeatContext context, Throwable throwable) throws Throwable {
if (throwable.getCause() instanceof MyException.class) {
// log the issue
// update the item that caused the exception in database so the reader doesn't return it again
} else {
throw throwable;
}
}
With this solution the state is changed in the database, however it doesn't call the reader, instead it calls the method process of the processor() again, getting in an infinite loop.
I imagine I can use a listener in my step to handle the exceptions, but I don't like that solution because I will have to clone a lot of code asumming this exception could be launched in different steps/processors of my code.
What am I doing wrong?
EDIT: After a lot of tests and using different listeners like SkipListener, I couldn't achieve what I wanted, Spring Batch is always doing a rollback of my UPDATE.
Debugging this is what I found:
Once my listener is invoked and I update my item, the program enters the method write in the class FaultTolerantChunkProcessor (line #327).
This method will try the next code (copied from github):
try {
doWrite(outputs.getItems());
} catch (Exception e) {
status = BatchMetrics.STATUS_FAILURE;
if (rollbackClassifier.classify(e)) {
throw e;
}
/*
* If the exception is marked as no-rollback, we need to
* override that, otherwise there's no way to write the
* rest of the chunk or to honour the skip listener
* contract.
*/
throw new ForceRollbackForWriteSkipException(
"Force rollback on skippable exception so that skipped item can be located.", e);
}
The method doWrite (line #151) inside the class SimpleChunkProcessor will try to write the list of output items, however, in my case the list is empty, so in the line #159 (method writeItems) will launch an IndexOutOfBoundException, causing the ForceRollbackForWriteSkipException and doing the rollback I'm suffering.
If I override the class FaultTolerantChunkProcessor and I avoid writing the items if the list is empty, then everything works as intended, the update is commited and the program skips the error and calls the reader again.
I don't know if this is actually a bug or it's caused by something I'm doing wrong in my code.
A SkipListener is better suited to your use case than an ExceptionHandler in my opinion, as it gives you access to the item that caused the exception. With the exception handler, you need to carry the item in the exception or the repeat context.
Moreover, the skip listener allows you to know in which phase the exception happened (ie in read, process or write), while with the exception handler you need to find a way to detect that yourself. If the skipping code is the same for all phases, you can call the same method that updates the item's status in all the methods of the listener.
When we pass our DbContext an object whose values have not changed, and try to perform an Update we get a 500 internal server error.
A user may open a dialog box to edit a record, change a value, change it back and then send the record to the database. Also we provide a Backup and Restore function and when the records are restored, some of them will not have changed since the backup was performed.
I was under the impression that a PUT would delete and re-create the record so I didn't feel there would be a problem.
For example, having checked that the Activity exists my ActivityController is as follows:
var activityEntityFromRepo = _activityRepository.GetActivity(id);
// Map(source object (Dto), destination object (Entity))
_mapper.Map(activityForUpdateDto, activityEntityFromRepo);
_activityRepository.UpdateActivity(activityEntityFromRepo);
// Save the updated Activity entity, added to the DbContext, to the SQL database.
if (await _activityRepository.SaveChangesAsync())
{
var activityFromRepo = _activityRepository.GetActivity(id);
if (activityFromRepo == null)
{
return NotFound("Updated Activity could not be found");
}
var activity = _mapper.Map<ActivityDto>(activityFromRepo);
return Ok(activity);
}
else
{
// The save failed.
var message = $"Could not update Activity {id} in the database.";
_logger.LogWarning(message);
throw new Exception(message);
};
My ActivityRepository is as follows:
public void UpdateActivity(Activity activity)
{
_context.Activities.Update(activity);
}
If any of the fields have changed then we don't get the error. Do I have to check every record for equality before the PUT? It seems unnecessary.
Perhaps I have missed something obvious. Any suggestions very welcome.
There is a lot of code missing here.
In your code you call your SaveChangesAsync (not the EF SaveChangesAsync).
Probably (but there is not the code to be sure) your SaveChangesAsync is something that returns false if there is an exception (and is not a good pattern because you "loose" the exception info) or if DbSet.SaveChangesAsync returns 0.
I think (but there is a lot of missing code) that this is your case. If you don't make any changes, SaveChangesAsync returns 0.
EDIT
The System.Exception is raised by your code (last line). EF never throws System.Exception.
I have a complex and big object graph that I want to insert in database by using a DbContext and SaveChanges method.
This object is a result of parsing a text file with 40k lines (around 3MB of data). Some collections inside this object have thousands of items.
I am able to parse the file correctly and add it to the context so that it can start tracking the object. But when I try to SaveChanges, it says:
Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while updating the entries. See the inner exception for details. ---> System.Data.SqlClient.SqlException: String or binary data would be truncated.
I would like to know if there is a smart and efficient way of discovering which object is causing the issue. It seems that a varchar field is too little to store the data. But it's a lot of tables and fields to check manually.
I would like to get a more specific error somehow. I already configured an ILoggerProvider and added the EnableSensitiveDataLogging option in my dbContext to be able to see which sql queries are being generated. I even added MiniProfiler to be able to see the parameter values, because they are not present in the log generated by the dbContext.
Reading somewhere in the web, I found out that in EF6 there is some validation that happens before the sql is passed to the database to be executed. But it seems that in EF Core this is not available anymore. So how can I solve this?
After some research, the only approach I've found to solve this, is implementing some validation by overriding dbContext's SaveChanges method. I've made a merge of these two approaches to build mine:
Implementing Missing Features in Entity Framework Core - Part 3
Validation in EF Core
The result is...
ApplicationDbContext.cs
public override int SaveChanges(bool acceptAllChangesOnSuccess)
{
ValidateEntities();
return base.SaveChanges(acceptAllChangesOnSuccess);
}
public override async Task<int> SaveChangesAsync(bool acceptAllChangesOnSuccess, CancellationToken cancellationToken = new CancellationToken())
{
ValidateEntities();
return await base.SaveChangesAsync(acceptAllChangesOnSuccess, cancellationToken);
}
private void ValidateEntities()
{
var serviceProvider = this.GetService<IServiceProvider>();
var items = new Dictionary<object, object>();
var entities = from entry in ChangeTracker.Entries()
where entry.State == EntityState.Added || entry.State == EntityState.Modified
select entry.Entity;
foreach (var entity in entities)
{
var context = new ValidationContext(entity, serviceProvider, items);
var results = new List<ValidationResult>();
if (Validator.TryValidateObject(entity, context, results, true)) continue;
foreach (var result in results)
{
if (result == ValidationResult.Success) continue;
var errorMessage = $"{entity.GetType().Name}: {result.ErrorMessage}";
throw new ValidationException(errorMessage);
}
}
}
Note that it's not necessary to override the other SaveChanges overloads, because they call these two.
The Error tells you that youre writing more characters to a field than it can hold.
This error for example would be thrown when you create a given field as NVARCHAR(4) or CHAR(4) and write 'hello' to it.
So you could simply check the length of the values you read in to find the one which is causing your problem. There is at least on which is too long for a field.
Why does EF error interface is so unhelpful ? How can one find the exact cause for a DbContext.SaveChanges exception, when the errors tell almost nothing about the parameters involved in the exception?
One serious shortcoming of EF in my opinion is detecting what is causing a given exception after issuing SaveChanges().
It has been long since I have been struggling with EF error interface. I once helped myself a bit by writing a method that flattens all the db context validation errors providing straightforward messages with Table, Column and error message for each error. Otherwise one has to dig deep in the error structures dbContext returns... I wonder why they did this this way!
In any case, to my problem:
I perform several add operations in several entities in my dbContext, and in the end issue one single SaveChanges().
Now, I GetValidationErrors() returns 0 errors, so no validation issues.
But SaveChanges throws the following exception:
The conversion of a datetime2 data type to a datetime data type
resulted in an out-of-range value. The statement has been terminated.
How does DbContext points out the specific cause of the error in any way when what he passes over to you is a nested labyrinth of inner objects that looks like:
?!
If it's the DbEntityValidationException you can just override the SaveChanges method and grab the error and re throw it with the errors parsed out something like this:
public partial class MyEntities
{
public override int SaveChanges()
{
try
{
return base.SaveChanges();
}
catch (DbEntityValidationException ex)
{
var errorMessages = ex.EntityValidationErrors
.SelectMany(x => x.ValidationErrors)
.Select(x => x.ErrorMessage);
var fullErrorMessage = string.Join("\n\r", errorMessages);
var exceptionMessage = string.Concat(ex.Message, "The validation errors are: ", fullErrorMessage);
throw new DbEntityValidationException(exceptionMessage, ex.EntityValidationErrors);
}
}
}
I have not had any luck with transactions and entity framework 5. I have the following code:
context.Database.Connection.Open();
transaction = context.Database.Connection.BeginTransaction(IsolationLevel.Serializable);
//some work happens
context.SaveChanges();
//some additional work
context.SaveChanges();
transaction.Commit();
At the very first context.SaveChanges call, I get an exception: "Connection is already part of a local or a distributed transaction"
Right now I am actually just doing a trivial proof of concept where all I am doing is attaching an entity, marking it as modified and then calling save changes.
As a troubleshooting deal, I put in an event handler for when the connection state changes and had a breakpoint in there. Doing that, I verified that the connection did not close on me between when I started the transaction and when I called save changes.
Any help figuring out why it is giving me that exception would be tremendously appreciated.
This is the way we used transactions before. It worked for us:
public void DoSomething()
{
using (var db = GetContext())
{
using (var ts = GetTransactionScope())
{
//do stuff
db.SaveChanges();
ts.Complete();
}
}
}
public TransactionScope GetTransactionScope()
{
var tso = new TransactionOptions { IsolationLevel = IsolationLevel.ReadCommitted };
return new TransactionScope(TransactionScopeOption.Required, tso);
}
If for some reason you have to do multiple SaveChanges calls in one transaction the recommended way is to wrap them in a TransactionScope:
using(var tran = new TransactionScope())
{
using(var context = new MyContext())
{
//some work happens
context.SaveChanges();
//some additional work
context.SaveChanges();
}
tran.Complete(); // without this call the transaction is rolled back.
}
The default isolation level is serializable. Each connection that is opened within the transaction enlists in this transaction. By default, EF always opens and closes connections when it executes queries.
I guess the cause of this exception you've got is that EF creates a transaction object itself when it executes SaveChanges. It tries to use its connection to start this transaction, but the connection is already part of the transaction you created. By using a TransactionScope, the EF transaction just enlists in the ambient transaction.