I have a complex and big object graph that I want to insert in database by using a DbContext and SaveChanges method.
This object is a result of parsing a text file with 40k lines (around 3MB of data). Some collections inside this object have thousands of items.
I am able to parse the file correctly and add it to the context so that it can start tracking the object. But when I try to SaveChanges, it says:
Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while updating the entries. See the inner exception for details. ---> System.Data.SqlClient.SqlException: String or binary data would be truncated.
I would like to know if there is a smart and efficient way of discovering which object is causing the issue. It seems that a varchar field is too little to store the data. But it's a lot of tables and fields to check manually.
I would like to get a more specific error somehow. I already configured an ILoggerProvider and added the EnableSensitiveDataLogging option in my dbContext to be able to see which sql queries are being generated. I even added MiniProfiler to be able to see the parameter values, because they are not present in the log generated by the dbContext.
Reading somewhere in the web, I found out that in EF6 there is some validation that happens before the sql is passed to the database to be executed. But it seems that in EF Core this is not available anymore. So how can I solve this?
After some research, the only approach I've found to solve this, is implementing some validation by overriding dbContext's SaveChanges method. I've made a merge of these two approaches to build mine:
Implementing Missing Features in Entity Framework Core - Part 3
Validation in EF Core
The result is...
ApplicationDbContext.cs
public override int SaveChanges(bool acceptAllChangesOnSuccess)
{
ValidateEntities();
return base.SaveChanges(acceptAllChangesOnSuccess);
}
public override async Task<int> SaveChangesAsync(bool acceptAllChangesOnSuccess, CancellationToken cancellationToken = new CancellationToken())
{
ValidateEntities();
return await base.SaveChangesAsync(acceptAllChangesOnSuccess, cancellationToken);
}
private void ValidateEntities()
{
var serviceProvider = this.GetService<IServiceProvider>();
var items = new Dictionary<object, object>();
var entities = from entry in ChangeTracker.Entries()
where entry.State == EntityState.Added || entry.State == EntityState.Modified
select entry.Entity;
foreach (var entity in entities)
{
var context = new ValidationContext(entity, serviceProvider, items);
var results = new List<ValidationResult>();
if (Validator.TryValidateObject(entity, context, results, true)) continue;
foreach (var result in results)
{
if (result == ValidationResult.Success) continue;
var errorMessage = $"{entity.GetType().Name}: {result.ErrorMessage}";
throw new ValidationException(errorMessage);
}
}
}
Note that it's not necessary to override the other SaveChanges overloads, because they call these two.
The Error tells you that youre writing more characters to a field than it can hold.
This error for example would be thrown when you create a given field as NVARCHAR(4) or CHAR(4) and write 'hello' to it.
So you could simply check the length of the values you read in to find the one which is causing your problem. There is at least on which is too long for a field.
Related
When we pass our DbContext an object whose values have not changed, and try to perform an Update we get a 500 internal server error.
A user may open a dialog box to edit a record, change a value, change it back and then send the record to the database. Also we provide a Backup and Restore function and when the records are restored, some of them will not have changed since the backup was performed.
I was under the impression that a PUT would delete and re-create the record so I didn't feel there would be a problem.
For example, having checked that the Activity exists my ActivityController is as follows:
var activityEntityFromRepo = _activityRepository.GetActivity(id);
// Map(source object (Dto), destination object (Entity))
_mapper.Map(activityForUpdateDto, activityEntityFromRepo);
_activityRepository.UpdateActivity(activityEntityFromRepo);
// Save the updated Activity entity, added to the DbContext, to the SQL database.
if (await _activityRepository.SaveChangesAsync())
{
var activityFromRepo = _activityRepository.GetActivity(id);
if (activityFromRepo == null)
{
return NotFound("Updated Activity could not be found");
}
var activity = _mapper.Map<ActivityDto>(activityFromRepo);
return Ok(activity);
}
else
{
// The save failed.
var message = $"Could not update Activity {id} in the database.";
_logger.LogWarning(message);
throw new Exception(message);
};
My ActivityRepository is as follows:
public void UpdateActivity(Activity activity)
{
_context.Activities.Update(activity);
}
If any of the fields have changed then we don't get the error. Do I have to check every record for equality before the PUT? It seems unnecessary.
Perhaps I have missed something obvious. Any suggestions very welcome.
There is a lot of code missing here.
In your code you call your SaveChangesAsync (not the EF SaveChangesAsync).
Probably (but there is not the code to be sure) your SaveChangesAsync is something that returns false if there is an exception (and is not a good pattern because you "loose" the exception info) or if DbSet.SaveChangesAsync returns 0.
I think (but there is a lot of missing code) that this is your case. If you don't make any changes, SaveChangesAsync returns 0.
EDIT
The System.Exception is raised by your code (last line). EF never throws System.Exception.
Why does EF error interface is so unhelpful ? How can one find the exact cause for a DbContext.SaveChanges exception, when the errors tell almost nothing about the parameters involved in the exception?
One serious shortcoming of EF in my opinion is detecting what is causing a given exception after issuing SaveChanges().
It has been long since I have been struggling with EF error interface. I once helped myself a bit by writing a method that flattens all the db context validation errors providing straightforward messages with Table, Column and error message for each error. Otherwise one has to dig deep in the error structures dbContext returns... I wonder why they did this this way!
In any case, to my problem:
I perform several add operations in several entities in my dbContext, and in the end issue one single SaveChanges().
Now, I GetValidationErrors() returns 0 errors, so no validation issues.
But SaveChanges throws the following exception:
The conversion of a datetime2 data type to a datetime data type
resulted in an out-of-range value. The statement has been terminated.
How does DbContext points out the specific cause of the error in any way when what he passes over to you is a nested labyrinth of inner objects that looks like:
?!
If it's the DbEntityValidationException you can just override the SaveChanges method and grab the error and re throw it with the errors parsed out something like this:
public partial class MyEntities
{
public override int SaveChanges()
{
try
{
return base.SaveChanges();
}
catch (DbEntityValidationException ex)
{
var errorMessages = ex.EntityValidationErrors
.SelectMany(x => x.ValidationErrors)
.Select(x => x.ErrorMessage);
var fullErrorMessage = string.Join("\n\r", errorMessages);
var exceptionMessage = string.Concat(ex.Message, "The validation errors are: ", fullErrorMessage);
throw new DbEntityValidationException(exceptionMessage, ex.EntityValidationErrors);
}
}
}
Can u tell me what is the problem?
If you are using two different instances of the DbContext (the db variable as you named it) then nothing will be saved when you call SaveChanges on a context different than the one where your entities are tracked. You need to use the Attach method first.
db.customer_images.Attach(item);
db.SaveChanges();
However I think in your case you can avoid the attach step if you refactor a bit you code and don't use the DbContext from the entity itself.
Before going through my answer, you must check, if you are attaching the item as shown in excepted answer or check this code:.
if (dbStudentDetails != null && dbStudentDetails.Id != 0)
{
// update scenario
item.Id = dbStudentDetails.Id;
_context.Entry(dbStudentDetails).CurrentValues.SetValues(item);
_context.Entry(dbStudentDetails).State = EntityState.Modified;
}
else
{
// create scenario
_context.StudentDetails.Add(item);
}
await _context.SaveChangesAsync();
If above solution doesn't work, then check the below answer.
Saw a very wired issue, and thought to must answer this. as this can
be a major issue if you have lots of constraints and indexes in your
SQL.
db.SaveChanges() wasn't throwing any error, but not working (I have tried Exception or SqlException). This was not working because the Unique constraint was not defined properly while creating the Entity Models.
How you can Identified the issue:
I connected my SQL Server and opened the SQL Profiler.
Just before the db.SaveChanges(), I cleared all my profiler logs and ran the db.SaveChanges(). It logged the statement. I copied the script from the profiler and ran the script in SQL Server.
And bingo, I can see the actual error, which is being thrown at SQL Server side.
(images: have some hints, how you can get the execute statement from Profiler and run on sql server)
What you can do For Entity Framework Core:
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<Students>().HasIndex(p => new { p.RollNumber, p.PhoneNumber }).IsUnique(true).IsClustered(false).HasDatabaseName("IX_Students_Numbers");
}
What you can do For Entity Framework 6 and below:
using System.Data.Entity.ModelConfiguration;
internal partial class StudentsConfiguration : EntityTypeConfiguration<Students>
{
public StudentsConfiguration()
{
HasIndex(p => new { p.RollNumber, p.PhoneNumber }).IsUnique(true).IsClustered(false).HasName("IX_Students_Numbers");
}
}
Try to query your entity by Id, eg:
entity = this.repo.GetById(item.id);
entity.is_front = false;
if (dbSaveChanges() > 0)
{
....
}
Why is this not updating in the database, but the commented out version does work?
public bool InsertOrUpdateItems(Items item)
{
using (var dbContext = new MyEntities())
{
var items = dbContext.Items.Find(item.ItemId);
if (items != null)
{
dbContext.Items.Attach(items);
dbContext.Entry(items).State = EntityState.Modified;
//dbContext.Entry(items).CurrentValues.SetValues(item);
dbContext.SaveChanges();
}
There are no exceptions being thrown, it just wont update. Even though the record is found.
If the line in question is commented out, there are no new values to update. Notice that you:
Get object named items from DB
You attach it (although it is attached by default)
You try to save the same items
Object that you pass as a parameter for InsertOrUpdateItems method (item) is not even touched in your code, except reading its Id for use with Find()
I'm currently working on a project which is using EF Code First with POCOs. I have 5 POCOs that so far depends on the POCO "User".
The POCO "User" should refer to my already existing MemberShip table "aspnet_Users" (which I map it to in the OnModelCreating method of the DbContext).
The problem is that I want to take advantage of the "Recreate Database If Model changes" feature as Scott Gu shows at: http://weblogs.asp.net/scottgu/archive/2010/07/16/code-first-development-with-entity-framework-4.aspx - What the feature basically does is to recreate the database as soon as it sees any changes in my POCOs. What I want it to do is to Recreate the database but to somehow NOT delete the whole Database so that aspnet_Users is still alive. However it seems impossible as it either makes a whole new Database or replaces the current one with..
So my question is: Am I doomed to define my database tables by hand, or can I somehow merge my POCOs into my current database and still take use of the feature without wipeing it all?
As of EF Code First in CTP5, this is not possible. Code First will drop and create your database or it does not touch it at all. I think in your case, you should manually create your full database and then try to come up with an object model that matches the DB.
That said, EF team is actively working on the feature that you are looking for: altering the database instead of recreating it:
Code First Database Evolution (aka Migrations)
I was just able to do this in EF 4.1 with the following considerations:
CodeFirst
DropCreateDatabaseAlways
keeping the same connection string and database name
The database is still deleted and recreated - it has to be to for the schema to reflect your model changes -- but your data remains intact.
Here's how: you read your database into your in-memory POCO objects, and then after the POCO objects have successfully made it into memory, you then let EF drop and recreate the database. Here is an example
public class NorthwindDbContextInitializer : DropCreateDatabaseAlways<NorthindDbContext> {
/// <summary>
/// Connection from which to ead the data from, to insert into the new database.
/// Not the same connection instance as the DbContext, but may have the same connection string.
/// </summary>
DbConnection connection;
Dictionary<Tuple<PropertyInfo,Type>, System.Collections.IEnumerable> map;
public NorthwindDbContextInitializer(DbConnection connection, Dictionary<Tuple<PropertyInfo, Type>, System.Collections.IEnumerable> map = null) {
this.connection = connection;
this.map = map ?? ReadDataIntoMemory();
}
//read data into memory BEFORE database is dropped
Dictionary<Tuple<PropertyInfo, Type>, System.Collections.IEnumerable> ReadDataIntoMemory() {
Dictionary<Tuple<PropertyInfo,Type>, System.Collections.IEnumerable> map = new Dictionary<Tuple<PropertyInfo,Type>,System.Collections.IEnumerable>();
switch (connection.State) {
case System.Data.ConnectionState.Closed:
connection.Open();
break;
}
using (this.connection) {
var metaquery = from p in typeof(NorthindDbContext).GetProperties().Where(p => p.PropertyType.IsGenericType)
let elementType = p.PropertyType.GetGenericArguments()[0]
let dbsetType = typeof(DbSet<>).MakeGenericType(elementType)
where dbsetType.IsAssignableFrom(p.PropertyType)
select new Tuple<PropertyInfo, Type>(p, elementType);
foreach (var tuple in metaquery) {
map.Add(tuple, ExecuteReader(tuple));
}
this.connection.Close();
Database.Delete(this.connection);//call explicitly or else if you let the framework do this implicitly, it will complain the connection is in use.
}
return map;
}
protected override void Seed(NorthindDbContext context) {
foreach (var keyvalue in this.map) {
foreach (var obj in (System.Collections.IEnumerable)keyvalue.Value) {
PropertyInfo p = keyvalue.Key.Item1;
dynamic dbset = p.GetValue(context, null);
dbset.Add(((dynamic)obj));
}
}
context.SaveChanges();
base.Seed(context);
}
System.Collections.IEnumerable ExecuteReader(Tuple<PropertyInfo, Type> tuple) {
DbCommand cmd = this.connection.CreateCommand();
cmd.CommandText = string.Format("select * from [dbo].[{0}]", tuple.Item2.Name);
DbDataReader reader = cmd.ExecuteReader();
using (reader) {
ConstructorInfo ctor = typeof(Test.ObjectReader<>).MakeGenericType(tuple.Item2)
.GetConstructors()[0];
ParameterExpression p = Expression.Parameter(typeof(DbDataReader));
LambdaExpression newlambda = Expression.Lambda(Expression.New(ctor, p), p);
System.Collections.IEnumerable objreader = (System.Collections.IEnumerable)newlambda.Compile().DynamicInvoke(reader);
MethodCallExpression toArray = Expression.Call(typeof(Enumerable),
"ToArray",
new Type[] { tuple.Item2 },
Expression.Constant(objreader));
LambdaExpression lambda = Expression.Lambda(toArray, Expression.Parameter(typeof(IEnumerable<>).MakeGenericType(tuple.Item2)));
var array = (System.Collections.IEnumerable)lambda.Compile().DynamicInvoke(new object[] { objreader });
return array;
}
}
}
This example relies on a ObjectReader class which you can find here if you need it.
I wouldn't bother with the blog articles, read the documentation.
Finally, I would still suggest you always back up your database before running the initialization. (e.g. if the Seed method throws an exception, all your data is in memory, so you risk your data being lost once the program terminates.) A model change isn't exactly an afterthought action anyway, so be sure to back your data up.
One thing you might consider is to use a 'disconnected' foreign key. You can leave the ASPNETDB alone and just reference the user in your DB using the User key (guid). You can access the logged in user as follows:
MembershipUser currentUser = Membership.GetUser(User.Identity.Name, true /* userIsOnline */);
And then use the User's key as a FK in your DB:
Guid UserId = (Guid) currentUser.ProviderUserKey ;
This approach decouples your DB with the ASPNETDB and associated provider architecturally. However, operationally, the data will of course be loosely connected since the IDs will be in each DB. Note also there will be no referential constraints, whcih may or may not be an issue for you.