EF6 lazy loading: loads entire collection when adding? - entity-framework

I have the following collection navigation property on my entity:
public virtual ICollection<OrderIntegrationLogEntry> LogEntries { get; set; }
And the OrderIntegrationLogEntry entity has a little configuration:
this.HasKey(i => new {i.Id, i.IntegrationId});
this.HasRequired(i => i.Integration).WithMany(i => i.LogEntries).HasForeignKey(i => i.IntegrationId).WillCascadeOnDelete(true);
It seems that this line of code:
integration.LogEntries.Add(new OrderIntegrationLogEntry
{
Id = Guid.NewGuid(),
CreatedUtc = DateTime.UtcNow,
Message = message,
Level = level,
Detail = detail
});
...results in a query which loads the contents of the collection:
SELECT [Extent1].[Id] AS [Id], [Extent1].[IntegrationId] AS [IntegrationId],
[Extent1].[CreatedUtc] AS [CreatedUtc], [Extent1].[Level] AS [Level],
[Extent1].[Message] AS [Message], [Extent1].[Detail] AS [Detail]
FROM [dbo].[OrderIntegrationLogEntries] AS [Extent1]
WHERE [Extent1].[IntegrationId] = #EntityKeyValue1
I wasn't expecting this: shouldn't it just stage an add? Do I need to configure in some other way?

As noted by Ivan you've called the LogEntries getter, which causes Lazy Loading.
If you don't want to turn off Lazy Loading, then instead of adding the log entity to the navigation property of the parent, just set the IntegrationId of the new entitiy and SaveChanges(). eg
var entry = new OrderIntegrationLogEntry()
{
Id = Guid.NewGuid(),
IntegrationId = integration.Id,
CreatedUtc = DateTime.UtcNow,
Message = message,
Level = level,
Detail = detail
);
db.OrderIntegrationLogEntries.Add(entry);
db.SaveChanges();
Also if this is SQL Server (and probably other platforms too) use sequential GUID generation. Inserting a row with a random guid as its leading key column is needlessly expensive. For SQL Server you can generate sequential GUIDs in the database with the NEWSEQUENTIALID() function as a default, or on the client
public class SQLGuidUtil
{
[DllImport("rpcrt4.dll", SetLastError = true)]
static extern int UuidCreateSequential(out Guid guid);
public static Guid NewSequentialId()
{
Guid guid;
UuidCreateSequential(out guid);
var s = guid.ToByteArray();
var t = new byte[16];
t[3] = s[0];
t[2] = s[1];
t[1] = s[2];
t[0] = s[3];
t[5] = s[4];
t[4] = s[5];
t[7] = s[6];
t[6] = s[7];
t[8] = s[8];
t[9] = s[9];
t[10] = s[10];
t[11] = s[11];
t[12] = s[12];
t[13] = s[13];
t[14] = s[14];
t[15] = s[15];
return new Guid(t);
}
}
How to Generate Sequential GUIDs for SQL Server in .NET
You should also consider flipping the order of your key columns if you expect the log entries to be accessed primarily on an Integration-by-Integration basis. That will store the log entries for an Integration together. EG
this.HasKey(i => new {i.IntegrationId, i.Id});
If you're not on Windows you can roll your own sequential GUID generator by starting from a random GUID and incrementing the 4 low-order bytes. The GUIDs would be sequential only within an AppDomain, but that shouldn't matter much.
Something like this:
namespace NewSequentialId
{
public class SQLGuidUtil
{
static object synclock = new object();
static uint seq = 0;
static byte[] seed = Guid.NewGuid().ToByteArray();
public static Guid NewSequentialId()
{
uint nextVal;
byte[] buf;
lock (synclock)
{
nextVal = seq++;
buf = (byte[])seed.Clone();
if (nextVal == 0xFFFFFFFF)
{
seed = Guid.NewGuid().ToByteArray();
seq = 0;
}
}
var seqbytes = BitConverter.GetBytes(nextVal);
if (BitConverter.IsLittleEndian)
{
buf[0] = seqbytes[3];
buf[1] = seqbytes[2];
buf[2] = seqbytes[1];
buf[3] = seqbytes[0];
}
else
{
buf[0] = seqbytes[0];
buf[1] = seqbytes[1];
buf[2] = seqbytes[2];
buf[3] = seqbytes[3];
}
return new Guid(buf);
}
}
}

In order to enable lazy loading, EF creates proxy-classes derived from your model. In these classes it overrides the navigation properties to implement lazy loading. This is why your navigation properties should be defined virtual so EF can override them.
When you call integration.LogEntries.Add, the getter of your LogEntries property is being called which triggers lazy loading operation.
You can temporarily disable lazy loading using the following code:
context.Configuration.LazyLoadingEnabled = false;

Related

Get dependent ids when querying principal

I'm trying to get just the ids for dependents if a principal is queried, every time the principal is queried.
My initial thought is to add it somehow in the OnModelCreating definitions, however that appears to be limited to filtering down larger sets of data, unless I'm missing something.
Something like this:
builder.Entity<ListingModel>()
.AlsoDoThis(
x => x.MenuIds.AddRange(
Menus.Where(y => y.ListingId == x.Id).Select(y => y.Id).ToList()
)
);
There is a need to not do this in code for each individual place I have a Select, since that functionality is normalized in some base classes. The base classes have a <TModel> passed in and don't inherently know what properties need to be handled this way.
I do have a workaround where I'm grabbing everything with an AutoInclude(), then filtering it out in the model definition with customer getter/setter to return a list of ids. But rather than being more performant (grabbing related FK ids at the DB level) it's transferring all of that data to the server and then programmatically selecting a list of ids, as far as I understand it.
private List<int> _topicsIds = new();
[NotMapped]
public List<int> TopicsIds
{
get { return Topics.Count > 0 ? Topics.Select(x => x.Id).ToList() : _topicsIds; }
set { _topicsIds = value; }
}
public List<TopicModel> Topics { get; set; } = new();
"Extra SQL that gets called with every select in a context" is (to my limited knowledge) almost what HasQueryFilter does, with a just slightly broader operation. I think this is the approach I'm looking for, just selecting more stuff instead of filtering stuff out.
You can project everything via Select
var result = ctx.ListingModels
.Select(lm => new // or to DTO
{
Id = lm.Id,
OtherProperty = lm.OtherProperty,
Ids = x.MenuIds.Select(m => m.Id).ToList()
})
.ToList();
To make more general solution we can use annotations and define how to project such entities.
During Model defining:
builder.Entity<TopicModel>()
.WithProjection(
x => x.MenuIds,
x => x.Menus.Where(y => y.ListingId == x.Id).Select(y => y.Id).ToList()
);
Then usage in common code:
public virtual List<TModel> GetList(List<int> ids)
{
var list = _context.Set<TModel>().Where(x => ids.Any(id => id == x.Id))
.ApplyCustomProjection(_context)
.ToList();
return list;
}
ApplyCustomProjection(_context) will find previously defined annotation and will apply custom projection.
And extensions implementation:
public static class ProjectionExtensions
{
public const string CustomProjectionAnnotation = "custom:member_projection";
public class ProjectionInfo
{
public ProjectionInfo(MemberInfo member, LambdaExpression expression)
{
Member = member;
Expression = expression;
}
public MemberInfo Member { get; }
public LambdaExpression Expression { get; }
}
public static bool IsUnderDotnetTool { get; }
= Process.GetCurrentProcess().ProcessName == "dotnet";
public static EntityTypeBuilder<TEntity> WithProjection<TEntity, TValue>(
this EntityTypeBuilder<TEntity> entity,
Expression<Func<TEntity, TValue>> propExpression,
Expression<Func<TEntity, TValue>> assignmentExpression)
where TEntity : class
{
// avoid registering non serializable annotations during migrations update
if (IsUnderDotnetTool)
return entity;
var annotation = entity.Metadata.FindAnnotation(CustomProjectionAnnotation);
var projections = annotation?.Value as List<ProjectionInfo> ?? new List<ProjectionInfo>();
if (propExpression.Body is not MemberExpression memberExpression)
throw new InvalidOperationException($"'{propExpression.Body}' is not member expression");
if (memberExpression.Expression is not ParameterExpression)
throw new InvalidOperationException($"'{memberExpression.Expression}' is not parameter expression. Only single nesting is allowed");
// removing duplicate
projections.RemoveAll(p => p.Member == memberExpression.Member);
projections.Add(new ProjectionInfo(memberExpression.Member, assignmentExpression));
return entity.HasAnnotation(CustomProjectionAnnotation, projections);
}
public static IQueryable<TEntity> ApplyCustomProjection<TEntity>(this IQueryable<TEntity> query, DbContext context)
where TEntity : class
{
var et = context.Model.FindEntityType(typeof(TEntity));
var projections = et?.FindAnnotation(CustomProjectionAnnotation)?.Value as List<ProjectionInfo>;
// nothing to do
if (projections == null || et == null)
return query;
var propertiesForProjection = et.GetProperties().Where(p =>
p.PropertyInfo != null && projections.All(pr => pr.Member != p.PropertyInfo))
.ToList();
var entityParam = Expression.Parameter(typeof(TEntity), "e");
var memberBinding = new MemberBinding[propertiesForProjection.Count + projections.Count];
for (int i = 0; i < propertiesForProjection.Count; i++)
{
var propertyInfo = propertiesForProjection[i].PropertyInfo!;
memberBinding[i] = Expression.Bind(propertyInfo, Expression.MakeMemberAccess(entityParam, propertyInfo));
}
for (int i = 0; i < projections.Count; i++)
{
var projection = projections[i];
var expression = projection.Expression.Body;
var assignExpression = ReplacingExpressionVisitor.Replace(projection.Expression.Parameters[0], entityParam, expression);
memberBinding[propertiesForProjection.Count + i] = Expression.Bind(projection.Member, assignExpression);
}
var memberInit = Expression.MemberInit(Expression.New(typeof(TEntity)), memberBinding);
var selectLambda = Expression.Lambda<Func<TEntity, TEntity>>(memberInit, entityParam);
var newQuery = query.Select(selectLambda);
return newQuery;
}
}

ASP.NET Core Entity Framework SQL Query SELECT

I am one of the many struggling to "upgrade" from ASP.NET to ASP.NET Core.
In the ASP.NET project, I made database calls from my DAL like so:
var result = context.Database.SqlQuery<Object_VM>("EXEC [sp_Object_GetByKey] #Key",
new SqlParameter("#Key", Key))
.FirstOrDefault();
return result;
My viewmodel has additional fields that my object does not, such as aggregates of related tables. It seems unnecessary and counter intuitive to include such fields in a database / table structure. My stored procedure calculates all those things and returns the fields as should be displayed, but not stored.
I see that ASP.NET Core has removed this functionality. I am trying to continue to use stored procedures and load view models (and thus not have the entity in the database). I see options like the following, but as a result I get "2", the number of rows being returned (or another mysterious result?).
using(context)
{
string cmd = "EXEC [sp_Object_getAll]";
var result = context.Database.ExecuteSQLCommand(cmd);
}
But that won't work because context.Database.ExecuteSQLCommand is only for altering the database, not "selecting".
I've also seen the following as a solution, but the code will not compile for me, as "set" is really set<TEntity>, and there isn't a database entity for this viewmodel.
var result = context.Set().FromSql("EXEC [sp_Object_getAll]");
Any assistance much appreciated.
Solution:
(per Tseng's advice)
On the GitHub Entity Framework Issues page, there is a discussion about this problem. One user recommends creating your own class to handle this sort of requests, and another adds an additional method that makes it run smoother. I changed the methods slights to accept slightly different params.
Here is my adaptation (very little difference), for others that are also looking for a solution:
Method in DAL
public JsonResult GetObjectByID(int ID)
{
SqlParameter[] parms = new SqlParameter[] { new SqlParameter("#ID", ID) };
var result = RDFacadeExtensions.GetModelFromQuery<Object_List_VM>(context, "EXEC [sp_Object_GetList] #ID", parms);
return new JsonResult(result.ToList(), setting);
}
Additional Class
public static class RDFacadeExtensions
{
public static RelationalDataReader ExecuteSqlQuery(
this DatabaseFacade databaseFacade,
string sql,
SqlParameter[] parameters)
{
var concurrencyDetector = databaseFacade.GetService<IConcurrencyDetector>();
using (concurrencyDetector.EnterCriticalSection())
{
var rawSqlCommand = databaseFacade
.GetService<IRawSqlCommandBuilder>()
.Build(sql, parameters);
return rawSqlCommand
.RelationalCommand
.ExecuteReader(
databaseFacade.GetService<IRelationalConnection>(),
parameterValues: rawSqlCommand.ParameterValues);
}
}
public static IEnumerable<T> GetModelFromQuery<T>(
DbContext context,
string sql,
SqlParameter[] parameters)
where T : new()
{
DatabaseFacade databaseFacade = new DatabaseFacade(context);
using (DbDataReader dr = databaseFacade.ExecuteSqlQuery(sql, parameters).DbDataReader)
{
List<T> lst = new List<T>();
PropertyInfo[] props = typeof(T).GetProperties();
while (dr.Read())
{
T t = new T();
IEnumerable<string> actualNames = dr.GetColumnSchema().Select(o => o.ColumnName);
for (int i = 0; i < props.Length; ++i)
{
PropertyInfo pi = props[i];
if (!pi.CanWrite) continue;
System.ComponentModel.DataAnnotations.Schema.ColumnAttribute ca = pi.GetCustomAttribute(typeof(System.ComponentModel.DataAnnotations.Schema.ColumnAttribute)) as System.ComponentModel.DataAnnotations.Schema.ColumnAttribute;
string name = ca?.Name ?? pi.Name;
if (pi == null) continue;
if (!actualNames.Contains(name)) { continue; }
object value = dr[name];
Type pt = pi.DeclaringType;
bool nullable = pt.GetTypeInfo().IsGenericType && pt.GetGenericTypeDefinition() == typeof(Nullable<>);
if (value == DBNull.Value) { value = null; }
if (value == null && pt.GetTypeInfo().IsValueType && !nullable)
{ value = Activator.CreateInstance(pt); }
pi.SetValue(t, value);
}//for i
lst.Add(t);
}//while
return lst;
}//using dr
}

EF Code-first: Increment a non-autoincremented field manually

I´m using an existing database from our ERP.
In all my database tables, there is a float field called "r_e_c_n_o_", but this field is not auto-incremented by the database and I can´t change it.
For all added entities I would like to increment this field "r_e_c_n_o_", how could I acomplish that in DbContext´s SaveChanges() method?
Using ADO.NET I´d do something like that:
public static int GetNext(string tableName, string fieldName)
{
var cmd = _conn.CreateCommand(string.Format("SELECT MAX({0}) + 1 FROM {1}", fieldName, tableName));
var result = (int)cmd.ExecuteScalar();
return result;
}
UPDATE:
Please take a look in the comment below, its just what I need to solve my problem:
public override int SaveChanges()
{
var entries = this.ChangeTracker.Entries();
Dictionary<string, int> lastRecnos = new Dictionary<string, int>();
foreach (var entry in entries)
{
var typeName = entry.Entity.GetType().Name;
if (lastRecnos.ContainsKey(typeName))
lastRecnos[typeName]++;
else
lastRecnos[typeName] = 0;//How can i get the max here?
int nextRecnoForThisEntity = lastRecnos[typeName];
var entity = entry.Entity as EntityBase;
entity.Recno = nextRecnoForThisEntity;
}
return base.SaveChanges();
}
Tks,
William

How can I update my DTO's ID when inserting multiple new entities

I'm using EF4. I'm adding a series of new entities from a list of DTOs, and I'm not saving changes until after all of them are added. I'm wanting to set the IDs of the DTOs to what the new entities' IDs are. How on earth do I do this? Does EF provide a mechanism for this?
With a single entity I would do this:
public void InsertMyDto(MyDto a_dto)
{
var newEntity = new MyEntity
{
Name = a_dto.Name,
Type = a_dto.Type.ToString(),
Price = a_dto.Price
};
_dataContext.MyEntities.AddObject(newEntity);
_dataContext.SaveChanges();
a_dto.ID = newEntity.ID;
}
This works fine, but what do I do in this case?
public void InsertMyDtos(IEnumerable<MyDto> a_dtos)
{
foreach (var myDto in a_dtos)
{
var newEntity = new MyEntity
{
Name = myDto.Name,
Type = myDto.Type.ToString(),
Price = myDto.Price
};
// Does some validation logic against the database that might fail.
_dataContext.MyEntities.AddObject(newEntity);
}
_dataContext.SaveChanges();
// ???
}
I want to save all at once, because I have validation work (not shown above) that is done against the database and fails before it gets to SaveChanges, and if it fails I want it to fail as a whole transaction (i.e. rollback).
I don't think that EF can help you here. It even can't help you for a single instance which forces you to write a_dto.ID = newEntity.ID. The counterpart of this code for multiple entites is to keep track of the pairs of dtos and new entities:
public void InsertMyDtos(IEnumerable<MyDto> a_dtos)
{
Dictionary<MyDto, MyEntity> dict = new Dictionary<MyDto, MyEntity>();
foreach (var myDto in a_dtos)
{
var newEntity = new MyEntity
{
Name = myDto.Name,
Type = myDto.Type.ToString(),
Price = myDto.Price
};
dict.Add(myDto, newEntity);
// Does some validation logic against the database that might fail.
_dataContext.MyEntities.AddObject(newEntity);
}
_dataContext.SaveChanges();
foreach (var item in dict)
item.Key.ID = item.Value.ID; // Key is MyDto, Value is MyEntity
}

Inserting new record into linking table with linq to entities with POCO

I have a Team table and a Player table in many to many relationship. There is a linking table called TeamOnPlayer. EF with POCO generates navigation propertie called Person for the Team entity and also generates a nav. prop. called Team for the People entity.
I'm trying to insert a new record into the TeamOnPlayer table, but EF and POCO hides it. I tried to do this:
public static void AddPersonToTeam(int TeamId, int PersonId)
{
using (var ef = new korfballReportEntities())
{
var team = GetTeam(TeamId);
var person = GetPerson(PersonId);
team.Person.Add(person);
person.Team.Add(team);
ef.SaveChanges();
}
}
The GetTeam(TeamId) and GetPerson(PersonId) gets the right team and person:
public static Team GetTeam(int id)
{
using (var ef = new korfballReportEntities())
{
var q = from l in ef.Team
where l.Id == id
select l;
return q.Single();
}
}
public static Person GetPerson(int id)
{
using (var ef = new korfballReportEntities())
{
var query = from p in ef.Person
where p.Id == id
select p;
return query.Single();
}
}
When it tries to call the team.Person.Add(person) it throws an exception:
"The ObjectContext instance has been disposed and can no longer be used for operations that require a connection." System.Exception {System.ObjectDisposedException}
Can anyone please show me the correct way?
Edit
Now I understand what the problem was, thanks to you. I was a bit confused about the using blocks you included. For example this:
using (var ef = new korfballReportEntities())
{
//switch lazy loading off, only in this single context
ef.Configuration.LazyLoadingEnabled = false;
var repository = new MyRepository(ef);
repository.AddPersonToTeam(int TeamId, int PersonId);
}
Where should I put it?
I've done something else. I simply did this, and it worked fine.
public static void AddPersonToTeam(int TeamId, int PersonId)
{
using (var ef = new korfballReportEntities())
{
var q = from t in ef.Team
where t.Id == TeamId
select t;
var team = q.Single();
var q2 = from p in ef.Person
where p.Id == PersonId
select p;
var person = q2.Single();
try
{
team.Person.Add(person);
person.Team.Add(team);
}
catch (Exception e)
{
}
ef.SaveChanges();
}
}
The only problem is, that i coludn't reuse my GetPerson(int id) and GetTeam(int id) method.
What do you think? Is it okay? Is this an ugly way?
My guess is that you are working with lazy loading - your navigation properties Team.Person and Person.Team are marked as virtual in your entity classes. The result is that your methods GetTeam and GetPerson do not exactly return Team and Person objects but instances of dynamically created proxy classes which are derived from those entities. This dynamic proxy supports lazy loading which means that EF tries to load the navigation collections Team.Person and Person.Team when you access them for the first time. This happens in your AddPersonToTeam method when you call Add on these collections.
Now the problem is that the proxies are created within an context which you immediately dispose in your GetTeam and GetPerson methods (at the end of the using block). The proxies have stored a reference to this context internally and will use this context to load the navigation collections from the database.
Because these contexts are already disposed you get the exception.
You should redesign your code a bit: Don't create a new context in your repository methods GetTeam and GetPerson. You should instead use the same context for all operations: Retrieving the Team, retrieving the Person and adding the relationship. For example:
public static void AddPersonToTeam(int TeamId, int PersonId)
{
using (var ef = new korfballReportEntities())
{
var team = GetTeam(ef, TeamId);
var person = GetPerson(ef, PersonId);
team.Person.Add(person);
//person.Team.Add(team); <- not necessary, EF will handle this
ef.SaveChanges();
}
}
public static Team GetTeam(korfballReportEntities ef, int id)
{
var q = from l in ef.Team
where l.Id == id
select l;
return q.Single();
}
public static Person GetPerson(korfballReportEntities ef, int id)
{
var query = from p in ef.Person
where p.Id == id
select p;
return query.Single();
}
Another approach is to make your "Repository"/"Service" not static, inject the context into the constructor and then use this context throughout the repository. Then you don't need to pass in the context into every method. A rough sketch:
using (var ef = new korfballReportEntities())
{
var repository = new MyRepository(ef);
repository.AddPersonToTeam(int TeamId, int PersonId);
}
public class MyRepository
{
private readonly korfballReportEntities _ef;
public MyRepository(korfballReportEntities ef)
{
_ef = ef;
}
public void AddPersonToTeam(int TeamId, int PersonId)
{
var team = GetTeam(TeamId);
var person = GetPerson(PersonId);
team.Person.Add(person);
_ef.SaveChanges();
}
public Team GetTeam(int id)
{
var q = from l in _ef.Team
where l.Id == id
select l;
return q.Single();
}
public Person GetPerson(int id)
{
var query = from p in _ef.Person
where p.Id == id
select p;
return query.Single();
}
}
Edit
One little thing about performance tuning: In this specific case lazy loading is not necessary and more disturbing. It causes to load a (potentially long) collection team.Person when you want to add only one additional Person to the collection. You can switch off lazy loading for this particular operation (I refer to my second example):
using (var ef = new korfballReportEntities())
{
//switch lazy loading off, only in this single context
ef.Configuration.LazyLoadingEnabled = false;
var repository = new MyRepository(ef);
repository.AddPersonToTeam(int TeamId, int PersonId);
}
public void AddPersonToTeam(int TeamId, int PersonId)
{
var team = GetTeam(TeamId);
var person = GetPerson(PersonId);
// if lazy loading is off, the collecton is null, so we must instantiate one
if (team.Person == null)
team.Person = new List<Person>();
team.Person.Add(person);
_ef.SaveChanges();
}