When EF (2.2 & 3) at query worm up I get following warning:
The Include operation for navigation '[x].something' is unnecessary and was ignored because the navigation is not reachable in the final query results.
I know the cause of it, I'm looking to a solution to suppress it.
What causes it? an extension that returns a new type of list, sort of a book page, object contains the list of items on the page with 3 extra props, total of all items, page and page size.
public static BookPageList<T> ToBookPageList<T>(this IQueryable<T> dataSet, IBookPageFilter filter)
{
var result = new BookPageList<T>();
IQueryable<T> range = dataSet
.Skip((filter.Page- 1) * filter.PageSize)
.Take(filter.PageSize);
result.AddRange(range.ToList());
result.Total = dataSet.Count(); // here wormup warning happens
result.PageSize = filter.PageSize;
result.Page = filter.Page;
return result;
}
In case data set has includes, warning is shown.
var bookPageItems = dbcontext.Item
.Include(x => x.NavProp1)
.Include(x => x.NavProp2)
.OrderBy(x => x.Id)
.ToBookPageList(filter);
I want to remove the warnings because it pollutes cloud logs, sometimes are just to many and changing logging verbosity is not an option.
Is there any way how it can be handled in the book page extension method? I really like this extension.
Related
i've a problem with the IgnoreQueryFilters.
I've implemented soft-delete using the HasQueryFilter ( in the OnModelCreating i apply the global query filter to every entity which implements a particular interface ).
The problem is that if i launch a query 2 times in the same request:
the first time asking for also the "IsDeleted = true" entities ( so including IgnoreQueryFilters ),
and the second time asking only for the "IsDeleted = false" ( so not including the IgnoreQueryFilters)
the second time i still get also the "deleted" entities.
I think that this happens because when i launch the query for the second time, the entities are already in the context and i get them instead of the right results.
Here how i build the method for "including / excluding" the deleted entities.
// this is my repo pattern implementation
public class MyEntityRepo()
{
....
public Task<List<MyEntity>> GetEntityByUserId(int userId, bool ignoreQueryFilter = false)
{
var query = context.blabla
.Include(c => c.blabla2)
.Where(c => c.ApplicationUserId == userId);
if (ignoreQueryFilter)
{
query = query.IgnoreQueryFilters();
}
var result = await query.ToListAsync();
return result;
}
}
Now if in a service i call it this way:
public void MyServiceMethod()
{
...
var IncludeDeleted = await myEntityRepo.GetEntityByUserId(1, true);
//Here i need to do a sync with other data and for this reason i need also the deleted entities
foreach( var e in includeDeleted)
{
// do something
}
await context.SaveChangesAsync();
//Now that my data is correctly synced i've to get the data again but this time excluding the deleted entities
// and it fails
var ExcludeDeleted = await myEntityRepo.GetEntityByUserId(1, false);
return ExcludeDeleted;
}
The only way i found to solve the problem is to do something like context.ChangeTracker.Clear() before the second call to myEntityRepo.GetEntityByUserId, but is this the right way to go?
Since in real the method is a little bit more complex and can be re-used in other areas, i'm not sure that calling a Clear is a good idea because tomorrow it might be called in a bigger method and cause unexpected problems.
What's the best practice to use when i need to get data with and without query filter?
Is it ok to clear the change tracker?
If yes, what's the best time to clear it? in the GetEntityByUserId if i just ignoredTheFilters ( for consistency ) or after, in the caller method, whenever i find a problem like this one?
Actually i've also thinked about removing the GlobalQueryFilter usage and replace it with methods in the repos that get or exclude deleted entities... yes i've to remember to always filter out but feels more practical.
I have this model
In my database when I filter Ofertas by Organismo I get these results
Where I see that organismo with id=6 has 2 different Ofertas
When I translate this query to EF in my Web API
// GET: api/Ofertas/organismo/vasco
[HttpGet("organismo/{organismoId}")]
public IEnumerable<Oferta> GetOfertasByOrganismo(int organismoId)
{
var result = _context.Ofertas.Include(o => o.Concurso).ThenInclude(c
=> c.Organismo).Where(o => o.Concurso.Organismo.Id ==
organismoId).ToList();
return result;
And I launch this request http://localhost:5000/api/ofertas/organismo/6
I get these results
result
Count = 1
[0]: {Boletus_back_end.Domain.Oferta}
That is, only one Oferta
In my Web API I have my Db Context disconnected
services.AddDbContext<BoletusContext>(opt=>
opt.UseSqlServer(Configuration.GetConnectionString("BoletusConexion"))
.EnableSensitiveDataLogging()
.UseQueryTrackingBehavior(QueryTrackingBehavior.NoTracking));
Any idea, about how to obtain all the related Ofertas please?
Thanks
You will probably need to post code, but this issue commonly appears when projects disable lazy loading and then try to query against materialized entities where requested related data has not been loaded, or they have loaded that data eagerly and the related entity is optional.
For example, something like this:
var results = context.Ofertas.AsNoTracking().ToList();
var test = results.Where(x => x.Concurso.Organismo.Id == 1).ToList();
The issue here is that we have not fetched the Concursos or Oransimos of the Ofertas in the original query, and we have materialized the results with the first ToList call. The solution is either to ensure we put our Where condition somewhere where it can be integrated into the query, or we ensure we eager load all related data and we also ensure that optional/null references are accounted for:
a) Within the Query:
var results = context.Ofertas.AsNoTracking()
.Where(x => x.Concurso.Organismo.Id == 1).ToList();
This will pass because our query will look for and return Ofertas that have the desired Organismo ID, even if the relationship in the data is optional. However, it will not eager load the Concurso or Organismo with the Ofertas returned. Those references can still be null.
b) Eager load related data and check for #Null:
var results = context.Ofertas
.Include(x => x.Concurso)
.ThenInclude(x => x.Organismo)
.AsNoTracking()
.ToList();
var test = results.Where(x => x.Concurso?.Organismo?.Id == 1).ToList();
// or
var test = results.Where(x => x.Concuso != null
&& x.Concurso.Organismo != null
&& x.Concurso.Organismo.Id == 1).ToList();
Like the original query this pre-fetches all Ofertas but eager loads all related data. The null checks would only be needed if/when a relationship can be NULL. If the relationships are required and eager loaded then the null checks are not needed, but it requires the data to either be eager loaded, or capable of being lazy loaded.
Where this can get interesting is that EF will populate related entities if/where those entities are already tracked. So for instance if anything had previously loaded one or more of the Concurso's that the first Oferta result references, as well as the Organismo ID #1, then those references would exist in the returned query data In this case I will eager load the Concursos with the Oferta, but pre-fetch just Organismo #1:
var organismo = context.Organismos.Single(x => x.Id == 1);
// The DbContext is now tracking this entity, we do nothing with this instance.
var results = context.Ofertas.Include(x => x.Concurso).AsNoTracking.ToList();
var test = results[0].Concurso.Organismo.Id == 1; // Passes in our case.
var test2 = results.Where(x => x.Concurso.Organismo.Id == 1).ToList(); // can still fail!
The first test will pass because when we loaded the Ofertas from the DbContext, that context was already tracking Organismo #1 so it pre-populated that even when we don't eager load the organismos. However, unless all references happen to be #1, the rest of the Organismo references will be null and the second test will continue to fail.
Sorry
The problem was the data in Concursos Table the record that related the Ofertra to the Concurso was missing
Now all works fine
test
Count = 2
[0]: {Boletus_back_end.Domain.Oferta}
[1]: {Boletus_back_end.Domain.Oferta}
Thanks
I have an entity class which is mapped to an SQL table:
public class EntityItem {
public virtual ICollection<EntityItem2> SomeItems { get; set; }
}
And I have the following two snippets:
var items = _repository.Table.Where(x => x.Id == id)
.Select(x => new ItemModel {
Items = x.SomeItems.Select(y => new SomeItem { //mapping is here...}).ToList()
});
And
var items = _repository.Table.Where(x => x.Id == id).Select(x => someModelMapper.BuildModel(x));
//inside a mapper
public ItemModel BuildModel(EntityType entity){
var model = new ItemModel();
model.Items = entity.SomeItems.Select(x => anotherMapper.BuildModel(x));
return model;
}
As a result, I am getting different SQL queries in both cases. Moreover, the second snippet is working much slower than the first one. As I can see in SQL profiler the second snipper is generating many SQL queries.
So my questions:
Why is that happening?
How to create new objects like in the second snippet but to avoid
lots of SQL queries?
The likely reason you are seeing a difference in performance is due to EF Core materializing the query prematurely. When a Linq statement is compiled, EF attempts to translate it into SQL. If you call a function within the expression, EF6 would have raised an exception to the effect that the method cannot be converted to SQL. EF Core tries to be clever, and when it encounters a method it cannot convert, it executes the query up to the point it could get to, and then continues to execute the rest as Linq2Object where you method can run. IMO this is a pretty stupid feature and represents a huge performance landmine, and while it's fine to offer it as a possible option, it should be disabled by default.
You're probably seeing extra queries due to lazy loading after the main query runs, to populate the view models in the mapping method.
For instance if I execute:
var results = context.Parents.Select(x => new ParentViewModel
{
ParentId = x.ParentId,
Name = x.Name,
OldestChildName = x.Children.OrderByDescending(c => c.BirthDate).Select(c => c.Name).FirstOrDefault() ?? "No Child"
}).Single(x => x.ParentId == parentId);
That would execute as one statement. Calling a method to populate the view model:
var results = context.Parents
.Select(x => buildParentViewModel(x))
.Single(x => x.ParentId == parentId);
would execute something like:
var results = context.Parents
.ToList()
.Select(x => new ParentViewModel
{
ParentId = x.ParentId,
Name = x.Name,
OldestChildName = x.Children.OrderByDescending(c => c.BirthDate).Select(c =>
c.Name).FirstOrDefault() ?? "No Child"
}).Single(x => x.ParentId == parentId);
at worst or:
var results = context.Parents
.Where(x => x.ParentId == parentId)
.ToList()
.Select(x => new ParentViewModel
{
ParentId = x.ParentId,
Name = x.Name,
OldestChildName = x.Children.OrderByDescending(c => c.BirthDate).Select(c =>
c.Name).FirstOrDefault() ?? "No Child"
}).Single();
... at best. These are due to the Extra .ToList() call prior to the Select which is roughly what the premature execution will do automatically. The issue with these queries compared to the first one is that when it comes to loading the child's name. In the first query, the SQL generated pulls the parent and related child's details in one query. In the alternative cases the query will execute to pull the parent's details, but getting the child details will constitute a lazy load call to get more details since that will be executed as Linq2Object.
The solution would be to use Automapper and it's built in ProjectTo method to populate your view model. This will place the mapping code in automatically so that it works like the first scenario without you needing to write out all of the mapping code.
I'm working in Entity Framework Core 2.0, and I'm trying to do something like this. I want a list of all the classes taught by the oldest teacher. I tried to make the two requests into one query, but I couldn't make it work. Is there a concise way to make this one query?
private async Task<Teacher> GetOldestTeacher(int schoolId)
{
using (var db = new SchoolContext())
{
return await db.Teacher
.Where(t => t.SchoolId == schoolId)
.OrderByDescending(t => t.DateOfBirth)
.FirstAsync();
}
}
public async Task<IEnumerable<Class>> GetOldestTeachersClasses(int schoolId)
{
var oldestTeacher = await GetOldestTeacher(schoolId);
using (var db = new SchoolContext())
{
return await db.Class
.Where(c => c.TeacherId == oldestTeacher.Id && c.SchoolId == schoolId)
.ToListAsync();
}
}
This isn't exactly my code, but it's close enough to what I'm shooting for. This works, but I'm looking to make it more efficient. Any help would be appreciated.
Assuming there exists a navigation property called Classes in the Teacher class, you could do this:
public async Task<IEnumerable<Class>> GetOldestTeachersClasses(int schoolId)
{
using (var db = new SchoolContext())
{
return await db.Teacher
.Where(t => t.SchoolId == schoolId)
.OrderByDescending(t => t.DateOfBirth)
.Take(1)
.SelectMany(t => t.Classes)
.ToListAsync();
}
}
If you use First, you inevitably ask EF to immediately load the object into memory. Because of that, if you use that returned object to do the further parts of the query, you would actually be doing at least 2 roundtrips to the database and performing 2 queries instead of one despite in your C# code there being only one expression.
The trick here is that you filter the teachers just as you've already done, and instead of loading the first one using First(), you Take() the topmost one, according to the ordering requirements. The key difference is that Take will not actually load the object immediately - instead, it enables you to further specify the query which will be translated to one proper SQL query.
The reason for using SelectMany() is that semantically, there still can be more than one teachers as the Take() call actually returns an IQueryable<Teacher> object. But the semantics of your filter criteria ensures that in reality, there will only be 0 or 1 teacher, so collecting his/her classes using SelectMany will result in only his/her taught classes. You could also do the same by doing a Join as DevilSuichiro remarked in the comments.
However, make sure you verify the generated SQL. I personally don't yet have experience with EF Core 2.0 only with 1.x, which sometimes failed to translate rather trivial queries to proper SQL statements and instead performed a lot of work in my app. The last time I checked the 2.0 roadmap it was promised to significantly improve the translator, but I've written this code just from the top of my head so a proper verification is required.
So LazyLoadingEnabled = false apparently doesn't do what I thought it did. And has next to no documentation.
I design my entity trees very carefully. I think hard about every relationship. When I load up an entity that is an Aggregate Root, I want everything that is navigable to from there to load. That's how the Aggregate root concept works after all, if I want something to be weekly related I'd stick an id on the entity and manage the relationship myself.
Is there a way to do this with Entity Framework?
You can try obtaining all NavigationProperties of the element type (in IQueryable). By accessing to the MetadataWorkspace of ObjectContext you can get those properties of a specific type and Include all easily.
Note that I suppose you're using EF5 or later version, whereas DbContext is used. We access to ObjectContext via the interface IObjectContextAdapter. Here is the code:
public static IQueryable<T> LoadAllRelatedObjects<T>(this IQueryable<T> source, DbContext context) {
//obtain the EntityType corresponding to the ElementType first
//then we can get all NavigationProperties of the ElementType
var items = (ObjectItemCollection) ((IObjectContextAdapter)context).ObjectContext.MetadataWorkspace.GetItemCollection(DataSpace.OSpace);
var entityType = items.OfType<EntityType>().Single(e => items.GetClrType(e) == source.ElementType);
return entityType.NavigationProperties
.Aggregate(source, (c, e) => c.Include(e.Name));
}
Note in new version (since EF5), the namespace of ObjectItemCollection (and other metadata items) is System.Data.Entity.Core.Metadata.Edm, while in the old version (before EF5), it's System.Data.Metadata.Emd.
Usage:
yourModel.YourEntities.LoadAllRelatedObjects(yourModel)
//. more query here ...
;
I don't think there's a way to have all of the navigation properties auto-eager load. How about making an extension method instead?
public static IQueryable<Company> LoadCompany(this IQueryable<Company> query) {
return query.Include(x => x.Divisions)
.Include(x => x.Departments)
.Include(x => x.Employees);
}
Example:
var query = from company in db.Companies.LoadCompany()
where company.Name == "Microsoft"
select company;