I'm currently working with ASP.NET Core and I want to use Automapper to map a Linq Expression. The mapping statement is:
var targetConditions = _mapper.Map<Expression<Func<Entity, bool>>>(filter);
where filter is a formal parameter in the form:
(Expression<Func<EntityDTO, bool>> filter
In the mapping profile I have the following map created:
CreateMap<Expression<Func<EntityDTO, bool>>, Expression<Func<Entity, bool>>>();
I'm using a generic repository pattern with EF. I want to get a list of DTOs filtered by, of course, DTO's fields from my controller. I then need to convert from DTO filter to entities filter in the Business Layer before doing any query using Linq for EF.
Even though the expression does get coverted from EntityDTO to Entity, the parameters in the lambda expressions inside don't, raising all sorts of errors when I further use it with EF. Any idea how can this be done?
Try this:
IQueryable<Customer> query = repository.getCustomers(); // entities query
query.ProjectTo<CustomerDto>().Where(dtoFilterExpression)
We use Entity Framework and this leads to big performance problems whenever we use Count() on child collections of database entities.
As workaround I've used joins of root collections of data context. Then the resulting sql query uses the desired COUNT(*). But this solution is really ugly.
The slow query is:
var booked = erf.Sessions.All(s => s.Exams.All(e => e.Candidates.Count() >= e.CandidatesExpected))
If by "child collections" you mean navigation properties of type ICollection<T> defined in your entities, then it's Linq-to-Entities (not Linq-to-Sql, as you specified). Thus your Count() method is just an extension method defined in Enumerable class, which is executed on entities already materialized into memory. To get results you are expecting, you need to use Count() on DbSet queries.
I am thinking of designing a business rule engine which basically generates an EF query from a set of string values stored in a database.
For e.g. I will store the connection string, table name, the where condition predicate, and select predicate as string fields in a db and would like to construct the EF query dynamically. For e.g.
var db = new DbContext(“connectionstring”);
var wherePredicate = Expression.FromString(“p => p.StartDate > new DateTime(2014,5,1))
var selectPredicate = Expression.FromString(“p => p”)
var results = db.Set(“Projects”).Where(wherepredicate).Select(selectPredicate)
For constructing the predicates I can use DynamicExpression or Dynamic LINQ library.
However how do I access db.Set(“Projects”) where Projects is the entity name and apply the where and select predicates? (or something like db[“Projects”].Where().Select).
I tried the non-generic version of the DbContext.Set(Type entityttype) method, however couldn’t figure out how to apply Where and Select predicates to the returned object.
I am trying to avoid generating SQL queries and instead rely on dynamically generated EF code.
This doesn't make much sense. You can create method that will work on string instead of generic type using reflection, but you'd have to return DbSet not DBSet<T>. And on that one you cannot execute LINQ's methods (basically), because there's no type (during compilation). Of course you can do it all the way using reflection, but then, why??? You're loosing 90% of what O/R mapper does for you.
I know some differences of LINQ to Entities and LINQ to Objects which the first implements IQueryable and the second implements IEnumerable and my question scope is within EF 5.
My question is what's the technical difference(s) of those 3 methods? I see that in many situations all of them work. I also see using combinations of them like .ToList().AsQueryable().
What do those methods mean, exactly?
Is there any performance issue or something that would lead to the use of one over the other?
Why would one use, for example, .ToList().AsQueryable() instead of .AsQueryable()?
There is a lot to say about this. Let me focus on AsEnumerable and AsQueryable and mention ToList() along the way.
What do these methods do?
AsEnumerable and AsQueryable cast or convert to IEnumerable or IQueryable, respectively. I say cast or convert with a reason:
When the source object already implements the target interface, the source object itself is returned but cast to the target interface. In other words: the type is not changed, but the compile-time type is.
When the source object does not implement the target interface, the source object is converted into an object that implements the target interface. So both the type and the compile-time type are changed.
Let me show this with some examples. I've got this little method that reports the compile-time type and the actual type of an object (courtesy Jon Skeet):
void ReportTypeProperties<T>(T obj)
{
Console.WriteLine("Compile-time type: {0}", typeof(T).Name);
Console.WriteLine("Actual type: {0}", obj.GetType().Name);
}
Let's try an arbitrary linq-to-sql Table<T>, which implements IQueryable:
ReportTypeProperties(context.Observations);
ReportTypeProperties(context.Observations.AsEnumerable());
ReportTypeProperties(context.Observations.AsQueryable());
The result:
Compile-time type: Table`1
Actual type: Table`1
Compile-time type: IEnumerable`1
Actual type: Table`1
Compile-time type: IQueryable`1
Actual type: Table`1
You see that the table class itself is always returned, but its representation changes.
Now an object that implements IEnumerable, not IQueryable:
var ints = new[] { 1, 2 };
ReportTypeProperties(ints);
ReportTypeProperties(ints.AsEnumerable());
ReportTypeProperties(ints.AsQueryable());
The results:
Compile-time type: Int32[]
Actual type: Int32[]
Compile-time type: IEnumerable`1
Actual type: Int32[]
Compile-time type: IQueryable`1
Actual type: EnumerableQuery`1
There it is. AsQueryable() has converted the array into an EnumerableQuery, which "represents an IEnumerable<T> collection as an IQueryable<T> data source." (MSDN).
What's the use?
AsEnumerable is frequently used to switch from any IQueryable implementation to LINQ to objects (L2O), mostly because the former does not support functions that L2O has. For more details see What is the effect of AsEnumerable() on a LINQ Entity?.
For example, in an Entity Framework query we can only use a restricted number of methods. So if, for example, we need to use one of our own methods in a query we would typically write something like
var query = context.Observations.Select(o => o.Id)
.AsEnumerable().Select(x => MySuperSmartMethod(x))
ToList – which converts an IEnumerable<T> to a List<T> – is often used for this purpose as well. The advantage of using AsEnumerable vs. ToList is that AsEnumerable does not execute the query. AsEnumerable preserves deferred execution and does not build an often useless intermediate list.
On the other hand, when forced execution of a LINQ query is desired, ToList can be a way to do that.
AsQueryable can be used to make an enumerable collection accept expressions in LINQ statements. See here for more details: Do i really need use AsQueryable() on collection?.
Note on substance abuse!
AsEnumerable works like a drug. It's a quick fix, but at a cost and it doesn't address the underlying problem.
In many Stack Overflow answers, I see people applying AsEnumerable to fix just about any problem with unsupported methods in LINQ expressions. But the price isn't always clear. For instance, if you do this:
context.MyLongWideTable // A table with many records and columns
.Where(x => x.Type == "type")
.Select(x => new { x.Name, x.CreateDate })
...everything is neatly translated into a SQL statement that filters (Where) and projects (Select). That is, both the length and the width, respectively, of the SQL result set are reduced.
Now suppose users only want to see the date part of CreateDate. In Entity Framework you'll quickly discover that...
.Select(x => new { x.Name, x.CreateDate.Date })
...is not supported (at the time of writing). Ah, fortunately there's the AsEnumerable fix:
context.MyLongWideTable.AsEnumerable()
.Where(x => x.Type == "type")
.Select(x => new { x.Name, x.CreateDate.Date })
Sure, it runs, probably. But it pulls the entire table into memory and then applies the filter and the projections. Well, most people are smart enough to do the Where first:
context.MyLongWideTable
.Where(x => x.Type == "type").AsEnumerable()
.Select(x => new { x.Name, x.CreateDate.Date })
But still all columns are fetched first and the projection is done in memory.
The real fix is:
context.MyLongWideTable
.Where(x => x.Type == "type")
.Select(x => new { x.Name, DbFunctions.TruncateTime(x.CreateDate) })
(But that requires just a little bit more knowledge...)
What do these methods NOT do?
Restore IQueryable capabilities
Now an important caveat. When you do
context.Observations.AsEnumerable()
.AsQueryable()
you will end up with the source object represented as IQueryable. (Because both methods only cast and don't convert).
But when you do
context.Observations.AsEnumerable().Select(x => x)
.AsQueryable()
what will the result be?
The Select produces a WhereSelectEnumerableIterator. This is an internal .Net class that implements IEnumerable, not IQueryable. So a conversion to another type has taken place and the subsequent AsQueryable can never return the original source anymore.
The implication of this is that using AsQueryable is not a way to magically inject a query provider with its specific features into an enumerable. Suppose you do
var query = context.Observations.Select(o => o.Id)
.AsEnumerable().Select(x => x.ToString())
.AsQueryable()
.Where(...)
The where condition will never be translated into SQL. AsEnumerable() followed by LINQ statements definitively cuts the connection with entity framework query provider.
I deliberately show this example because I've seen questions here where people for instance try to 'inject' Include capabilities into a collection by calling AsQueryable. It compiles and runs, but it does nothing because the underlying object does not have an Include implementation anymore.
Execute
Both AsQueryable and AsEnumerable don't execute (or enumerate) the source object. They only change their type or representation. Both involved interfaces, IQueryable and IEnumerable, are nothing but "an enumeration waiting to happen". They are not executed before they're forced to do so, for example, as mentioned above, by calling ToList().
That means that executing an IEnumerable obtained by calling AsEnumerable on an IQueryable object, will execute the underlying IQueryable. A subsequent execution of the IEnumerable will again execute the IQueryable. Which may be very expensive.
Specific Implementations
So far, this was only about the Queryable.AsQueryable and Enumerable.AsEnumerable extension methods. But of course anybody can write instance methods or extension methods with the same names (and functions).
In fact, a common example of a specific AsEnumerable extension method is DataTableExtensions.AsEnumerable. DataTable does not implement IQueryable or IEnumerable, so the regular extension methods don't apply.
ToList()
Execute the query immediately
AsEnumerable()
lazy (execute the query later)
Parameter: Func<TSource, bool>
Load EVERY record into application memory, and then handle/filter them. (e.g. Where/Take/Skip, it will select * from table1, into the memory, then select the first X elements) (In this case, what it did: Linq-to-SQL + Linq-to-Object)
AsQueryable()
lazy (execute the query later)
Parameter: Expression<Func<TSource, bool>>
Convert Expression into T-SQL (with the specific provider), query remotely and load result to your application memory.
That’s why DbSet (in Entity Framework) also inherits IQueryable to get the efficient query.
Do not load every record, e.g. if Take(5), it will generate select top 5 * SQL in the background. This means this type is more friendly to SQL Database, and that is why this type usually has higher performance and is recommended when dealing with a database.
So AsQueryable() usually works much faster than AsEnumerable() as it generate T-SQL at first, which includes all your where conditions in your Linq.
ToList() will being everything in memory and then you will be working on it.
so, ToList().where ( apply some filter ) is executed locally.
AsQueryable() will execute everything remotely i.e. a filter on it is sent to the database for applying.
Queryable doesn't do anything til you execute it. ToList, however executes immediately.
Also, look at this answer Why use AsQueryable() instead of List()?.
EDIT :
Also, in your case once you do ToList() then every subsequent operation is local including AsQueryable(). You can't switch to remote once you start executing locally.
Hope this makes it a little bit more clearer.
Encountered a bad performance on below code.
void DoSomething<T>(IEnumerable<T> objects){
var single = objects.First(); //load everything into memory before .First()
...
}
Fixed with
void DoSomething<T>(IEnumerable<T> objects){
T single;
if (objects is IQueryable<T>)
single = objects.AsQueryable().First(); // SELECT TOP (1) ... is used
else
single = objects.First();
}
For an IQueryable, stay in IQueryable when possible, try not be used like IEnumerable.
Update. It can be further simplified in one expression, thanks Gert Arnold.
T single = objects is IQueryable<T> q?
q.First():
objects.First();
I'm using EF 5.0 Code First API, and I need to add the ability for building dynamic OrderBy expressions (the UI has a filter panel where users can choose multiple oredering criteria (e. g. select top 20 Customers, order by LastName ASC, then by Birst Date DESC).
ObjectContext API exposes OrderBy("it.PropertyName") method which looks great, despite it's missing compile time verification. However, I can't find any analogues in the DbContext API.
Googling the problem I found such advice:
((IObjectContextAdapter)myContextInstance).ObjectContext.CreateQuery<MyEntity>("<ESQL Query>")
However, I would not like to give up LINQ query at all.
Are there any ways to have something like this (mix ESQL and LINQ, as it could be by using ObjectContext):
var customers = myDbContext.Customers.OrderBy("it.LastName desc").Where(c => c.Age < 18)
Something like this? I'm not sure about mixing ESQL and LINQ, but your last query can be done entirely from LINQ.
var customers = myDbContext.Customers.OrderByDescending(a => a.LastName).ThenBy(c => c.BirthDate).Where(b => b.Age < 18);