Related
I don't understand why this doesn't translate. It seems to be exactly the use case described here.
The LINQ expression
DbSet<A>()
.GroupJoin(
inner: DbSet<B>(),
outerKeySelector: a => a.AId,
innerKeySelector: b => b.AId,
resultSelector: (a, bs) => new {
a = a,
bs = bs
})
produces the error:
could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
The LINQ code producing the exception is
from a in ctx.As
join b in ctx.Bs on a.aId equals b.aId into bs
select new {A = a, Bs = bs.ToList()};
Edit: maybe I misunderstood the doc and this is an example of something that does NOT translate.
Executing a query like the following example generates a result of Blog & IEnumerable. Since databases (especially relational databases) don't have a way to represent a collection of client-side objects, GroupJoin doesn't translate to the server in many cases. It requires you to get all of the data from the server to do GroupJoin without a special selector (first query below). But if the selector is limiting data being selected then fetching all of the data from the server may cause performance issues (second query below). That's why EF Core doesn't translate GroupJoin.
But then my question becomes instead : how do I achieve the result I'm looking for without requiring navigational properties ?
The explanation in the linked documentation just follows the EF Core team vision and is ridiculous, because of course it can easily be translated - I had a long discussion with the team here Query with GroupBy or GroupJoin throws exception #17068 and continue here Query: Support GroupJoin when it is final query operator #19930, trying to convince them why it should be supported, with no luck regardless of the arguments.
The whole point is (and that's the current workaround) it can be processed like if it was correlated subquery (SelectMany), which is translated and processed properly (even though the query result shape has no SQL equivalent.
Anyway, the current status is "Needs Design" (whatever that means), and the workaround is to replace the join with correlated subquery (which is what EF Core is using internally when "expanding" collection navigation properties during the query translation).
In your case, replace
join b in ctx.Bs on a.aId equals b.aId into bs
with
let bs = ctx.Bs.Where(b => a.aId == b.aId)
However, I highly recommend adding and using navigation properties. Not sure why you "can't use" them, in LINQ to Entities which do not project entities they serve just metadata for relationships, thus produce automatically the necessary joins. By not defining them you just put on yourself unneeded limitations (additionally to EF Core limitations/bugs). In general EF Core works better and support more things when using navigation properties instead of manual joins.
Try this query, it should work with EF Core:
var query =
from a in ctx.As
select new {A = a, Bs = ctx.Bs.Where(b => b.Id == a.aId).ToList()};
Probably the .ToList() cannot be translated. Use include instead
var result = ctx.As
.Include(a => a.Bs)
.ToList();
Where you must have a navigation property for the Bs in the A class:
public class A
{
public int aId { get; set; }
public List<B> Bs { get; set; }
}
See:
Eager Loading of Related Data
Relationships - EF Core
EF Core will not allow you to do a GroupJoin without following up with a SelectMany in order to flatten the list. The GroupJoin has no equivalent implementation in SQL, however the GroupJoin/SelectMany is equivalent to an inner-join or left-join (depending on if you use DefaultIfEmpty) so it works fine:
context.Users.GroupJoin(
context.UserRoles,
u => u.UserId,
r => r.UserId,
(user, roles) => new { user, roles })
//Will not work without this line
.SelectMany(x => x.roles.DefaultIfEmpty(), (x, r) => new { x.user, role = r })
.ToList();
If you actually want your results to be grouped (as opposed to trying to do a left-join) you have a few options:
You can materialize the results of a left join, then group the results in-memory (the code below uses my LeftJoin function shown in LEFT OUTER JOIN in LINQ):
context.Users.LeftJoin(
context.UserRoles,
u => u.UserId,
r => r.UserId,
(user, roles) => new { user, roles })
.ToList()
.GroupBy(x => x.user, (u, x) => new
{
User = u,
Roles = x.Select(z => z.role).Where(r => r != null).ToList()
})
.ToList();
You can use a sub-query. Note that EF is smart enough to use a left-join when it generates the SQL:
context.Users.Select(u => new
{
User = u,
Roles = context.UserRoles.Where(r => r.UserId == u.UserId).ToList()
})
.ToList();
If you prefer the GroupJoin syntax, but don't want to have to keep calling all the other functions to flatten, materialize, then re-group the results, you can use my JoinMany() extension method. This method uses the sub-query approach but wraps it in a generic method that looks very similar to the GroupJoin function:
context.Users.JoinMany(
context.UserRoles,
(u, r) => u.UserId == r.UserId,
(user, roles) => new { user, roles })
.ToList();
Supporting code:
public static class QueryableExtensions
{
public static IQueryable<TResult> LeftJoin<TOuter, TInner, TKey, TResult>(
this IQueryable<TOuter> outer,
IEnumerable<TInner> inner, Expression<Func<TOuter, TKey>> outerKeySelector,
Expression<Func<TInner, TKey>> innerKeySelector,
Expression<Func<TOuter, TInner, TResult>> resultSelector)
{
return outer
.GroupJoin(inner, outerKeySelector, innerKeySelector, (o, i) => new { o, i })
.SelectMany(o => o.i.DefaultIfEmpty(), (x, i) => new { x.o, i })
.ApplySelector(x => x.o, x => x.i, resultSelector);
}
public static IQueryable<TResult> JoinMany<TOuter, TInner, TResult>(
this IQueryable<TOuter> outers, IQueryable<TInner> inners,
Expression<Func<TOuter, TInner, bool>> condition,
Expression<Func<TOuter, IEnumerable<TInner>, TResult>> resultSelector)
{
//Use a placeholder "p => true" expression for the sub-query
Expression<Func<TOuter, JoinResult<TOuter, IEnumerable<TInner>>>> joinSelector = o =>
new JoinResult<TOuter, IEnumerable<TInner>> { Outer = o, Inner = inners.Where(p => true) };
//Create the where-clause that will be used for the sub-query
var whereClause = Expression.Lambda<Func<TInner, bool>>(
condition.Body.ReplaceParameter(condition.Parameters[0], joinSelector.Parameters[0]),
condition.Parameters[1]);
//Replace the placeholder expression with our new where clause
joinSelector = Expression.Lambda<Func<TOuter, JoinResult<TOuter, IEnumerable<TInner>>>>(
joinSelector.Body.VisitExpression(node =>
(node is LambdaExpression le && le.Parameters.Count == 1 && le.Parameters[0].Type == typeof(TInner)
&& le.Body is ConstantExpression ce && ce.Value is bool b && b)
? whereClause : null),
joinSelector.Parameters[0]);
return outers.Select(joinSelector).ApplySelector(x => x.Outer, x => x.Inner, resultSelector);
}
private static IQueryable<TResult> ApplySelector<TSource, TOuter, TInner, TResult>(
this IQueryable<TSource> source,
Expression<Func<TSource, TOuter>> outerProperty,
Expression<Func<TSource, TInner>> innerProperty,
Expression<Func<TOuter, TInner, TResult>> resultSelector)
{
var p = Expression.Parameter(typeof(TSource), $"param_{Guid.NewGuid()}".Replace("-", string.Empty));
Expression body = resultSelector?.Body
.ReplaceParameter(resultSelector.Parameters[0], outerProperty.Body.ReplaceParameter(outerProperty.Parameters[0], p))
.ReplaceParameter(resultSelector.Parameters[1], innerProperty.Body.ReplaceParameter(innerProperty.Parameters[0], p));
var selector = Expression.Lambda<Func<TSource, TResult>>(body, p);
return source.Select(selector);
}
public class JoinResult<TOuter, TInner>
{
public TOuter Outer { get; set; }
public TInner Inner { get; set; }
}
}
public static class ExpressionExtensions
{
public static Expression ReplaceParameter(this Expression source, ParameterExpression toReplace, Expression newExpression)
=> new ReplaceParameterExpressionVisitor(toReplace, newExpression).Visit(source);
public static Expression VisitExpression(this Expression source, Func<Expression, Expression> onVisit)
=> new DelegateExpressionVisitor (onVisit).Visit(source);
}
public class DelegateExpressionVisitor : ExpressionVisitor
{
Func<Expression, Expression> OnVisit { get; }
public DelegateExpressionVisitor(Func<Expression, Expression> onVisit)
{
this.OnVisit = onVisit;
}
public override Expression Visit(Expression node)
{
return OnVisit(node) ?? base.Visit(node);
}
}
public class ReplaceParameterExpressionVisitor : ExpressionVisitor
{
public ParameterExpression ToReplace { get; }
public Expression ReplacementExpression { get; }
public ReplaceParameterExpressionVisitor(ParameterExpression toReplace, Expression replacement)
{
this.ToReplace = toReplace;
this.ReplacementExpression = replacement;
}
protected override Expression VisitParameter(ParameterExpression node)
=> (node == ToReplace) ? ReplacementExpression : base.VisitParameter(node);
}
In my MVC web application I am creating a search function where i need to compare a search string with objects in my product repository - But how do I make sure that the search is not case sensitive?
I can use ToLower() on my search string - but the repository?
Controller:
public ActionResult Search(string q, int page = 1)
{
string search = q.ToLower();
int productCounter = repository.Products.Where(p => p.Name.Contains(search) || p.Description.Contains(search)).Count();
ProductsListViewModel model = new ProductsListViewModel
{
Products = repository.Products
.Where(p => p.Name.Contains(search) || p.Description.Contains(search))
.OrderBy(p => p.ProductID)
.Skip((page - 1) * PageSize)
.Take(PageSize),
PagingInfo = new PagingInfo
{
CurrentPage = page,
ItemsPerPage = PageSize,
TotalItems = productCounter == 0 ? 0 : productCounter
}
};
return View("List", model);
}
You can replace this:
p.Name.Contains(search)
with this:
p.Name.IndexOf(search, StringComparison.OrdinalIgnoreCase) >= 0
You can use IndexOf to get access to IgnoreCase in In memory collections.
You may be disappointed when you apply this to EF provider scenarios.
Is this correctly tagged EF ?
EF Provider specification does not use IndexOf.
For that matter the Contains(str,comparer) is also not supported.
Supported Linq to Entities features
If using SQLServer with EF the original issue is governed by the column collation property. eg SQL_Latin1_General_CP1_CI_AS case insensitive latin.
If you have DB first, you control the collation sequence at DB level.
All explained here nicely ... LINQ to Entities case sensitive comparison
If using code first the DEFAULT sql server db collation is used.
more info on default collation Set database collation in Entity Framework Code-First Initializer
You can use Invariant Culture and enhance the query a bit by using Length as well:
public ActionResult Search(string q, int page = 1)
{
string s;
string search = q.ToUpperInvariant();
int productCounter = repository.Products.Where(p => p.Name.ToUpperInvariant().Contains(search) || p.Description.ToUpperInvariant().Contains(search)).Count();
int searchlength = search.Length;
ProductsListViewModel model = new ProductsListViewModel
{
Products = repository.Products
.Where(p => (p.Name.Length >= searchlength && p.Name.ToUpperInvariant().Contains(search)) || (p.Description.Length >= searchlength && p.Description.ToUpperInvariant().Contains(search)))
.OrderBy(p => p.ProductID)
.Skip((page - 1) * PageSize)
.Take(PageSize),
PagingInfo = new PagingInfo
{
CurrentPage = page,
ItemsPerPage = PageSize,
TotalItems = productCounter == 0 ? 0 : productCounter
}
};
return View("List", model);
}
I am trying to create a multiple include method in my repository to use as follows:
repository.Include<Post>(x => x.Images, x => x.Tags).First(x => x.Id == 1)
I tried something as:
public IQueryable<T> Include<T>(params Expression<Func<T, Object>>[] paths) where T : class {
return paths.Aggregate(_context.Set<T>(), (x, path) => x.Include(path));
} // Include
But I get the error:
Cannot implicitly convert type 'System.Linq.IQueryable' to 'System.Data.Entity.DbSet'.
Note that the original include is the following:
public static IQueryable Include(
this IQueryable source,
Expression> path
) where T : class;
Can I make this work without turning my repository method into static?
Thank You,
Miguel
If you really want to create your own .Include non-extension method that allows for multiple paths in one call, internally translating to the already provided .Include method, you can do something like
public IQueryable<T> Include<T>(params Expression<Func<T, object>>[] paths)
where T : class
{
IQueryable<T> query = _context.Set<T>();
foreach (var path in paths)
query = query.Include(path);
return query;
}
This is pretty close to what you have already, but avoids the pitfall with Enumerable.Aggregate that you encountered: you'd have the same problem if you replace IQueryable<T> query with var query in my version.
Note that using many .Include may harm performance. If it does in your situation, you can rewrite it to use multiple queries, which you can run one after the other in a transaction.
Personally, as you can call the already provided .Include extension method (using System.Data.Entity;) on any IQueryable, I'd just write it as:
repository.Posts.Include(x => x.Images).Include(x => x.Tags).First(x => x.Id == 1)
public ICollection<SomeClass> Filter(string name, int skip, int take,out int total, params Expression<Func<SomeClass, object>>[] includeProperties) {
IQueryable<SomeClass> query = Session.All<SomeClass>().AsNoTracking();
//query = includeProperties.Aggregate(query, (current, property) => current.Include(property));
foreach (var property in includeProperties){
query = query.Include(property);
}
if (!string.IsNullOrWhiteSpace(name)){
query = query.Where(x => x.Name.Contains(name));
var page = query.OrderBy(x => x.Name)
.Skip(skip)
.Take(take)
.GroupBy(p => new{Total = query.Count()})
.FirstOrDefault();
total = (page != null) ? page.Key.Total : 0;
if (page == null) {
return new List<SomeClass>();
}
return page.ToList();
} else {
var page = query.OrderBy(x => x.Name)
.Skip(skip)
.Take(take)
.GroupBy(p => new { Total = query.Count() })
.FirstOrDefault();
total = (page != null) ? page.Key.Total : 0;
if (page == null) {
return new List<SomeClass>();
}
return page.ToList();
}
}
I guess the shortest way is as follows:
public static class LinqExtensions
{
/// <summary>
/// Acts similar of .Include() LINQ method, but allows to include several object properties at once.
/// </summary>
public static IQueryable<T> IncludeMultiple<T>(this IQueryable<T> query, params Expression<Func<T, object>>[] paths)
where T : class
{
foreach (var path in paths)
query = query.Include(path);
return query;
}
}
I am using Entity Framework version 4. I need to compare a large (~1 million record) SQL Server table to a longish (~2000) array of complex objects returned from a web service. Five different properties need to be compared to determine whether an instance of the complex object is already in the database.
I created a function that returns an expression for use in .Where and .Any methods. It looks like this (where A is the complex object, and tblA is the EF class):
function Expression<tblA, bool> GetSearchPredicate(A a)
{
return ta => ta.Field1.Equals(a.Field1)
&& ta.Field2.Equals(a.Field2)
&& ta.Field3.Equals(a.Field3)
&& ta.Field4.Equals(a.Field4)
&& ta.Field5.Equals(a.Field5);
}
This works. And I can compare all 2000 instances of A by doing this:
IEnumerable<A> objects = [web service call];
var result = objects.Select(a => !db.tblA.Any(GetSearchPredicate(a)));
That works, too. But it's slow. So I looked into building a utility method that could build an expression that could be transmitted down to the database directly through EF.
I used the code in this question as a basis for building that utility method. The example in that question shows comparing a single property to a series of constants, while my version would have to compare multiple properties to multiple constants. My best effort is below:
public static IQueryable<TEntity> WhereIn<TEntity>
(
this ObjectQuery<TEntity> query,
IEnumerable<Expression<Func<TEntity, bool>>> predicates
)
{
if (predicates == null) throw new ArgumentNullException("predicates");
IEnumerable<ParameterExpression> p = predicates.Select(pred => pred.Parameters.Single()).ToArray();
IEnumerable<Expression> equals = predicates.Select(value =>
(Expression)value.Body);
Expression bigEqual = equals.Aggregate((accumulate, equal) =>
Expression.Or(accumulate, equal));
var result1 = Expression.Lambda<Func<TEntity, bool>>(bigEqual, p.First());
var result = query.Where(result1);
return result;
}
This would be invoked like this:
IEnumerable<A> objects = [web service call];
var result = db.tblA.WhereIn(objects.Select(a => GetSearchPredicate(a)));
What I get is a message saying that "ta" (the placeholder for the TEntity object) is not bound. I thought this was because I had multiple expressions (the variable predicates) being combined, and maybe this message was being thrown because I was only passing the parameter from the first of the predicates IEnumerable. But this happens even if predicates is one expression long.
I am reasonably sure, based on the method I linked to, that I could build an expression comparing each of the five properties to a constant (the values of A.Field1 through A.Field5), rather than passing in the parameter predicates that already has them assembled into a series of expressions. But I would rather not, since that would require my method to know that it's working with types A and tblA, and that's the opposite of generic and general-purpose. (It'd also be complex and messy.)
I hope the examples I've shown explain what I want to do. Can it be done in a generic way?
You will need to replace the parameter in the predicate bodies with a single parameter. Something like this should work:
public static Expression<Func<T, bool>> BuildOr<T>(
IEnumerable<Expression<Func<T, bool>>> predicates)
{
Expression body = null;
ParameterExpression p = null;
Expression<Func<T, bool>> first = null;
foreach (Expression<Func<T, bool>> item in predicates)
{
if (first == null)
{
first = item;
}
else
{
if (body == null)
{
body = first.Body;
p = first.Parameters[0];
}
var toReplace = item.Parameters[0];
var itemBody = ReplacementVisitor.Transform(item, toReplace, p);
body = Expression.OrElse(body, itemBody);
}
}
if (first == null)
{
throw new ArgumentException("Sequence contains no elements.", "predicates");
}
return (body == null) ? first : Expression.Lambda<Func<T, bool>>(body, p);
}
private sealed class ReplacementVisitor : ExpressionVisitor
{
private IList<ParameterExpression> SourceParameters { get; set; }
private Expression ToFind { get; set; }
private Expression ReplaceWith { get; set; }
public static Expression Transform(
LambdaExpression source,
Expression toFind,
Expression replaceWith)
{
var visitor = new ReplacementVisitor
{
SourceParameters = source.Parameters,
ToFind = toFind,
ReplaceWith = replaceWith,
};
return visitor.Visit(source.Body);
}
private Expression ReplaceNode(Expression node)
{
return (node == ToFind) ? ReplaceWith : node;
}
protected override Expression VisitConstant(ConstantExpression node)
{
return ReplaceNode(node);
}
protected override Expression VisitBinary(BinaryExpression node)
{
var result = ReplaceNode(node);
if (result == node) result = base.VisitBinary(node);
return result;
}
protected override Expression VisitParameter(ParameterExpression node)
{
if (SourceParameters.Contains(node)) return ReplaceNode(node);
return SourceParameters.FirstOrDefault(p => p.Name == node.Name) ?? node;
}
}
Your WhereIn method then becomes:
public static IQueryable<TEntity> WhereIn<TEntity>(
this ObjectQuery<TEntity> query,
IEnumerable<Expression<Func<TEntity, bool>>> predicates)
{
if (predicates == null) throw new ArgumentNullException("predicates");
var predicate = BuildOr(predicates);
return query.Where(predicate);
}
I am trying to order an IQueryable of entities by date from a passed in Expression< Func< T, object>> and am getting the error: "Unable to cast the type 'System.Nullable`1' to type 'System.Object'. LINQ to Entities only supports casting Entity Data Model primitive types." The entity has a nullable datetime property on it on which I am trying to sort:
Example: (where e.Date is a nullable DateTime)
Expression<Func<T,object>> sorter = (e) => e.Date;
IOrderedQueryable<T> sortedData = data.OrderBy(sorter);
Thanks in advance!
I wrote a simple class for ordering entities based on a lambda expression at runtime.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Linq.Expressions;
using System.Reflection;
namespace DataModeling
{
public class QueryOrderer<TEntity>
where TEntity : class
{
private LambdaExpression defaultSortExpression;
private Dictionary<string, LambdaExpression> orderFieldLookup;
public QueryOrderer()
{
orderFieldLookup = new Dictionary<string, LambdaExpression>();
}
public void AddOrderMapping<TProp>(string fieldName, Expression<Func<TEntity, TProp>> selector)
{
orderFieldLookup[fieldName] = selector;
}
public void SetDefaultSortExpression<TProp>(Expression<Func<TEntity, TProp>> selector)
{
defaultSortExpression = selector;
}
public IQueryable<TEntity> GetOrderedEntries(string field, bool isDescending, IQueryable<TEntity> entries)
{
return orderEntries(entries, field, isDescending);
}
private IQueryable<TEntity> orderEntries(IQueryable<TEntity> entries, string fieldName, bool isDescending)
{
dynamic lambda = getOrderByLambda(entries, fieldName);
if (lambda == null)
{
return entries;
}
if (isDescending)
{
return Queryable.OrderByDescending(entries, lambda);
}
else
{
return Queryable.OrderBy(entries, lambda);
}
}
private dynamic getOrderByLambda(IQueryable<TEntity> entries, string fieldName)
{
if (!String.IsNullOrWhiteSpace(fieldName) && orderFieldLookup.ContainsKey(fieldName))
{
return orderFieldLookup[fieldName];
}
else
{
return defaultSortExpression;
}
}
}
}
You use this class by initially setting up all of the fields:
QueryOrderer<User> orderer = new QueryOrderer<User>();
orderer.SetDefaultSortExpression(u => u.FullName);
orderer.AddOrderMapping("UserId", u => u.UserId);
orderer.AddOrderMapping("Name", u => u.FullName);
orderer.AddOrderMapping("Email", u => u.Email);
orderer.AddOrderMapping("CreatedOn", u => u.CreatedOn);
...
var users = orderer.GetOrderedEntries("CreatedOn", isDescending: false, context.Users);
I nice feature of this code is that it handles look-up values perfectly. For instance, if you're trying to sort using the description rather than a key, you can use the outer context when building up the sort expression:
orderer.AddOrderMapping("UserType",
u => context.UserTypes
.Where(t => t.UserTypeId == u.UserTypeId)
.Select(t => t.Description)
.FirstOrDefault());
Entity Framework is smart enough to just fold the sub-query right into the outer query.
Two problem here: First you use object in your sorter, you should use DateTime. Secondly every element must have a place in the order so you have to define what should happen with elements where Date is null:
Expression<Func<T, DateTime>> sorter = (e) => e.Date ?? DateTime.MaxValue;
IOrderedQueryable<T> sortedData = data.OrderBy(sorter);
Try to reconstruct expression body
private LambdaExpression CreateLambdaPropertyGetter(Expression<Func<TEntity, object>> expression)
{
Expression body;
if (expression.Body is UnaryExpression && ((UnaryExpression)expression.Body).NodeType == ExpressionType.Convert)
body = ((UnaryExpression)expression.Body).Operand;
else
body = expression.Body;
var lambda = Expression.Lambda(body, expression.Parameters);
return lambda;
}
Try using Func delegate instead on Expression<Func>
Func<T,object> sorter = (e) => e.Date;
IOrderedEnumerable<T> sortedData = data.OrderBy(sorter);