Reusable Functions in Linq To Entites - entity-framework

I have 2 reusable functions that return lists. If the code from these functions is written directly into the linq to entities query all is good. However, separating these out into functions causes an error as it cannot be translated to a stored expression. I'm sure there must be a way of doing this though. Any ideas how to solve this problems. Ideally I want the reusable parts to be used outside of linq to entity queries also.
var activityBands = DbContext.ActivityBand
.OrderBy(x => x.ActivityBandDescription)
.Where(x => x.Active && x.ClientAccountId == clientAccountId)
.Select(x => new ActivityBandDdl
{
Name = x.ActivityBandDescription,
ActivityBandId = x.ActivityBandId,
ApplyAwr = x.ApplyAwr,
AssignmentLineTimeTypeIds = TimeTypesForActivityBand(x.DailyRate) ,
AssignmentTypeIds = AssTypesForActivityBand(x.StagePayment)
}).ToList();
public static Func<bool, List<int>> TimeTypesForActivityBand =
(dailyRate) => (new int[] { 1, 2, 3, 4 }).Where(t =>
((t != 1 && t != 2) || !dailyRate) //No Timed or NTS for daily rates
).ToList();
public static Func<bool, List<int>> AssTypesForActivityBand =
(stagePayment) => (new int[] { 2,3,4,5,6,7,8,9,10 }).Where(t =>
( t!=2 || !stagePayment) //Only stage pay ass have stage pay activity bands
).ToList();

TL;DR;
suggested solution for your problem:
get LinqKit ... have a look at it's Expand() function (in the docs: combining expressions)
https://github.com/scottksmith95/LINQKit#combining-expressions
the details:
the problem boils down to: what is the difference between the queries in both cases...
A LINQ query works with an expression tree ... in other words: just because the code you typed directly into the query and the code you typed into the static Func<...> looks the same, in fact, is the same, the resulting expression trees in both cases are not the same
what is an expression tree?
imagine a simpler query like ... someIQueryable.Where(x => x.a==1 && x.b=="foo")
the lamda that is passed to Where(...) can be seen as a straigt forward c# lambda expression that can be used as a Func
and it can also be seen as a Expression>
the later is a tree of objects that form the expression, in other words a description about the way, in that the passed in parameter can be evaluated to a bool without actually having the executable code, but just the description about what to do ... take the member a from the parameter, equality-compare it to the constant 1 ... take the boolean AND of the result with the result of: take the member b from the parameter, equality-compare it to the constant "foo" ... return the result of the boolean AND
why all of this?
it's the way LINQ works ... LINQ to entiteis takes the expression tree, looks at all the operations, finds the corresponding SQL, and builds an SQL statement which is executed in the end ...
when you have your extracted Func<...> there is a little problem ... at some point in the resulting expression tree there is something like ... take the parameter x and CALL the static Func ... the expression tree does no longer contain a description of whats happening inside the Func, but just a call to that ... as long as you want to compile that to a .net runtime executable function, it's all fun and games ... but when you try to parse it into SQL, LINQ to entiteis does not know a corresponding SQL for "call some c# function" ... therefore it tells you that this part of the expression tree can not be converted into a store expression

Related

How to run dynamic query containing subquery with EF Core?

I'm able to run all sorts of dynamically composed queries with EF Core except when containing sub-queries.
Hence, this is not a duplicate of EF Core dynamic lambda subquery not working or any other I was able to find for that matter.
A runnable repro to demonstrate the issue can be found on dotnetfiddle.net/4opEqr that uses a dynamically composed expression representing the following query:
efContext.Products.Where(p => p.Id == efContext.OrderItem.Max(i => i.ProductId)).ToList();
The exception I get is
System.InvalidOperationException : The LINQ expression 'InternalDbSet<OrderItem> { }
.Max(i => i.ProductId)' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
I observe the same behaviour with Microsoft.EntityFrameworkCore.SqlServer as with Microsoft.EntityFrameworkCore.InMemory (versions 6.0.8 and 7.0.0-preview.7.22376.2).
The problem seems to be both the way EF Core processes nested queryable expressions inside expression tree, and the way you are trying to test the dynamic expressions you create.
Shortly, here
productQueryable.Where(p => p.Id == orderItemQueryable.Max(i => i.ProductId))
productQueryable is regular variable (and .Where is regular call) and orderItemQueryable is compiler generated closure (as part of the queryable Where predicate expression), while here
() => productQueryable.Where(p => p.Id == orderItemQueryable.Max(i => i.ProductId))
they both are compiler generated closures.
So, the EF Core does not correctly process Constant expression holding nested queryable variable value inside "root" query expression, so for subqueries you should either pass directly IQueryable.Expression property value, or simulate closure (but not constant) expression. While for root queryable you should either wrap it in constant expression, or in simulated closure (but not directly).
Since wrapping the queryable variables work for both cases, the solution is to always wrap such variables in closure emulating expression. And closure emulating expression is any expression which contains constant expression holding some class instance holding the actual variable inside class property or field.
It can be implemented in several ways, for instance using System.Tuple class as holder:
static Expression MakeClosure<T>(T value)
{
var closure = new Tuple<T>(value);
return Expression.Property(Expression.Constant(closure), nameof(closure.Item1));
}
or real compiler generated closure class instance:
static Expression MakeClosure<T>(T value)
{
var closure = new { value };
return Expression.Property(Expression.Constant(closure), nameof(closure.value));
}
or the same using the body of compiler generated lambda expression containing closure:
static Expression MakeClosure<T>(T value)
{
Expression<Func<T>> lambda = () => value;
return lambda.Body;
}
Finally, in all the cases change the sample code as
var productQueryableExp = MakeClosure(productQueryable);
var orderItemQueryableExp = MakeClosure(orderItemQueryable);
and everything will work as expected. At least with EF Core. What about the other library you seem to be using (Remote.Linq), have no idea (that's I guess would be another question).

EF Core 3 GroupBy multiple columns Count Throws with extensions but linq works

Here is the one that throws full exception:
var duplicateCountOriginal = _db.TableName
.GroupBy(g => new {g.ColumnA, g.ColumnB, g.ColumnC})
.Count(g => g.Count() > 1);
Exception:
System.ArgumentException: Expression of type 'System.Func2[System.Linq.IGrouping2[Microsoft.EntityFrameworkCore.Storage.ValueBuffer,Microsoft.EntityFrameworkCore.Storage.ValueBuffer],Microsoft.EntityFrameworkCore.Storage.ValueBuffer]' cannot be used for parameter of type 'System.Func2[Microsoft.EntityFrameworkCore.Storage.ValueBuffer,Microsoft.EntityFrameworkCore.Storage.ValueBuffer]' of method 'System.Collections.Generic.IEnumerable1[Microsoft.EntityFrameworkCore.Storage.ValueBuffer] Select[ValueBuffer,ValueBuffer](System.Collections.Generic.IEnumerable1[Microsoft.EntityFrameworkCore.Storage.ValueBuffer], System.Func2[Microsoft.EntityFrameworkCore.Storage.ValueBuffer,Microsoft.EntityFrameworkCore.Storage.ValueBuffer])' (Parameter 'arg1')
But the same thing works when it is written as linq (I prefer extensions)
var duplicateCount =
from a in _db.TableName
group a by new {a.ColumnA, a.ColumnB, a.ColumnC}
into g
where g.Count() > 1
select g.Key;
duplicateCount.Count()
I am unable to understand why one works or the other doesn't.
Also if I change the first one a little bit based on EF Core 3 changes like the following
var duplicateCountOriginal = _db.TableName
.GroupBy(g => new {g.ColumnA, g.ColumnB, g.ColumnC})
.AsEnumerable()
.Count(g => g.Count() > 1);
I get the following exception:
System.InvalidOperationException: Client projection contains reference to constant expression of 'Microsoft.EntityFrameworkCore.Metadata.IPropertyBase' which is being passed as argument to method 'TryReadValue'. This could potentially cause memory leak. Consider assigning this constant to local variable and using the variable in the query instead. See https://go.microsoft.com/fwlink/?linkid=2103067 for more information.
According to me, the link given by ms has no meaning to the whatever problem here is.
Please LMK if there is any logical explanation.
There is no logical explanation. Just EF Core query translation is still far from perfect and have many defects/bugs/unhandled cases.
In this particular the problem is not the query syntax or method syntax (what you call extensions), but the lack of Select after GroupBy. If you rewrite the method syntax query similar to the one using query syntax, i.e. add .Where, .Select and then Count:
var duplicateCount = _db.TableName
.GroupBy(g => new {g.ColumnA, g.ColumnB, g.ColumnC})
.Where(g => g.Count() > 1)
.Select(g => g.Key)
.Count();
then it will be translated and executed successfully.

Passing an aggregate select expression to Dynamic Linq's GroupBy

I have simplified the following example from my code and hoping there's no obvious compilation errors because of it. Lets say I have the following entities (not what i actually have, please assume I have no EF or schema issues, this is just for example):
public class Company
{
public string GroupProperty {get;set;}
public virtual ICollection<PricingForm> PricingForms {get;set;}
}
public class PricingForm
{
public decimal Cost {get;set;}
}
And I want to query like so:
IQueryable DynamicGrouping<T>(IQueryable<T> query)
{
Expression<Func<Company, decimal?>> exp = c => c.PricingForms.Sum(fr => fr.Cost);
string selector = "new (it.Key as Key, #0(it) as Value)";
IQueryable grouping = query.GroupBy("it.GroupProperty", "it").Select(selector, exp);
return grouping;
}
I get the following error when calling the groupby/select line:
System.Linq.Dynamic.ParseException: 'Argument list incompatible with lambda expression'
What type is "it" when grouped? I have tried using other expressions that assume it is an IGrouping<string, Company>, or a IQueryable<Company>, same error. I've tried just selecting "Cost" and moving the Sum() aggregate into the selector string (i.e. Sum(#0(it)) as Value) and always seem to get the same error.
I eventually tried something along the lines of:
Expression<Func<IEnumerable<Company>, decimal?>> exp = l => l.SelectMany(c => c.PricingForms).Sum(fr => fr.Cost);
However this one, I get farther but when attempting to iterate through the results I got a different error.
The LINQ expression node type 'Invoke' is not supported in LINQ to Entities.
So, with this dynamic grouping and injecting my own select expression, what should I assume the datatype of 'it' is? Will this even work?
The type of it is IGrouping<TKey, TElement>, where TKey is dynamic based on the keySelector result type, and TElement is the element type of the input IQueryable. Luckily IGrouping<TKey, TElement> inherits (is a) IEnumerable<TElement>, so as soon as you know the input element type, you can safely base selector on IEnumerable<TElement>.
In other words, the last attempt based on Expression<Func<IEnumerable<Company>, decimal?>> is correct.
The new error you are getting is because #0(it) generates Expression.Invoke call which is not supported by EF. The easiest way to fix that is to use LINQKit Expand method:
Expression<Func<Company, decimal?>> exp = c => c.PricingForms.Sum(fr => fr.Cost);
string selector = "new (it.Key as Key, #0(it) as Value)";
IQueryable grouping = query.GroupBy("it.GroupProperty", "it").Select(selector, exp);
// This would fix the EF invocation expression error
grouping = grouping.Provider.CreateQuery(grouping.Expression.Expand());
return grouping;

How to bind parameters in replaced expression nodes in Entity Framework on the fly

I'm trying to replace a function call like (simplified) Utility.GetString(MyEntity.SomePropertyWithRelatedEntity)=="abc" with an expression visitor into something like p => p.SubRelatedEntities.FirstOrDefault(sre => sre.SomeFlag==true).SomePropertyWithRelatedEntity.
It means, the datamodel goes like:
MyEntity -> RelatedEntity -> SubRelatedEntity
I'm trying to return a string value from the SubRelatedEntity, based on some rules in the RelatedEntity, so I don't have to re-write / copy/paste the whole filtering rules in every usage; that's why I put inside a "call-signature", so my expression visitor can identify it and replace the fake-call to Utility.GetString to some complicated lambda expressions.
My expression visitor contains something like:
public override Expression Visit(Expression node)
{
if (node == null)
return null;
Expression result = null;
if (node.NodeType == ExpressionType.Call)
{
MethodCallExpression mce = node as MethodCallExpression;
if (mce.Method.DeclaringType == typeof(Utility) && mce.Method.Name == "GetString")
{
Expression<Func<RelatedEntity, string>> exp = re => re.SubRelatedEntities.FirstOrDefault(sre => sre.SomeFlag == true).SomeStringValue;
result = exp.Body;
}
else
result = base.Visit(node);
}
else
result = base.Visit(node);
return result;
}
Now, the problem is, the "sre" parameter is not bound when called the injected lambda expression. After much research, I see the lambda parameters should be replaced with another expression visitor, specifically searching for the new parameters and replacing them with the old ones. In my situation, however, I don't have an "old parameter" - I have the expression MyEntity.SomePropertyWithRelatedEntity (e.g. an property filled with the related entities) which I need to insert somehow in the generated lambda.
I hope my problem is understandable. Thank you for any insights!
After getting no answers for long time and trying hard to find a solution, I've solved it at the end :o)! It goes like this:
The newly injected lambda expression gets an ParameterExpression - well, this is a 'helper', used when directly calling the lambda, what I don't want (hence, 'parameter not bound' exception when ToEnumerable is called). So, the clue is to make a specialized ExpressionVisitor, which replaces this helper with the original expression, which is of course available in the Arguments[] for the method call, which I try to replace.
Works like a charm, like this you can reuse the same LINQ expressions, something like reusable sub-queries, instead of writing all the same LINQ stuff all time. Notice as well, that expression calling a method is not allowed in EF, in Linq2Sql it worked. Also, all the proposed web articles only replace the parameter instances, when constructing/merging more LINQ expressions together - here, I needed to replace a parameter with an faked-method-call argument, e.g. the method should not be called, it only stands for a code-marker, where I need to put my LINQ sub-query.
Hope this helps somebody, at the end it's pretty simple and logical, when one knows how the expression trees are constructed ;-).
Bye,
Andrej

Using Expression Trees as an argument constraint

Can I use an Expression Tree as an argument constraint in a FakeIteasy CallTo assertion?
Given a method on an interface with the following signature:
interface IRepository<TEntity>
{
TEntity Single(Expression<Func<TEntity, bool>> predicate);
Being called in code like so:
Flight flight = repository.Single(f => f.ID == id);
I have in mind a unit test doing something like this:
Expression<Func<Flight, bool>> myExpression = flight => flight.ID == 1;
A.CallTo(() => repository.Single(
A<Expression<Func<Flight, bool>>>.That.Matches(myExpression)))
.Returns(new Flight());
However this produces a warning: Try specifying type arguments explicitly.
I am currently having to use the Ignored property which is not ideal.
The "Matches"-method takes a lambda but you're trying to pass it the expression. What are you trying to say with the "Matches"-call? Are you matching on equality? In that case you'd just write:
A.CallTo(() => repository.Single(myExpression)).Returns(new Flight());
If you want to constrain the expression on something else you'd have to pass a predicate of the type: Func<Expression<Func<Flight, bool>>, bool> to the "Matches"-method.
Thanks Patrik,
Examining the expression was exactly what I needed to do, i.e. parse the expression (f => f.ID == id) and execute the Right side of the == to get its runtime value.
In code this looks like this:
A.CallTo(() => flightRepository.Single(A<Expression<Func<Flight, bool>>>.That
.Matches(exp => Expression.Lambda<Func<int>>(((BinaryExpression)exp.Body).Right).Compile().Invoke() == 1)))
.Returns(new Flight());
However I can't help thinking that there must be a more elegant way to achieve the same end. I'll leave that for another day though.
Thanks again,
Michael McDowell
I had the same problem while attempting to assert an expression as an argument but I was using Moq. The solution should work for you though as well...
I give most of the credit to this answer to a similar question:
Moq Expect On IRepository Passing Expression
It basically says you can do a ToString() on the expressions and compare them. It is kind of hacky but it only has one downside; the variables names in the lambda expression must match.
Here is an example...
[Test]
public void TestWhichComparesExpressions()
{
// setup
_mockRepository.Setup(x => x.GetByFilter(MatchQuery())).Returns(new List<Record>());
// execute
var records = _service.GetRecordsByFilter();
// assert
Assert.IsNotNull(records);
Assert.AreEqual(0, records.Count());
}
private static Expression<Func<DomainRecord, bool>> MatchQuery()
{
return MatchExpression(ServiceClass.QueryForTheRecords); // constant
}
// https://stackoverflow.com/questions/288413/moq-expect-on-irepository-passing-expression/1120836#1120836
private static Expression<Func<DomainRecord, bool>> MatchExpression(Expression<Func<DomainRecord, bool>> expression)
{
return It.Is<Expression<Func<DomainRecord, bool>>>(e => e.ToString() == expression.ToString());
}
I decided to put the expression into a constant on the class which used it which guaranteed it would be the same in the test if someone changed the lambda expressions's variable names.