I'm able to run all sorts of dynamically composed queries with EF Core except when containing sub-queries.
Hence, this is not a duplicate of EF Core dynamic lambda subquery not working or any other I was able to find for that matter.
A runnable repro to demonstrate the issue can be found on dotnetfiddle.net/4opEqr that uses a dynamically composed expression representing the following query:
efContext.Products.Where(p => p.Id == efContext.OrderItem.Max(i => i.ProductId)).ToList();
The exception I get is
System.InvalidOperationException : The LINQ expression 'InternalDbSet<OrderItem> { }
.Max(i => i.ProductId)' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
I observe the same behaviour with Microsoft.EntityFrameworkCore.SqlServer as with Microsoft.EntityFrameworkCore.InMemory (versions 6.0.8 and 7.0.0-preview.7.22376.2).
The problem seems to be both the way EF Core processes nested queryable expressions inside expression tree, and the way you are trying to test the dynamic expressions you create.
Shortly, here
productQueryable.Where(p => p.Id == orderItemQueryable.Max(i => i.ProductId))
productQueryable is regular variable (and .Where is regular call) and orderItemQueryable is compiler generated closure (as part of the queryable Where predicate expression), while here
() => productQueryable.Where(p => p.Id == orderItemQueryable.Max(i => i.ProductId))
they both are compiler generated closures.
So, the EF Core does not correctly process Constant expression holding nested queryable variable value inside "root" query expression, so for subqueries you should either pass directly IQueryable.Expression property value, or simulate closure (but not constant) expression. While for root queryable you should either wrap it in constant expression, or in simulated closure (but not directly).
Since wrapping the queryable variables work for both cases, the solution is to always wrap such variables in closure emulating expression. And closure emulating expression is any expression which contains constant expression holding some class instance holding the actual variable inside class property or field.
It can be implemented in several ways, for instance using System.Tuple class as holder:
static Expression MakeClosure<T>(T value)
{
var closure = new Tuple<T>(value);
return Expression.Property(Expression.Constant(closure), nameof(closure.Item1));
}
or real compiler generated closure class instance:
static Expression MakeClosure<T>(T value)
{
var closure = new { value };
return Expression.Property(Expression.Constant(closure), nameof(closure.value));
}
or the same using the body of compiler generated lambda expression containing closure:
static Expression MakeClosure<T>(T value)
{
Expression<Func<T>> lambda = () => value;
return lambda.Body;
}
Finally, in all the cases change the sample code as
var productQueryableExp = MakeClosure(productQueryable);
var orderItemQueryableExp = MakeClosure(orderItemQueryable);
and everything will work as expected. At least with EF Core. What about the other library you seem to be using (Remote.Linq), have no idea (that's I guess would be another question).
Related
I have 2 reusable functions that return lists. If the code from these functions is written directly into the linq to entities query all is good. However, separating these out into functions causes an error as it cannot be translated to a stored expression. I'm sure there must be a way of doing this though. Any ideas how to solve this problems. Ideally I want the reusable parts to be used outside of linq to entity queries also.
var activityBands = DbContext.ActivityBand
.OrderBy(x => x.ActivityBandDescription)
.Where(x => x.Active && x.ClientAccountId == clientAccountId)
.Select(x => new ActivityBandDdl
{
Name = x.ActivityBandDescription,
ActivityBandId = x.ActivityBandId,
ApplyAwr = x.ApplyAwr,
AssignmentLineTimeTypeIds = TimeTypesForActivityBand(x.DailyRate) ,
AssignmentTypeIds = AssTypesForActivityBand(x.StagePayment)
}).ToList();
public static Func<bool, List<int>> TimeTypesForActivityBand =
(dailyRate) => (new int[] { 1, 2, 3, 4 }).Where(t =>
((t != 1 && t != 2) || !dailyRate) //No Timed or NTS for daily rates
).ToList();
public static Func<bool, List<int>> AssTypesForActivityBand =
(stagePayment) => (new int[] { 2,3,4,5,6,7,8,9,10 }).Where(t =>
( t!=2 || !stagePayment) //Only stage pay ass have stage pay activity bands
).ToList();
TL;DR;
suggested solution for your problem:
get LinqKit ... have a look at it's Expand() function (in the docs: combining expressions)
https://github.com/scottksmith95/LINQKit#combining-expressions
the details:
the problem boils down to: what is the difference between the queries in both cases...
A LINQ query works with an expression tree ... in other words: just because the code you typed directly into the query and the code you typed into the static Func<...> looks the same, in fact, is the same, the resulting expression trees in both cases are not the same
what is an expression tree?
imagine a simpler query like ... someIQueryable.Where(x => x.a==1 && x.b=="foo")
the lamda that is passed to Where(...) can be seen as a straigt forward c# lambda expression that can be used as a Func
and it can also be seen as a Expression>
the later is a tree of objects that form the expression, in other words a description about the way, in that the passed in parameter can be evaluated to a bool without actually having the executable code, but just the description about what to do ... take the member a from the parameter, equality-compare it to the constant 1 ... take the boolean AND of the result with the result of: take the member b from the parameter, equality-compare it to the constant "foo" ... return the result of the boolean AND
why all of this?
it's the way LINQ works ... LINQ to entiteis takes the expression tree, looks at all the operations, finds the corresponding SQL, and builds an SQL statement which is executed in the end ...
when you have your extracted Func<...> there is a little problem ... at some point in the resulting expression tree there is something like ... take the parameter x and CALL the static Func ... the expression tree does no longer contain a description of whats happening inside the Func, but just a call to that ... as long as you want to compile that to a .net runtime executable function, it's all fun and games ... but when you try to parse it into SQL, LINQ to entiteis does not know a corresponding SQL for "call some c# function" ... therefore it tells you that this part of the expression tree can not be converted into a store expression
I created an extension method that lets me treat a List as DbSet for testing purposes (actually, I found this idea in another question here on stack overflow, and it's been fairly useful). Coded as follows:
public static DbSet<T> AsDbSet<T>(this List<T> sourceList) where T : class
{
var queryable = sourceList.AsQueryable();
var mockDbSet = new Mock<DbSet<T>>();
mockDbSet.As<IQueryable<T>>().Setup(m => m.Provider).Returns(queryable.Provider);
mockDbSet.As<IQueryable<T>>().Setup(m => m.Expression).Returns(queryable.Expression);
mockDbSet.As<IQueryable<T>>().Setup(m => m.ElementType).Returns(queryable.ElementType);
mockDbSet.As<IQueryable<T>>().Setup(m => m.GetEnumerator()).Returns(queryable.GetEnumerator());
mockDbSet.Setup(d => d.Add(It.IsAny<T>())).Callback<T>(sourceList.Add);
mockDbSet.Setup(d => d.Find(It.IsAny<object[]>())).Callback(sourceList.Find);
return mockDbSet.Object;
}
I had been using Add for awhile, and that works perfectly. However, when I try to add the callback for Find, I get a compiler error saying that it can't convert a method group to an action. Why is sourceList.Add an Action, but sourceList.Find is a method group?
I'll admit I'm not particularly familiar with C# delegates, so it's likely I'm missing something very obvious. Thanks in advance.
The reason Add works is because the List<T>.Add method group contains a single method which takes a single argument of type T and returns void. This method has the same signature as an Action<T> which is one of the overloads of the Callback method (the one with a single generic type parameter, Callback<T>), therefore the List<T>.Add method group can be converted to an Action<T>.
With Find, you are trying to call the Callback method (as opposed to Callback<T>) which expects an Action parameter (as opposed to Action<T>). The difference here is that an Action does not take any parameters, but an Action<T> takes a single parameter of type T. The List<T>.Find method group cannot be converted to an Action because all the Find methods (there is only one anyway) take input parameters.
The following will compile:
public static DbSet<T> AsDbSet<T>(this List<T> sourceList) where T : class
{
var mockDbSet = new Mock<DbSet<T>>();
mockDbSet.Setup(d => d.Find(It.IsAny<object[]>())).Callback<Predicate<T>>(t => sourceList.Find(t));
return mockDbSet.Object;
}
Note that I have called .Callback<Predicate<T>> because the List<T>.Find method expects and argument of type Predicate. Also note I have had to write t => sourceList.Find(t) instead of sourceList.Find because Find returns a value (which means it doesn't match the signature of Action<Predicate<T>>). By writing it as a lambda expression the return value will be thrown away.
Note that although this compiles it will not actually work because the DbSet.Find method actually takes an object[] for it's parameter, not a Predicate<T>, so you will likely have to do something like this:
public static DbSet<T> AsDbSet<T>(this List<T> sourceList) where T : class
{
var mockDbSet = new Mock<DbSet<T>>();
mockDbSet.Setup(d => d.Find(It.IsAny<object[]>())).Callback<object[]>(keyValues => sourceList.Find(keyValues.Contains));
return mockDbSet.Object;
}
This last point has more to do with how to use the Moq library that how to use method groups, delegates and lambdas - there is all sorts of syntactic sugar going on with this line which is hiding what is actually relevant to the compiler and what isn't.
I have a entity class Foo I've made partial containing the following code
private readonly static Expression<Func<Foo, int>> MyKeyExpression = (x) => x.Key;
public int MyKey
{
get { return MyKeyExpression.Compile()(this); }
}
The above works as in I can use MyKey in EntityFrameworks linq queries.
Why don't the following work?
private readonly static Expression<Func<Foo, int>> MyKeyExpression = (x) => x.Key;
// Set in the constructor with
// _myKeyDelegate = MyKeyExpression.Compile();
private readonly Func<Foo,int> _myKeyDelegate;
public int MyKey
{
get { return _myKeyDelegate(this); }
}
I understand the difference between a delegate and an expression(or maybe i don't?) but is confused how EntityFramework is able to interpret the property differently since MyKeyExpression.Compile() returns just that delegate which is then invoked returning an int. Perhaps its my lack of understanding of how the compiler actually handles C# Properties?
Example of usage where first example works but second examples throw a exception about not being able to translate it to SQL.:
dbContext.Foo.Delete(x => x.MyKey == 5)
I would say you don't fully understand difference between delegates and expressions.
Delegate is a reference to code compiled into IL. Only thing you can with it is execute it within .net CLR.
Expression object is a expression represented as tree, (you can think of AST). You can compile it to IL (Compile method) or you can inspect it and generate code for other execution environment, for example into SQL (that's what EF does).
When C# compiler compiles code, first it builds syntax tree and then compiles it. Basically expression is result of first part without second, so you could use SQL translator to compile it to SQL. Or you can write you own and translate it to anything else.
It's very strange what you are saying...
EF ignores the content of the getter and the setter of a mapped property (MyKey).
The query should be generated with a WHERE clause based on MyKey independent of what getter does.
How did you map the MyKey property? There is the setter missing so EF does not generate a field on the DB table and does not map it automatically.
I'm trying to replace a function call like (simplified) Utility.GetString(MyEntity.SomePropertyWithRelatedEntity)=="abc" with an expression visitor into something like p => p.SubRelatedEntities.FirstOrDefault(sre => sre.SomeFlag==true).SomePropertyWithRelatedEntity.
It means, the datamodel goes like:
MyEntity -> RelatedEntity -> SubRelatedEntity
I'm trying to return a string value from the SubRelatedEntity, based on some rules in the RelatedEntity, so I don't have to re-write / copy/paste the whole filtering rules in every usage; that's why I put inside a "call-signature", so my expression visitor can identify it and replace the fake-call to Utility.GetString to some complicated lambda expressions.
My expression visitor contains something like:
public override Expression Visit(Expression node)
{
if (node == null)
return null;
Expression result = null;
if (node.NodeType == ExpressionType.Call)
{
MethodCallExpression mce = node as MethodCallExpression;
if (mce.Method.DeclaringType == typeof(Utility) && mce.Method.Name == "GetString")
{
Expression<Func<RelatedEntity, string>> exp = re => re.SubRelatedEntities.FirstOrDefault(sre => sre.SomeFlag == true).SomeStringValue;
result = exp.Body;
}
else
result = base.Visit(node);
}
else
result = base.Visit(node);
return result;
}
Now, the problem is, the "sre" parameter is not bound when called the injected lambda expression. After much research, I see the lambda parameters should be replaced with another expression visitor, specifically searching for the new parameters and replacing them with the old ones. In my situation, however, I don't have an "old parameter" - I have the expression MyEntity.SomePropertyWithRelatedEntity (e.g. an property filled with the related entities) which I need to insert somehow in the generated lambda.
I hope my problem is understandable. Thank you for any insights!
After getting no answers for long time and trying hard to find a solution, I've solved it at the end :o)! It goes like this:
The newly injected lambda expression gets an ParameterExpression - well, this is a 'helper', used when directly calling the lambda, what I don't want (hence, 'parameter not bound' exception when ToEnumerable is called). So, the clue is to make a specialized ExpressionVisitor, which replaces this helper with the original expression, which is of course available in the Arguments[] for the method call, which I try to replace.
Works like a charm, like this you can reuse the same LINQ expressions, something like reusable sub-queries, instead of writing all the same LINQ stuff all time. Notice as well, that expression calling a method is not allowed in EF, in Linq2Sql it worked. Also, all the proposed web articles only replace the parameter instances, when constructing/merging more LINQ expressions together - here, I needed to replace a parameter with an faked-method-call argument, e.g. the method should not be called, it only stands for a code-marker, where I need to put my LINQ sub-query.
Hope this helps somebody, at the end it's pretty simple and logical, when one knows how the expression trees are constructed ;-).
Bye,
Andrej
I am re-writing this question to make it clearer what I need to do. I am trying to use Rhino-Mock to test:
public IQueryable<TxRxMode> GetAllModes()
{
return m_context.TxRxModes.Where(txRxMode => txRxMode.Active);
}
Here's the code:
var context = MockRepository.GenerateStub<IProjectContext>();
//Returns an empty list
context.Expect(c => c.TxRxModes.Where(Arg<Func<TxRxMode, bool>>.Is.Anything)).Return(new List<TxRxMode>().AsQueryable());
TxRxModes in an IObjectSet property on the context and I want it to return an empty IQueryable<TxRxMode> object when the return m_context.TxRxModes.Where(txRxMode => txRxMode.Active); code is called.
When I run this, the Expect method call throws the an ArgumentNullException:
Value cannot be null.
Parameter name: predicate
I have tried the simpler:
IObjectSet<TxRxMode> modes = MockRepository.GenerateStub<IObjectSet<TxRxMode>>();
context.Expect(c => c.TxRxModes).Return(modes);
but this throws a null reference exception when I call
return m_context.TxRxModes.Where(txRxMode => txRxMode.Active);
Basically, this is part of the method I am trying to mock, so the key question is how do I mock this Where statement?
Where is actually a global static method and you shouldn't be mocking it. It operates on an IEnumerable however and you could just mock that.
Its kind of a hassle doing it with rhino mocks however. I would recommend doing the mock manually (if you need to do it at all).