Combining Linq expressions using EFCore 3.0 - entity-framework

Question

I have a function doing a complicated Where query on my db context and then applies another transformation passed to it:
static IQueryable<T> Query<T>(Func<IQueryable<ServicesData>, IQueryable<T>> f, string path1 = null, string path2 = null, string path3 = null, string path4 = null, string path5 = null) {
try {
using (var dbc = new MyDbContext() ) {
var res = dbc.ServicesData
.Where(sd =>
(path1 == null || (path1.Contains("%") || path1.Contains("_") ? EF.Functions.Like(sd.Path1, path1) : sd.Path1 == path1))
&& (path2 == null || (path2.Contains("%") || path2.Contains("_") ? EF.Functions.Like(sd.Path2, path2) : sd.Path2 == path2))
&& (path3 == null || (path3.Contains("%") || path3.Contains("_") ? EF.Functions.Like(sd.Path3, path3) : sd.Path3 == path3))
&& (path4 == null || (path4.Contains("%") || path4.Contains("_") ? EF.Functions.Like(sd.Path4, path4) : sd.Path4 == path4))
&& (path5 == null || (path5.Contains("%") || path5.Contains("_") ? EF.Functions.Like(sd.Path5, path5) : sd.Path5 == path5)));
return f(res.ToList().AsQueryable());
//return f(res).ToList().AsQueryable();
}
} catch (Exception ex_) {
return VList<T>.Empty.AsQueryable();
}
}
This is used ie like this:
IQueryable<int> Int1InLastHour(IQueryable<ServicesData> input) {
var lastHour = DateTimeOffset.Now.AddHours(-1).ToUnixTimeMilliseconds();
return input
.Where(v => (v.Time <= lastHour) && (v.Int1 is object))
.Select(v => v.Int1.Value);
}
var lastHourProcessTime = Query(Int1InLastHour, "Publisher", "%", "ItemProcessTime").Sum();
This works, however since I call res.ToList() before calling f the linq in f is done in memory and not on the DB SQL
If I try to replace f(res.ToList().AsQueryable()) with f(res).ToList().AsQueryable() I get an exception:
{"Processing of the LINQ expression '[EntityShaperExpression][ServicesData]' by 'RelationalProjectionBindingExpressionVisitor' failed. This may indicate either a bug or a limitation in EF Core. See https://go.microsoft.com/fwlink/?linkid=2101433 for more detailed information."}
Is there any way for me to solve this ? can I somehow pass the query (Func<IQueryable<ServicesData>, IQueryable<T>>) and then combine it to the query in Query before excecuting it on the dbc ?

Answer 1

A few issues. You can split the querying to break down your results, but the scope of your DbContext needs to be at the outermost point of the chain, not inside the inner-most:
This here:
static IQueryable<T> Query<T>(Func<IQueryable<ServicesData>, IQueryable<T>> f, string path1 = null, string path2 = null, string path3 = null, string path4 = null, string path5 = null) {
try {
using (var dbc = new MyDbContext() ) { // DbContext should not be scoped here...
var res = dbc.ServicesData
As the simplest re-factor:
static IQueryable<T> Query<T>(MyDbContext dbc, Func<IQueryable<ServicesData>, IQueryable<T>> f, string path1 = null, string path2 = null, string path3 = null, string path4 = null, string path5 = null) {
try
{
var res = dbc.ServicesData.AsQueryable();
if(path1 != null)
if(path1.Contains("%") || path1.Contains("_"))
res = res.Where(EF.Functions.Like(sd.Path1, path1));
else
res = res.Where(sd.Path1 == path1);
// Repeat for Path 2 - 5 ....
return f(res);
}
catch (Exception ex_)
{
return VList<T>.Empty.AsQueryable();
}
}
Firstly, we pass in the DbContext. If the context is scoped here, the list must be materialized before being returned. The goal is to allow callers to further reduce the expression before executing the list. This means the DbContext needs to be scoped outside of this initial generation and passed in. With IoC containers managing lifetime scope you can bypass this if the DbContext is injected and scoped to a Request or common lifetime scope.
The next improvement suggestion is to move the conditional checks for the parameters out of the Linq and into regular conditions so that the Like / Equals check will only be added if the condition was provided. This will result in simpler, faster SQL being run on the server.
So the end result would look something like:
using (var dbContext = new MyDbContext())
{
var lastHourProcessTime = Query(dbContext, Int1InLastHour, "Publisher", "%", "ItemProcessTime").Sum();
}
I sort of get where you're trying to go here, but abstracting expressions from EF is bound to lead to confusing code and still prone to limitations and bugs. IMO keeping it simpler generally leads to less issues, but give this a go and see if it gets you closer.