best practice on conditional queries with linq to entities - entity-framework

I often find myself writing querys like this:
var voyages = db.VoyageRequests.Include("Carrier")
.Where(u => (fromDate.HasValue ? u.date >= fromDate.Value : true) &&
(toDate.HasValue ? u.date <= toDate.Value : true) &&
u.Carrier != null &&
u.status == (int)VoyageStatus.State.InProgress)
.OrderBy(u => u.date);
return voyages;
With conditionals inside the where statement:
fromDate.HasValue ? u.date >= fromDate.Value : true
I know the other way to do it'll be like:
var voyages = db.VoyageRequests.Include("Carrier").Where(u => u.Carrier != null &&
u.status == (int)VoyageStatus.State.InProgress);
if (fromDate.HasValue)
{
voyages = voyages.Where(u => u.date >= fromDate.Value);
}
if (toDate.HasValue)
{
voyages = voyages.Where(u => u.date <= toDate.Value);
}
return voyages.OrderBy(u => u.date);
Is there any real difference that may affect performance when this 2 approaches get transform to SQL expression?

The second query will create the simpler SQL because the evaluation of fromDate.HasValue and toDate.HasValue happens on the client. In the first query the ternary operators get evaluated on the database server as part of the SQL query. Both fromDate and toDate will be transmitted as constants to the server while in the second query only then if .HasValue is true.
We are talking about a few bytes more in length of the SQL statement and I don't believe that the server-side evaluation of the ternaries has any significant effect on query performance.
I would choose what you find more readable. Personally I would decide for the second query.

If you want the simple SQL and the more readable C# you can create an Extension as suggested by Viktor Mitev http://mentormate.com/blog/improving-linq-to-entities-queries/
public static class WhereExtensions
{
// Where extension for filters of any nullable type
public static IQueryable<TSource> Where<Tsource, TFilter>
(
this IQueryable <TSource> source,
Nullable <TFilter> filter,
Expression<Func<TSource, bool>> predicate
) where TFilter : struct
{
if (filter.HasValue)
{
source = source.Where(predicate);
}
return source;
}
// Where extension for string filters
public static IQueryable<TSource> Where<TSource>
(
this IQueryable<TSource> source,
string filter,
Expression<Func<TSource, bool>> predicate
)
{
if (!string.IsNullOrWhiteSpace(filter))
{
source = source.Where(predicate);
}
return source;
}
// Where extension for collection filters
public static IQueryable<TSource> Where<TSource, TFilter>
(
this IQueryable<TSource> source,
IEnumerable<TFilter> filter,
Expression<Func<TSource, bool>> predicate
)
{
if (filter != null && filter.Any())
{
source = source.Where(predicate);
}
return source;
}
Then your "secound query" will look like this:
var voyages = db.VoyageRequests.Include("Carrier")
.Where(u => u.Carrier != null && u.status == (int)VoyageStatus.State.InProgress)
.Where(u => u.date >= fromDate)
.Where(u => u.date <= toDate)
.OrderBy(u => u.date);
I don't know if it is more readable or if it will be confusing for some developers because it is more difficult to read directly form the code what part of the filtering is in use.
Maybe it will be more readable if you name the extensions function something like WhereIfFilterNotNull (or something meaningful :-)
var voyages = db.VoyageRequests.Include("Carrier")
.Where(u => u.Carrier != null && u.status == (int)VoyageStatus.State.InProgress)
.WhereIfFilterNotNull(u => u.date >= fromDate)
.WhereIfFilterNotNull(u => u.date <= toDate)
.OrderBy(u => u.date);

Related

Why is generated query from expression using case instead of where statements?

I am working on a tool, that translates filter strings into linq/efcore expressions. Say following filter string is given:
{
'condition': 'OR',
'rules': [
{ 'column': 'Foo', 'value': 'Bar' },
{ 'column': 'Foo', 'value': 'Baz' },
]
}
Using an efcore extension I can filter a model like so:
_dbContext.MyModel
.Filter(filterString)
.ToList();
The issue is, that the generated query uses case instead of where statements:
SELECT
*
FROM
[MyModel] AS [c]
WHERE
(
CASE
WHEN [c].[Foo] = "Bar" THEN CAST(1 AS bit)
ELSE CAST(0 AS bit)
END | CASE
WHEN [c].[Foo] = "Baz" THEN CAST(1 AS bit)
ELSE CAST(0 AS bit)
END
) = CAST(1 AS bit)
instead of:
SELECT
*
FROM
[MyModel] AS [c]
WHERE ([c].[Foo] = "Bar" OR [c].[Foo] = "Baz")
The last query takes much less time. Why is case used instead of where? Is it possible to instruct the parser to use where?
Extension:
public static IQueryable<T> Filter<T>(this IQueryable<T> query, string? filter)
{
if (string.IsNullOrEmpty(filter))
{
return query;
}
Expression<Func<T, bool>>? predicate;
try
{
predicate = new FilterExpressionParser().ParseExpressionOf<T>(JsonDocument.Parse(filter));
}
catch (Exception ex)
{
throw new FilterException($"Filter \"{filter}\" could not be parsed into a predicate.", ex);
}
return query.Where(predicate);
}
The extension tries to parse the filter string as a json document and create an linq expression from it. The logic for that is inside the FilterExpressionParser:
using System.Linq.Expressions;
using System.Reflection;
using System.Text.Json;
public enum FilterConditionType
{
AND,
OR
}
public class FilterExpressionParser
{
public Expression<Func<T, bool>> ParseExpressionOf<T>(JsonDocument json)
{
var param = Expression.Parameter(typeof(T));
var conditions = ParseTree<T>(json.RootElement, param)!;
if (conditions.CanReduce)
{
conditions = conditions.ReduceAndCheck();
}
return Expression.Lambda<Func<T, bool>>(conditions, param);
}
private delegate Expression Binder(Expression? left, Expression? right);
private Expression? ParseTree<T>(JsonElement condition, ParameterExpression parameterExpression)
{
Expression? left = null;
var conditionString = condition.GetProperty(nameof(condition)).GetString()?.ToUpper();
if (!Enum.TryParse(conditionString, out FilterConditionType gate))
{
throw new ArgumentOutOfRangeException(nameof(condition), $"Not expected condition type: {condition}.");
}
JsonElement rules = condition.GetProperty(nameof(rules));
Binder binder = gate == FilterConditionType.AND ? Expression.And! : Expression.Or!;
Expression? bind(Expression? left, Expression? right) => left == null ? right : binder(left, right);
foreach (var rule in rules.EnumerateArray())
{
string? column = rule.GetProperty(nameof(column)).GetString();
object? toCompare = value.GetString().GetProperty(nameof(value));
Expression property = Expression.Property(parameterExpression, column);
BinaryExpression? right = Expression.Equal(property, Expression.Constant(toCompare, property.Type))
left = bind(left, right);
}
return left;
}
}
For combining predicates you have used bitwise operators Expression.And and Expression.Or Bitwise and shift operators
In C# generated result looks like
e => (e.Some > 1) & (e.Some < 10) | (e.Some == -1)
So, EF is also trying to convert bit operations to the SQL.
Instead of them use Expression.AndAlso and Expression.OrElse which are Boolean logical operators
e => (e.Some > 1) && (e.Some < 10) || (e.Some == -1)
For analysis generated expressions, I would suggest to use ReadableExpressions.Visualizers and probably you will find mistake by yourself.

LINQ Generic GroupBy And SelectBy With Method Syntax

I'm trying to make it generic but didn't succeed. It doesnt get related User entity.
return from transactions in context.Transaction
where transactions.Date.Month == date.Month && transactions.Date.Year == date.Year
join users in context.UserProfile on transactions.UserID equals users.UserID
orderby transactions.MonthlyTotalExperiencePoint
group transactions by transactions.UserID into groupedList
select groupedList.First()).Take(50).ToList();
I've tried that:
public async Task<List<object>> GetAllGroup<TReturn, TOrderKey, TGroupKey>(Expression<Func<IGrouping<TGroupKey, TEntity>, TReturn>> selectExp,
Expression<Func<TEntity, bool>> whereExp,
Expression<Func<TEntity, TOrderKey>> orderbyExp,
bool descending,
int top,
Expression<Func<TEntity, TGroupKey>> groupByExp,
params Expression<Func<TEntity, object>>[] includeExps)
{
var query = DbSet.Where(whereExp);
query = !descending ? query.OrderBy(orderbyExp) : query.OrderByDescending(orderbyExp);
if (includeExps != null)
query = includeExps.Aggregate(query, (current, exp) => current.Include(exp));
return await query.GroupBy(groupByExp).Select(selectExp).Take(top).ToListAsync();
}
It gives error at usage:
var item = await transactionRepository.GetAllGroup(x => x.FirstOrDefault(), x => x.Date != null, x => x.MonthlyTotalExperiencePoint, true, 50, x => x.MonthlyTotalExperiencePoint, x => x.User);

Can I convert this extension method to use IDbSet<> instead of DbContext?

Currently I use the below method like this: db.GetProjectsAllowed(profileId, profOrgList, projectList). I would like to convert this to use IDbSet<Project>, but I'm not sure how to get the second LINQ query.
public static IQueryable<Project> GetProjectsAllowed
(
this IMkpContext db,
Guid profileId,
List<Guid> profOrgIds = null,
List<Guid> projectIds = null
)
{
var projects =
(
from p in db.Project
.Include(p => p.Proposals)
.Include(p => p.RoleAssignments)
.Include("RoleAssignments.AssigneeSnapshot")
where p.IsActive
select p);
if (profOrgIds != null && profOrgIds.Any())
{
var profileIds = db.ProfileOrganization
.Where(po => po.IsActive && profOrgIds.Contains(po.OrganizationId))
.Select(po => po.ProfileId);
projects = projects.Where(p => profileIds.Contains(p.CreatedById));
}
if (projectIds != null && projectIds.Any())
projects = projects.Where(proj => projectIds.Contains(proj.ProjectId));
return projects;//.ToList();
}
Can I convert this to use IDbSet<Project> or not?
Here, why not split this into two extension methods? This makes your GetProjectsAllowed extension method more cohesive and single responsible.
First:
public static IEnumerable<Guid> GetProfileIds(
this IDbSet<ProfileOrganization> profileOrganizations,
IEnumerable<Guid> profOrgIds = null)
{
return profOrgIds == null ? null :
from po in profileOrganizations
where po.IsActive
where profOrgIds.Contains(po.OrganizationId)
select po.OrganizationId;
}
And second:
public static IQueryable<Project> GetProjectsAllowed(
this IDbSet<Project> projects,
IEnumerable<Guid> profileIds,
IEnumerable<Guid> projectIds = null)
{
var activeProjects =
from project in projects
//.Include(..
where project.IsActive
select project;
if (profileIds != null && profileIds.Any())
{
activeProjects = activeProjects.Where(p => profileIds.Contains(p.CreatedById));
}
if (projectIds != null && projectIds.Any())
{
activeProjects = activeProjects.Where(proj => projectIds.Contains(proj.ProjectId));
}
return activeProjects;//.ToList();
}
And then the consumer can call it like this:
var profileIds = db.ProfileOrganization.GetProfileIds(profOrgIds);
var projectsAllowed = db.Projects.GetProjectsAllowed(profileIds, projectIds);

entity framework: conditional filter

Let's say I have Customers table and I want to filter it by the following:
Country: All, US, UK, Canada
Income: All, low, high, medium
Age:All, teenager, adult, senior
if I had to build an SQL string for this filter, it would be something like this:
if (Country != "All") sql += "country = " + Country
if (Income != "All") sql += "and income = " + Income
if (Age != "All") sql += "and age = " + Age;
So, basically, the user can filter by some, but not necessary all fields.
How do you do this using Entity Framework ?
Thanks !
LINQ to Entity queries return IQueryable's, so you can build your query this way:
IQueryable<Person> query = context.People;
if (Country != "All")
{
query = query.Where(p => p.Country == Country);
}
if (Income != "All")
{
query = query.Where(p => p.Income == Income);
}
if (Age != "All")
{
query = query.Where(p => p.Age == Age);
}
List<Person> fetchedPeople = query.ToList();
This case is almost too simple, but this is very helpful in more complex situations when you need to add filtering dynamically.
You can include conditional parameter this way:
return Customers.Where(
customer =>
customer.Name == Name &&
(Age == "All" || customer.Age == Age) &&
(Income == "All" || customer.Income == Income) &&
(Country == "All" || customer.Country == Country)
).ToList();
If some condition is true (e.g. country is equal to All), then all parameter condition becomes true, and this parameter does not filter result.
You can use extension method to help readable and maintainable code.
LinqExtensions.cs
public static class LinqExtensions
{
public static IQueryable<TSource> WhereIf<TSource>(this IQueryable<TSource> source, bool condition, Expression<Func<TSource, bool>> predicate)
{
return condition ? source.Where(predicate) : source;
}
}
Refactoring code
List<Person> peoples = context.People
.WhereIf(Country != "All", p => p.Country == Country)
.WhereIf(Income != "All", p => p.Income == Income)
.WhereIf(Age != "All", p => p.Age == Age)
.ToList();

conditional where in linq query

I want to return the total sum from a linq query, I pass in 2 parameters that may/may not be included in the query.
OrgId - int
reportType - int
So two questions:
How can I update the query below so that if OrgId = 0 then ignore the organisation field(Return All)?
LocumClaimTF can be True/False/Both, if both then ignore the where query for this field.
Heres what I have done so far, this is working but I'd like something for efficient.
// Using reportType set preferences for LocumClaimTF
bool locumClaimTF1, locumClaimTF2 = false;
if (reportType == 0)
{
locumClaimTF1 = false;
locumClaimTF2 = false;
}
else if (reportType == 1)
{
locumClaimTF1 = true;
locumClaimTF2 = true;
}
else // 2
{
locumClaimTF1 = true;
locumClaimTF2 = false;
}
if (OrgID != 0) // Get by OrgID
{
return _UoW.ShiftDates.Get(x => x.shiftStartDate >= StartDate && x.shiftEndDate <= EndDate)
.Where(x => x.Shift.LocumClaimTF == locumClaimTF1 || x.Shift.LocumClaimTF == locumClaimTF2)
.Where(x => x.Shift.organisationID == OrgID)
.GroupBy(s => s.assignedLocumID)
.Select(g => new dataRowDTO { dataLabel = string.Concat(g.FirstOrDefault().User.FullName), dataCount = g.Count(), dataCurrencyAmount = g.Sum(sd => sd.shiftDateTotal.Value) }
).Sum(g=>g.dataCurrencyAmount);
}
else // Ignore OrgID - Get ALL Orgs
{
return _UoW.ShiftDates.Get(x => x.shiftStartDate >= StartDate && x.shiftEndDate <= EndDate)
.Where(x => x.Shift.LocumClaimTF == locumClaimTF1 || x.Shift.LocumClaimTF == locumClaimTF2)
.GroupBy(s => s.assignedLocumID)
.Select(g => new dataRowDTO { dataLabel = string.Concat(g.FirstOrDefault().User.FullName), dataCount = g.Count(), dataCurrencyAmount = g.Sum(sd => sd.shiftDateTotal.Value) }
).Sum(g => g.dataCurrencyAmount);
}
I'm using EF with unit of work pattern to get data frm
A few things come to mind, from top to bottom:
For handling the Booleans
bool locumClaimTF1 = (reportType == 1 || reportType == 2);
bool locumClaimTF2 = (reportType == 1);
From what I read in the query though, if the report type is 1 or 2, you want the Shift's LocumClaimTF flag to have to be True. If that is the case, then you can forget the Boolean flags and just use the reportType in your condition.
Next, for composing the query, you can conditionally compose where clauses. This is a nice thing about the fluent Linq syntax. However, let's start temporarily with a regular DbContext rather than the UoW because that will introduce some complexities and questions that you'll need to look over. (I will cover that below)
using (var context = new ApplicationDbContext()) // <- insert your DbContext here...
{
var query = context.ShiftDates
.Where(x => x.shiftStartDate >= StartDate
&& x.shiftEndDate <= EndDate);
if (reportType == 1 || reportType == 2)
query = query.Where(x.Shift.LocumClaimTF);
if (OrgId > 0)
query = query.Where(x => x.Shift.organisationID == OrgID);
var total = query.GroupBy(s => s.assignedLocumID)
.Select(g => new dataRowDTO
{
dataLabel = tring.Concat(g.FirstOrDefault().User.FullName),
dataCount = g.Count(),
dataCurrencyAmount = g.Sum(sd => sd.shiftDateTotal.Value)
})
.Sum(g=>g.dataCurrencyAmount);
}
Now this here didn't make any sense. Why are you going through the trouble of grouping, counting, and summing data, just to sum the resulting sums? I suspect you've copied an existing query that was selecting a DTO for the grouped results. If you don't need the grouped results, you just want the total. So in that case, do away with the grouping and just take the sum of all applicable records:
var total = query.Sum(x => x.shiftDateTotal.Value);
So the whole thing would look something like:
using (var context = new ApplicationDbContext()) // <- insert your DbContext here...
{
var query = context.ShiftDates
.Where(x => x.shiftStartDate >= StartDate
&& x.shiftEndDate <= EndDate);
if (reportType == 1 || reportType == 2)
query = query.Where(x.Shift.LocumClaimTF);
if (OrgId > 0)
query = query.Where(x => x.Shift.organisationID == OrgID);
var total = query.Sum(x => x.shiftDateTotal.Value);
return total;
}
Back to the Unit of Work: The main consideration when using this pattern is ensuring that this Get call absolutely must return back an IQueryable<TEntity>. If it returns anything else, such as IEnumerable<TEntity> then you are going to be facing significant performance problems as it will be returning materialized lists of entities loaded to memory rather than something that you can extend to build efficient queries to the database. If the Get method does not return IQueryable, or contains methods such as ToList anywhere within it followed by AsQueryable(), then have a talk with the rest of the dev team because you're literally standing on the code/EF equivalent of a land mine. If it does return IQueryable<TEntity> (IQueryable<ShiftDate> in this case) then you can substitute it back into the above query:
var query = _UoW.ShiftDates.Get(x => x.shiftStartDate >= StartDate && x.shiftEndDate <= EndDate);
if (reportType == 1 || reportType == 2)
query = query.Where(x.Shift.LocumClaimTF);
if (OrgId > 0)
query = query.Where(x => x.Shift.organisationID == OrgID);
var total = query.Sum(x => x.shiftDateTotal.Value);
return total;