How can I generate SQL with WHERE statement in EFCore Query - entity-framework

I am trying to write the EF Query with a filter and the generated SQL is not using WHERE statement on the SQL Server. It extracts all data and does the filter on the Client's Memory. I am quite worried about the performance and would like to apply the filter on the SQL Server.
The requirement is to count the number of records in an Hour time slot.
public async Task<int> GetNumberofSchedules(DateTime dt)
{
return await _context.Schedules.CountAsync(
s => s.state == 0
&& s.AppointmentDate.Value.Date == dt.Date
&& s.AppointmentTime.Value.Hours >= dt.Hour
&& s.AppointmentTime.Value.Hours < dt.AddHours(1).Hour
);
}
Sample Data
if given DateTime is 07/04/2017 08:20, it should return the number of records between 07/04/2017 08:00 and 07/04/2017 09:00
The count does return 6 which is correct but when I trace the generated SQL in SQL Profiler, it's retrieving all data.
Generated SQL in SQL Profiler
SELECT
[s].[EnrolmentRecordID], [s].[AF1], [s].[AcademicYearID], [s].[AdminName], [s].[AppForm], [s].[AppointmentDate],
[s].[AppointmentTime], [s].[BDT], [s].[BKSB], [s].[DateCreated],
[s].[DateModified], [s].[Employer], [s].[EmployerInvited],
[s].[EmployerReps], [s].[MIAPP], [s].[NI], [s].[ProposedQual],
[s].[SMT], [s].[StudentInvited], [s].[StudentName]
FROM
[dbo].[EN_Schedules] AS [s]
I would like to amend my EF code to generate WHERE statement and do the filter on the server side. How can I achieve it?
Update1:
If I remove filters for TimeSpan value, it generates the correct SQL statement as the following: So, it seems to me that I need to apply the filter differently for TimeSpan Field.
exec sp_executesql N'SELECT COUNT(*) FROM [dbo].[EN_Schedules] AS [s]
WHERE ([s].[state] = 0) AND (CONVERT(date, [s].[AppointmentDate]) =
#__dt_Date_0)',N'#__dt_Date_0 datetime2(7)',#__dt_Date_0='2017-04-07
00:00:00'
**Update2: **
By using Ivan's solution, I ended up doing like this:
var startTime = new TimeSpan(dt.Hour, 0, 0);
var endTime = new TimeSpan(dt.Hour + 1, 0, 0);
return await _context.Schedules.CountAsync(
s => s.state == 0
&& s.AppointmentDate.Value.Date == dt.Date
&& s.AppointmentTime.Value >= startTime
&& s.AppointmentTime.Value < endTime
);

It's indeed converted to client evaluation - looks like many TimeSpan operations are still not supported well by EF Core.
After a lot of trial and error, the only way currently you can make it translate to SQL is to prepare TimeSpan limits as variables and use them inside the query:
var startTime = new TimeSpan(dt.Hour, 0, 0);
var endTime = startTime + new TimeSpan(1, 0, 0);
return await _context.Schedules.CountAsync(
s => s.state == 0
&& s.AppointmentDate.Value.Date == dt.Date
&& s.AppointmentTime.Value >= startTime
&& s.AppointmentTime.Value < endTime
);

It looks like Client Evaluation is the reason.
Disable it with ConfigureWarnings call. It will give an exception if a LINQ statement cann't be translated to SQL:
public class FooContext : DbContext
{
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder
.UseSqlServer("foo_connstr")
.ConfigureWarnings(warnings => warnings.Throw(RelationalEventId.QueryClientEvaluationWarning));
}
}

Related

What is the relevant rules of Flink Window TVF and CEP SQL?

I am trying to parse Flink windowing TVF sql column level lineage, I initial a custom FlinkChainedProgram and set some Opt rules.
Mostly works fine except Window TVF SQL and CEP SQL.
for example, I get a logical plan as
insert into sink_table(f1, f2, f3, f4)
SELECT cast(window_start as String),
cast(window_start as String),
user_id,
cast(SUM(price) as Bigint)
FROM TABLE(TUMBLE(TABLE source_table, DESCRIPTOR(event_time), INTERVAL '10' MINUTES))
GROUP BY window_start, window_end, GROUPING SETS ((user_id), ());
rel#1032:FlinkLogicalCalc.LOGICAL.any.None: 0.[NONE].[NONE](input=FlinkLogicalAggregate#1030,select=CAST(window_start) AS EXPR$0, CAST(window_start) AS EXPR$1, null:BIGINT AS EXPR$2, user_id, null:VARCHAR(2147483647) CHARACTER SET "UTF-16LE" AS EXPR$4, CAST($f4) AS EXPR$5)
As we seen, the Optimized RelNode Tree contains null column so that MetadataQuery can't get origin column info.
What rules should I set in Logical Optimized phase to parse Window TVF SQL and CEP SQL? Thanks
I solved the field blood relationship method of Flink CEP SQL and added the getColumnOrigins(Match rel, RelMetadataQuery mq, int iOutputColumn) method in org.apache.calcite.rel.metadata.org.apache.calcite.rel.metadata. RelMdColumnOrigins:
/**
* Support field blood relationship of CEP.
* The first column is the field after PARTITION BY, and the other columns come from the measures in Match
*/
public Set<RelColumnOrigin> getColumnOrigins(Match rel, RelMetadataQuery mq, int iOutputColumn) {
if (iOutputColumn == 0) {
return mq.getColumnOrigins(rel.getInput(), iOutputColumn);
}
final RelNode input = rel.getInput();
RexNode rexNode = rel.getMeasures().values().asList().get(iOutputColumn - 1);
RexPatternFieldRef rexPatternFieldRef = searchRexPatternFieldRef(rexNode);
if (rexPatternFieldRef != null) {
return mq.getColumnOrigins(input, rexPatternFieldRef.getIndex());
}
return null;
}
private RexPatternFieldRef searchRexPatternFieldRef(RexNode rexNode) {
if (rexNode instanceof RexCall) {
RexNode operand = ((RexCall) rexNode).getOperands().get(0);
if (operand instanceof RexPatternFieldRef) {
return (RexPatternFieldRef) operand;
} else {
// recursive search
return searchRexPatternFieldRef(operand);
}
}
return null;
}
Source address: https://github.com/HamaWhiteGG/flink-sql-lineage/blob/main/src/main/java/org/apache/calcite/rel/metadata/RelMdColumnOrigins.java
I have given detailed test cases, you can refer to: https://github.com/HamaWhiteGG/flink-sql-lineage/blob/main/src/test/java/com/dtwave/flink/lineage/cep/CepTest.java
Flink CEP SQL test case:

How to Compare two dates in LINQ in Entity Framework in .NET 5

using DbFunctions = System.Data.Entity.DbFunctions;
Using the above namespace I tried below ways but nothing worked.
This code is throwing an exception which states...
public async Task<int> SomeFunction(){
var count = await _context.Drives.CountAsync(c => DbFunctions.TruncateTime(c.CreatedOn) == DateTime.Today);
var count1 = await _context.Drives.Where(c => DbFunctions.TruncateTime(c.CreatedOn) == DateTime.Today).CountAsync();
var data = _context.Drives.Where(c => !c.IsDeleted).ToList();
//This throw an exception
// "This function can only be invoked from LINQ to Entities."
var count2 = data.Count(x=> DbFunctions.TruncateTime(c.CreatedOn) == DateTime.Today)
}
The LINQ expression 'DbSet().Where(d => DbFunctions.TruncateTime((Nullable)d.CreatedOn) == (Nullable)DateTime.Today)' could not be translated
Can someone help me out how can I compare two dates (only date not with time) in LINQ and Entity Framework?
The problem is, that you are applying DbFunctions.TruncateTime to data.
data is of type List<Drive> on not IQueryable<Drive>, because you already called ToList().
So you Count would be evaluated in memory and not on the database.
If that is really, what you want, then you can just use it like this:
var count2 = data.Count(x=> x.CreatedOn.Day == DateTime.Today);
If you want to invoke your query on the database, then you can use
var count2 = _context.Drives.Count(x=> !x.IsDeleted && DbFunctions.TruncateTime(x.CreatedOn) == DateTime.Today);
Depending on the versions and database, you are using, solution 1 may also work on the database
var count2 = _context.Drives.Count(x=> !x.IsDeleted && c.CreatedOn.Day == DateTime.Today);
Another solution would be to just use a time range, which will definetily work on the database and in memory
var start = DateTime.Today;
var end = start.AddDays(1);
var count2InMemory = data.Count(x => x.CreatedOn >= start && x.CreatedOn < end);
var count2InDatabase = _context.Drives.Count(x=> !x.IsDeleted && x.CreatedOn >= start && x.CreatedOn < end);
A final side-note:
You should use async and await when you query the database.

Efcore 2.2- where clause runs after selection and returns false results

I have this simple query:
Expression<Func<Tips, bool>> lastTipsPredicate = x => x.Status == (int)EnumGringo.LU_Status.active;
IQueryable<Tips> lastQueryBase(DbSet<Tips> t) => t.OrderByDescending(x => x.CreateDate).Take(6);
IEnumerable<Tips> latestTips = await base.GetAllByCondition(lastTipsPredicate, lastQueryBase);
This is my base repository:
public virtual async Task<IEnumerable<TEntity>> GetAllByCondition(Expression<Func<TEntity, bool>> predicate, Func<DbSet<TEntity>, IQueryable<TEntity>> baseQuery = null)
{
IQueryable<TEntity> q = context.Set<TEntity>();
if (baseQuery != null)
{
q = baseQuery(context.Set<TEntity>());
}
return await q.Where(predicate).ToListAsync();
}
This generate this sql query(From profiler):
exec sp_executesql N'SELECT [t].[ID], [t].[CreateDate], [t].[Description], [t].[InsertedByGringo], [t].[IsRecommended], [t].[IsSubscribe], [t].[LanguageType], [t].[SeoId], [t].[Slug], [t].[Status], [t].[Title], [t].[UserID], [t].[ViewCount]
FROM (
SELECT TOP(#__p_0) [x].[ID], [x].[CreateDate], [x].[Description], [x].[InsertedByGringo], [x].[IsRecommended], [x].[IsSubscribe], [x].[LanguageType], [x].[SeoId], [x].[Slug], [x].[Status], [x].[Title], [x].[UserID], [x].[ViewCount]
FROM [Tips] AS [x]
ORDER BY [x].[CreateDate] DESC
) AS [t]
WHERE [t].[Status] = 1',N'#__p_0 int',#__p_0=6
Which returns only 5 records and not 6 as I expected, 1 record is filtered out because it has status!=1.
While this query is the correct one and returns the last 6 records:
SELECT top 6 [t].[ID], [t].[CreateDate], [t].[Description], [t].[InsertedByGringo], [t].[IsRecommended], [t].[IsSubscribe], [t].[LanguageType], [t].[SeoId], [t].[Slug], [t].[Status], [t].[Title], [t].[UserID], [t].[ViewCount]
FROM Tips as [t]
WHERE [t].[Status] = 1
ORDER BY [t].CreateDate DESC
How can I generate the second query instead of the first one?
Efcore 2.2- where clause runs after selection and returns false results
It's neither EF Core nor LINQ problem, but the way your repository method builds the LINQ query.
If you want to apply filtering (Where) first and then optionally the rest, then you should change the baseQuery func input type from DbSet<TEntity> to IQueryable<TEntity>, and the implementation as follows:
public virtual async Task<IEnumerable<TEntity>> GetAllByCondition(
Expression<Func<TEntity, bool>> predicate,
Func<IQueryable<TEntity>, IQueryable<TEntity>> baseQuery = null)
{
var q = context.Set<TEntity>()
.Where(predicate); // <-- (1)
if (baseQuery != null)
q = baseQuery(q); // <-- (2)
return await q.ToListAsync();
}

Query database and unsaved changes for validation with Entity Framework 6

I have a data layer using Entity Framework 6 Database First. One of my entities represents a time span - it has a start date and an end date.
public class Range
{
public Guid ID { get; set; }
public DateTime StartDate { get; set; }
public DateTime EndDate { get; set; }
}
I want to add validation that ensures Ranges never have overlapping dates. The application will be saving added, modified and deleted Ranges all at once. So I was playing with overriding ValidateEntity in my DbContext. Here is what I ended up with and then realized I still have a problem:
protected override DbEntityValidationResult ValidateEntity(DbEntityEntry entityEntry, IDictionary<object, object> items)
{
var result = new DbEntityValidationResult(entityEntry, new List<DbValidationError>());
if (entityEntry.Entity is Range && (entityEntry.State == EntityState.Added || entityEntry.State == EntityState.Modified))
{
Range range = entityEntry.Entity as Range;
if (Ranges.Local.Any(p => range.EndDate >= p.StartDate && range.StartDate <= p.EndDate) ||
Ranges.Any(p => range.EndDate >= p.StartDate && range.StartDate <= p.EndDate))
{
result.ValidationErrors.Add(
new System.Data.Entity.Validation.DbValidationError("EndDate", "Cannot overlap another range.")); //Could be StartDate's fault but we just need save to fail
}
}
if (result.ValidationErrors.Count > 0)
{
return result;
}
else
{
return base.ValidateEntity(entityEntry, items);
}
}
The problem is that Ranges.Local does not query the database, and Ranges, which queries the database, does not include the unsaved changes.
For example: If I have a Range starting 9/1 and ending 9/8 in the database, which I modified in memory to start 8/1 and end 8/8, and I have also added a new Range starting 9/6 ending 9/12, then Ranges.Any(p => blockOut.EndDate >= p.StartDate && blockOut.StartDate <= p.EndDate) will return true for the added Range because the currently saved Range overlaps. But it is actually valid because in memory, those dates have been changed and when it saves there will be no overlaps.
Is there any way to query the combination of Local and the database, i.e. only query the database for records that are not in Local?
EDIT:
Updated code that I think works in response to Steve Py's answer below:
//Check Local first because we may be able to invalidate without querying the DB
if (BlockOutPeriods.Local.Any(p => blockOut.ID != p.ID && blockOut.EndDate >= p.StartDate && blockOut.StartDate <= p.EndDate))
{
result.ValidationErrors.Add(
new System.Data.Entity.Validation.DbValidationError("EndDate",
"Block out cannot overlap another block out."));
}
else
{
var editedIDs = BlockOutPeriods.Local.Select(p => p.ID);
//!Contains avoids reading & mapping all records to Range model objects - better?
if (BlockOutPeriods.Any(p => blockOut.EndDate >= p.StartDate && blockOut.StartDate <= p.EndDate && !editedIDs.Contains(p.ID)))
{
result.ValidationErrors.Add(
new System.Data.Entity.Validation.DbValidationError("EndDate",
"Block out cannot overlap another block out."));
}
}
You could use a Union between the local state and data state, providing an IEqualityComparer to match the PKs so that local is used without duplicating from data set.
var result = Ranges.Local
.Where(p => range.EndDate >= p.StartDate || range.StartDate <= p.EndDate)
.Union(
Ranges.Where(p => range.EndDate >= p.StartDate || range.StartDate <= p.EndDate),
new RangeEqualityComparer())
.Any(p => p.RangeId != range.RangeId && range.EndDate >= p.StartDate && range.StartDate <= p.EndDate);
This should:
pull any ranges from Local which potentially span the desired start/end date.
union with any ranges from DB which potentially span the desired date range.
check for the existence of any other record (<> my ID) that covers the range.

The specified type member 'Date' is not supported in LINQ to Entities Exception

I got a exception while implementing the following statements.
DateTime result;
if (!DateTime.TryParse(rule.data, out result))
return jobdescriptions;
if (result < new DateTime(1754, 1, 1)) // sql can't handle dates before 1-1-1753
return jobdescriptions;
return jobdescriptions.Where(j => j.JobDeadline.Date == Convert.ToDateTime(rule.data).Date );
Exception
The specified type member 'Date' is not supported in LINQ to Entities. Only initializers, entity members, and entity navigation properties are supported.
I know what the exception means but i don't know how to get rid of it. Any help?
You can use the TruncateTime method of the EntityFunctions to achieve a correct translations of the Date property into SQL:
using System.Data.Objects; // you need this namespace for EntityFunctions
// ...
DateTime ruleData = Convert.ToDateTime(rule.data).Date;
return jobdescriptions
.Where(j => EntityFunctions.TruncateTime(j.JobDeadline) == ruleData);
Update: EntityFunctionsis deprecated in EF6, Use DbFunctions.TruncateTime
LINQ to Entities cannot translate most .NET Date methods (including the casting you used) into SQL since there is no equivalent SQL.
The solution is to use the Date methods outside the LINQ statement and then pass in a value. It looks as if Convert.ToDateTime(rule.data).Date is causing the error.
Calling Date on a DateTime property also cannot be translated to SQL, so a workaround is to compare the .Year .Month and .Day properties which can be translated to LINQ since they are only integers.
var ruleDate = Convert.ToDateTime(rule.data).Date;
return jobdescriptions.Where(j => j.Deadline.Year == ruleDate.Year
&& j.Deadline.Month == ruleDate.Month
&& j.Deadline.Day == ruleDate.Day);
For EF6 use DbFunctions.TruncateTime(mydate) instead.
"EntityFunctions.TruncateTime" or "DbFunctions.TruncateTime" in ef6 Is Working but it has some performance issue in Big Data.
I think the best way is to act like this:
DateTime ruleDate = Convert.ToDateTime(rule.data);
DateTime startDate = SearchDate.Date;
DateTime endDate = SearchDate.Date.AddDay(1);
return jobdescriptions.Where(j.Deadline >= startDate
&& j.Deadline < endDate );
it is better than using parts of the date to. because query is run faster in large data.
Need to include using System.Data.Entity;. Works well even with ProjectTo<>
var ruleDate = rule.data.Date;
return jobdescriptions.Where(j => DbFunctions.TruncateTime(j.Deadline) == ruleDate);
What it means is that LINQ to SQL doesn't know how to turn the Date property into a SQL expression. This is because the Date property of the DateTime structure has no analog in SQL.
It worked for me.
DateTime dt = DateTime.Now.Date;
var ord = db.Orders.Where
(p => p.UserID == User && p.ValidityExpiry <= dt);
Source: Asp.net Forums
I have the same problem but I work with DateTime-Ranges.
My solution is to manipulate the start-time (with any date) to 00:00:00
and the end-time to 23:59:59
So I must no more convert my DateTime to Date, rather it stays DateTime.
If you have just one DateTime, you can also set the start-time (with any date) to 00:00:00 and the end-time to 23:59:59
Then you search as if it were a time span.
var from = this.setStartTime(yourDateTime);
var to = this.setEndTime(yourDateTime);
yourFilter = yourFilter.And(f => f.YourDateTime.Value >= from && f.YourDateTime.Value <= to);
Your can do it also with DateTime-Range:
var from = this.setStartTime(yourStartDateTime);
var to = this.setEndTime(yourEndDateTime);
yourFilter = yourFilter.And(f => f.YourDateTime.Value >= from && f.YourDateTime.Value <= to);
As has been pointed out by many here, using the TruncateTime function is slow.
Easiest option if you can is to use EF Core. It can do this. If you can't then a better alternative to truncate is to not change the queried field at all, but modify the bounds. If you are doing a normal 'between' type query where the lower and upper bounds are optional, the following will do the trick.
public Expression<Func<PurchaseOrder, bool>> GetDateFilter(DateTime? StartDate, DateTime? EndDate)
{
var dtMinDate = (StartDate ?? SqlDateTime.MinValue.Value).Date;
var dtMaxDate = (EndDate == null || EndDate.Value == SqlDateTime.MaxValue.Value) ? SqlDateTime.MaxValue.Value : EndDate.Value.Date.AddDays(1);
return x => x.PoDate != null && x.PoDate.Value >= dtMinDate && x.PoDate.Value < dtMaxDate;
}
Basically, rather than trimming PoDate back to just the Date part, we increment the upper query bound and user < instead of <=