GroupBy Expression failed to translate - entity-framework

//Model
public class Application
{
[Key]
public int ApplicationId { get; set; }
public DateTime CreatedAt { get; set; }
public DateTime ConfirmedDate { get; set; }
public DateTime IssuedDate { get; set; }
public int? AddedByUserId { get; set; }
public virtual User AddedByUser { get; set; }
public int? UpdatedByUserId { get; set; }
public virtual User UpdatedByuser { get; set; }
public string FirstName { get; set; }
public string MiddleName { get; set; }
public string LastName { get; set; }
public string TRN { get; set; }
public string EmailAddress { get; set; }
public string Address { get; set; }
public int ParishId { get; set; }
public Parish Parish { get; set; }
public int? BranchIssuedId { get; set; }
public BranchLocation BranchIssued { get; set; }
public int? BranchReceivedId { get; set; }
public BranchLocation BranchReceived {get; set; }
}
public async Task<List<Application>> GetApplicationsByNameAsync(string name)
{
if (string.IsNullOrEmpty(name))
return null;
return await _context.Application
.AsNoTracking()
.Include(app => app.BranchIssued)
.Include(app => app.BranchReceived)
.Include(app => app.Parish)
.Where(app => app.LastName.ToLower().Contains(name.ToLower()) || app.FirstName.ToLower()
.Contains(name.ToLower()))
.GroupBy(app => new { app.TRN, app })
.Select(x => x.Key.app)
.ToListAsync()
.ConfigureAwait(false);
}
The above GroupBy expression fails to compile in VS Studio. My objective is to run a query filtering results by name containing a user given string and then it should group the results by similar TRN numbers returning a list of those applications to return to the view. I think I am really close but just cant seem to figure out this last bit of the query. Any guidance is appreciated.
Error being presented
InvalidOperationException: The LINQ expression 'DbSet<Application>
.Where(a => a.LastName.ToLower().Contains(__ToLower_0) || a.FirstName.ToLower().Contains(__ToLower_0))
.GroupBy(
source: a => new {
TRN = a.TRN,
app = a
},
keySelector: a => a)' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync()
UPDATE
Seems it is definitely due to a change in how .net core 3.x and EF core play together since recent updates. I had to change it to client evaluation by using AsEnumerable() instead of ToListAsync(). The rest of the query given by Steve py works with this method. I was unaware even after reading docs how the groupby really worked in LINQ, so that has helped me a lot. Taking the query to client side eval may have performance issues however.

The GroupBy support in EF core is a joke.
This worked perfectly on the server in EF6
var nonUniqueGroups2 = db.Transactions.GroupBy(e => new { e.AccountId, e.OpeningDate })
.Where(grp => grp.Count() > 1).ToList();
In EF core it causes an exception "Unable to translate the given 'GroupBy' pattern. Call 'AsEnumerable' before 'GroupBy' to evaluate it client-side." The message is misleading, do not call AsEnumerable because this should be handled on the server.
I have found a workaround here. An additional Select will help.
var nonUniqueGroups = db.Transactions.GroupBy(e => new { e.AccountId, e.OpeningDate })
.Select(x => new { x.Key, Count = x.Count() })
.Where(x => x.Count > 1)
.ToList();
The drawback of the workaround is that the result set does not contain the items in the groups.
There is an EF Core issue. Please vote on it so they actually fix this.

Based on this:
I want to group by TRN which is a repeating set of numbers eg.12345, in the Application table there may be many records with that same sequence and I only want the very latest row within each set of TRN sequences.
I believe this should satisfy what you are looking for:
return await _context.Application
.AsNoTracking()
.Include(app => app.BranchIssued)
.Include(app => app.BranchReceived)
.Include(app => app.Parish)
.Where(app => app.LastName.ToLower().Contains(name.ToLower()) || app.FirstName.ToLower()
.Contains(name.ToLower()))
.GroupBy(app => app.TRN)
.Select(x => x.OrderByDescending(y => y.CreatedAt).First())
.ToListAsync()
.ConfigureAwait(false);
The GroupBy expression should represent what you want to group by. In your case, the TRN. From there when we do the select, x represents each "group" which contains the Enumarable set of Applications that fall under each TRN. So we order those by the descending CreatedAt date to select the newest one using First.
Give that a shot. If it's not quite what you're after, consider adding an example set to your question and the desired output vs. what output / error this here produces.

I experience a similar issue where I find it interesting and stupid at the same time. Seems like EF team prohibits doing a WHERE before GROUP BY hence it does not work. I don't understand why you cannot do it but this seems the way it is which is forcing me to implement procedures instead of nicely build code.
LMK if you find a way.
Note: They have group by only when you first group then do where (where on the grouped elements of the complete table => does not make any sense to me)

Related

EF Framework converting to EF Core Syntax Error

The following syntax when migrated to EF Core has the following error
InvalidOperationException: The LINQ expression 'DbSet()
.Join(
inner: DbSet(),
outerKeySelector: ij => ij.ImportDefinitionId,
innerKeySelector: id => id.ImportDefinitionId,
resultSelector: (ij, id) => new {
ij = ij,
id = id
})
.Join(
inner: DbSet(),
outerKeySelector: <>h__TransparentIdentifier0 => <>h__TransparentIdentifier0.id.ImportTypeId,
innerKeySelector: it => it.ImportTypeId,
resultSelector: (<>h__TransparentIdentifier0, it) => new {
<>h__TransparentIdentifier0 = <>h__TransparentIdentifier0,
it = it
})
.GroupJoin(
inner: DbSet(),
outerKeySelector: <>h__TransparentIdentifier1 => <>h__TransparentIdentifier1.<>h__TransparentIdentifier0.ij.ImportJobId,
innerKeySelector: ijp => ijp.ImportJobId,
resultSelector: (<>h__TransparentIdentifier1, ijpGroup) => new {
<>h__TransparentIdentifier1 = <>h__TransparentIdentifier1,
ijpGroup = ijpGroup
})' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly
by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList',
or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038
for more information.
(from ij in ImportJobs
join id in ImportDefinitions
on ij.ImportDefinitionId equals id.ImportDefinitionId
join it in ImportTypes
on id.ImportTypeId equals it.ImportTypeId
join ijp in ImportJobParameters
on ij.ImportJobId equals ijp.ImportJobId into ijpGroup
where ij.JobQueuedTimeUtc >= DateTime.Now.AddDays(-30)
orderby ij.JobQueuedTimeUtc descending
select
new
{
ImportDefinition = id,
ImportType = it,
LastImportJob = ij,
LastImportJobParameters = ijpGroup
}).ToList()
My attempt to change this is as follows
(from ij in ImportJobs
join id in ImportDefinitions
on ij.ImportDefinitionId equals id.ImportDefinitionId
join it in ImportTypes
on id.ImportTypeId equals it.ImportTypeId
from ijp in ImportJobParameters.Where(ijp => ij.ImportJobId == ijp.ImportJobId).DefaultIfEmpty()
where ij.JobQueuedTimeUtc >= DateTime.Now.AddDays(-60)
orderby ij.JobQueuedTimeUtc descending
select
new
{
ImportDefinition = id,
ImportType = it,
LastImportJob = ij,
LastImportJobParameter = ijp
}).ToList()
.GroupBy(i => new { i.ImportDefinition, i.ImportType, i.LastImportJob })
.Select(i => new { i.Key.ImportDefinition, i.Key.ImportType, i.Key.LastImportJob, LastImportJobParameters = i.Select(s => s.LastImportJobParameter) })
however this results in a IEnumerable of LastImportJobParameters having 1 item of null where previously there would be 0 items. Just wondering if there is an equivalent EF Core statement otherwise I will filter out once materialised.
** Classes simplified **
public class ImportJob
{
[Key]
public int? ImportJobId { get; set; }
[Required]
public Int16? ImportDefinitionId { get; set; }
[NotMapped]
public ImportDefinition ImportDefinition { get; set; }
public DateTime? JobQueuedTimeUtc { get; set; }
[NotMapped]
public List<ImportJobParameter> ImportJobParameters { get; set; }
}
public class ImportJobParameter
{
[Key]
public int? ImportJobParameterId { get; set; }
[Required]
public int? ImportJobId { get; set; }
[Required]
public short? ImportParameterId { get; set; }
public string ParameterName { get; set; }
public string ParameterValue { get; set; }
}
public class ImportDefinition
{
[Key]
public Int16? ImportDefinitionId
{
get;
set;
}
[Required]
[StringLength(255)]
public string Name
{
get;
set;
}
public ImportType ImportType
{
get;
set;
}
[Required]
public Int16? ImportTypeId
{
get;
set;
}
}
public class ImportType
{
[Key]
public Int16? ImportTypeId
{
get; set;
}
[Required]
[StringLength(100)]
public string Name
{
get;
set;
}
}
Do not use GroupJoin for eager loading, only for LEFT JOIN. EF Core team won't to fix this limitation. Make subquery for retrieveing detail data:
var query =
from ij in ImportJobs
join id in ImportDefinitions
on ij.ImportDefinitionId equals id.ImportDefinitionId
join it in ImportTypes
on id.ImportTypeId equals it.ImportTypeId
where ij.JobQueuedTimeUtc >= DateTime.Now.AddDays(-30)
orderby ij.JobQueuedTimeUtc descending
select new
{
ImportDefinition = id,
ImportType = it,
LastImportJob = ij,
LastImportJobParameters = ImportJobParameters
.Where(ijp => ij.ImportJobId == ijp.ImportJobId)
.ToList()
};
The real and probably faster solution is to fix the entity model and eliminate joins. In fact, it looks like all you have to do is remove [NotMapped] and write :
var flattened=context.Jobs
.Where(job=>job.JobQueuedTimeUtc >= date)
.Select(job=>new {
ImportDefinition = job.ImportDefinition ,
ImportType = job.ImportDefinition.ImportType,
LastImportJob = job,
LastImportJobParameter = job.ImportJobParameters
}).ToList()
What the original query does is a GroupJoin, a client-side operation with no equivalent in SQL. EF executes a LEFT JOIN and then regroups the right-hand rows in memory to reconstruct the Parameters collection. This is an expensive client-side operation that can load far more into memory than programmers realize, especially if they try to filter the right hand objects. EF Core doesn't support this
GroupJoin doesn't translate to the server in many cases. It requires you to get all of the data from the server to do GroupJoin without a special selector (first query below). But if the selector is limiting data being selected then fetching all of the data from the server may cause performance issues (second query below). That's why EF Core doesn't translate GroupJoin.
If the right-hand was an execution log with eg 10K executions per job, executing a GroupJoin to get the last 10 would result in all logs getting loaded and sorted in memory only for 99.9% of them to get rejected.
What the second query does is emulate a GroupJoin, by executing a LEFT JOIN, then grouping the objects in memory. Since this is a LEFT JOIN, nulls are expected on the right hand.
To get the result you want you'll have to filter the parameters, and then convert them to a list or array. Otherwise, every time you try to access LastImportJobParameters the LINQ subquery would run again :
.Select(i => new {
i.Key.ImportDefinition,
i.Key.ImportType,
i.Key.LastImportJob,
LastImportJobParameters = i.Where(s.LastImportJobParameter!=null)
.Select(s => s.LastImportJobParameter)
.ToList() })

Z.EntityFramework.Extensions.EFCore BulkMerge - Non Key Indexing to Update Records on Mass

I have a situation where I require to update records on mass. However, mytable is built to be quite dynamic and it's primary key is DBGenerated. I was wondering if anyone has used EF Extensions to accomplish the update based on other fields. I have tried the below from their documentation but it doesn't map and just reinserts the lot again. I have tried to modify the options several times and not had much success.
I need to map the 'update-keys' as three other columns, keeping the original PK. Anyone able to suggest a better route whilst maintaining speed? .. I don't really want to loop and manually update each one at a time
https://bulk-operations.net/bulk-merge
await _db.Enotes.BulkMergeAsync(orderEnotes, operation =>
{
operation.AutoMapKeyExpression = prop => new { prop.Entity, prop.EnoteType, prop.KeyValue1 };
operation.InsertIfNotExists = true;
operation.InsertKeepIdentity = false;
});
Class
public class EnoteEntity
{
[Key]
[DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public Int64 EnoteID { get; set; }
public string Entity { get; set; }
public string EnoteType { get; set; }
public string KeyValue1 { get; set; }
public string KeyValue2 { get; set; }
Code removed for brevity...
You currently use the library Entity Framework Extensions but you are looking at the Bulk Operations documentation so there is some difference.
Here is the right documentation about Bulk Merge: https://entityframework-extensions.net/bulk-merge
As you will find out, you should not use AutoMapKeyExpression but ColumnPrimaryKeyExpression to specify a custom key (Online Example):
await _db.Enotes.BulkMergeAsync(orderEnotes, operation =>
{
operation.ColumnPrimaryKeyExpression = prop => new { prop.Entity, prop.EnoteType, prop.KeyValue1 };
});
In addition,
The option InsertIfNotExists is only for the BulkInsert and InsertKeepIdentity is already false by default.

Nested Tables Using a DTO

I need help getting my WebApi Controller to work.
I have a 3 table Models like this.
First Table
public class MainTable{
public int MainTableID { get; set; }
... Other Fields
public ICollection<SubTable> SubTables { get; set; }
}
Second Table
public class SubTable{
public int SubTableID { get; set; }
... Other Fields
public int MainTableID { get; set; }
[ForeignKey("MainTableID ")]
[JsonIgnore]
public virtual MainTable MainTable{ get; set; }
public ICollection<SubSubTable> SubSubTables { get; set; }
}
Third Table
public class SubSubTable{
public int SubSubTableID { get; set; }
... Other Fields
public int SubTableID { get; set; }
[ForeignKey("SubTableID")]
[JsonIgnore]
public virtual SubTable SubTable{ get; set; }
}
I need to flatten the first model because of other relationships not mentioned in this post so I am using a dto
DTO
public class TableDTO
{
public int MainTableID { get; set; }
... Other Fields (there is a lot of flattening happening here but I am going to skip it to keep this simple)
public ICollection<SubTable> SubTables { get; set; }
}
Now that I got all of that out of the way. To my question.. I am linking this all to a web api controller.
If I use the DTO and create a controller like this
Controller with DTO
public IQueryable<TableDTO> GetMainTable()
{
var mainTable = from b in db.MainTables
.Include(b => b.SubTable.Select(e => e.SubSubTable))
select new TableDTO()
{
MainTableID = b.MainTableID
eager mapping of all the fields,
SubTables = b.SubTables
};
return mainTable;
}
This works for everything except the SubSubTable which returns null. If I ditch the DTO and create a controller like this
Controller without DTO
public IQueryable<MainTable> GetMainTable()
{
return db.MainTables
.Include(c => c.SubTables)
.Include(c => c.SubTables.Select(b => b.SubSubTables));
}
This works perfect and the JSon returns everything I need, except that I lose the DTO which I desperately need for other aspects of my code. I have rewritten my code in every way I can think of but nothing works. I am pretty sure that this can be done with the DTO but I don't know what it would take to make it work, and as they say "You don't know what you don't know" so hopefully someone here knows.
In Entity Framework 6 (and lower), Include is always ignored when the query ends in a projection, because the Include path can't be applied to the end result. Stated differently, Include only works if it can be positioned at the very end of the LINQ statement. (EF-core is more versatile).
This doesn't help you, because you explicitly want to return DTOs. One way to achieve this is to do the projection after you materialize the entities into memory:
var mainTable = from b in db.MainTables
.Include(b => b.SubTable.Select(e => e.SubSubTable))
.AsEnumerable()
select new MessageDTO()
{
MainTableID = b.MainTableID ,
// eager mapping of all the fields,
SubTables = b.SubTables
};
The phrase, "eager mapping of all the fields" suggests that the projection isn't going to narrow down the SELECT clause anyway, so it won't make much of a difference.
Another way could be to load all SubSubTable objects into the context that you know will be in the MainTables you fetch from the database. EF will populate all SubTable.SubSubTables collections by relationship fixup.
If this works:
public IQueryable<MainTable> GetMainTable()
{
return db.MainTables
.Include(c => c.SubTables)
.Include(c => c.SubTables.Select(b => b.SubSubTables));
}
Then use this one and just add a Select() to the end with a ToList(). Note the IEnumerable in the return type:
public IEnumerable<MainTableDto> GetMainTable()
{
return db.MainTables
.Include(c => c.SubTables)
.Include(c => c.SubTables.Select(b => b.SubSubTables))
.Select(c=> new MainTableDto { SubTables=c.SubTables /* map your properties here */ })
.ToList();
}
Not sure about the types though (at one place you have MainTableDto, at another you mention MessageDto?).

breeze projection : error selecting non scalar navigation properties

Ho to everybody. Below you can see the partial result of the EF Power Tools reverse engineering made on the database of mine.
public partial class Articoli {
public decimal Id_Articolo { get; set; }
public string Codice { get; set; }
public virtual ICollection<Udc_Dettaglio> FK_Udc_Dettaglio_Articoli { get; set; }
}
public partial class Udc_Dettaglio {
public decimal Id_Udc { get; set; }
public decimal Id_Articolo { get; set; }
public virtual Articoli FK_Udc_Dettaglio_Articoli { get; set; }
}
public Udc_DettaglioMap() {
// Primary Key
this.HasKey(t => new { t.Id_Udc, t.Id_Articolo });
// Relationships
this.HasRequired(t => t.FK_Udc_Dettaglio_Articoli)
.WithMany(t => t.FK_Udc_Dettaglio_Articoli)
.HasForeignKey(d => d.Id_Articolo);
}
If, by using breeze, i try to perform this query
breeze.EntityQuery.from("Articoli").expand("FK_Udc_Dettaglio_Articoli")
or this one
breeze.EntityQuery.from("Articoli").select("Codice,FK_Udc_Dettaglio_Articoli")
everything works fine, but if i try with this
breeze.EntityQuery.from("Articoli").select("Codice,FK_Udc_Dettaglio_Articoli.Id_Udc")
it throw an exeption :
Unable to locate property 'Id_Udc' on type 'System.Collections.Generic.ICollection`1[AWM.Models.Udc_Dettaglio]'."
Am I missing something in EF model definition ? i guess not because expand utility is working ...
I believe you are trying to build a query that EF doesn't support (and that OData query syntax doesn't support either).
Your FK_Udc_Dettaglio_Articoli is a collection navigation property of Articoli. By trying to go after just the Id_Udc property of each element in that collection, you are actually trying to create a projection within a projection ... and EF doesn't support that as far as I know.
To verify, try to construct a LINQ query on the server side that does what you want. Something like
dbContext.Articoli.Select(
a => new {a.Codice, a.FK_Udc_Dettaglio_Articoli.Select(f => new {f.Id_Udc}));
You'll quickly discover that that query does not compile. You get a message something like "anonymous type projection initializer should be simple name or member access expansion".

Code First Entity Framework Slow When Properties are Missing

I'm using Entity Framework 4.3.1 Code First.
I have a pretty simple expression and entity model.
using (var PMCtx = new PMContext("PMEntities"))
{
var results =
PMCtx.Fetch<vwSDHOriginalMW>()
.Where(x => x.DT >= StartDate && x.DT < EndDate)
.ToList();
return results;
}
public class vwSDHOriginalMW : IEntityObject, IPMContext
{
public int Schedule { get; set; }
public DateTime DT { get; set; }
public int HE { get; set; }
public Decimal OrgMW { get; set; }
public Decimal DELIVERMW { get; set; }
public string NERCCode { get; set; }
public string NERCCodeStatus { get; set; }
public int SDHSDHID { get; set; }
}
This was taking 15 seconds every time, not just the first time. The model is mapped to a view in a Sql Server 2008 database. I output the query that EF was sending, and ran it in SSMS and it took a fraction of a second.
Why is this so slow in Entity Framework?
IEntityObject appears to be a marker interface so that the original programmer could be sure these were the only that get put into the generic.
EDIT 1
Fetch ends up going through some layer wrappers to get to the data layer where it does this:
private DbSet<TEntity> FetchSet<TEntity>()
where TEntity : class, IEntityObject
{
Type PassedType = typeof(TEntity);
if (!CheckedTypes.Any(x => x.FullName == PassedType.FullName))
if (!PassedType.GetInterfaces().Any(x => CtxInterfaces.Contains(x)))
throw new ArgumentException("Type passed is not a DbSet type of constructed context.");
else
CheckedTypes.Add(PassedType);
return privateContext.Set<TEntity>();
}
Cleaned up example of the query EF is sending
SELECT [Schedule],
[DT],
[HE],
[OrgMW],
[DELIVERMW],
[NERCCode],
[NERCCodeStatus],
[SDHSDHID],
[ScheduleDeliveryHourHistoryID]
FROM [vwSDHOriginalMW]
WHERE ([DT] >= '2/17/2013') AND ([DT] < '2/21/2013')
EDIT 2
The view in the database actually had one more column than my entity model had properties.
I added the property to the model.
public class vwSDHOriginalMW : IEntityObject, IPMContext
{
public int Schedule { get; set; }
public DateTime DT { get; set; }
public int HE { get; set; }
public Decimal OrgMW { get; set; }
public Decimal DELIVERMW { get; set; }
public string NERCCode { get; set; }
public string NERCCodeStatus { get; set; }
public int SDHSDHID { get; set; }
//missing property
public int ScheduleDeliveryHourHistoryID { get; set; }
}
After adding the property yesterday, it sped up tremendously for a while, ran in 4 seconds instead of 15. But today it's slow again, and nothing has changed.
UPDATE:
I have narrowed it down a little further. There are two methods that I can use that end up using the same FetchSet. The one that I am using returns an IQueryable instead of an IEnumerable. This seems normal, and since I am filtering afterward, most desirable. However the method that returns IQueryable takes 15 seconds while the IEnumerable takes less than a second. (I am calling ToList() on both) FetchAll turns out just to be a wrapper that calls Fetch and returns IEnumerable instead of IQueryable
public IQueryable<TEntity> Fetch<TEntity>() where TEntity : class, Common.IEntityObject
{
return privateContext.Fetch<TEntity>();
}
public IEnumerable<TEntity> FetchAll<TEntity>() where TEntity : class, Common.IEntityObject
{
return privateContext.FetchAll<TEntity>();
}
If I change
IEnumerable<vwSDHOriginalMW> results =
PMCtx.Fetch<vwSDHOriginalMW>()
.Where(x => x.DT >= StartDate && x.DT < EndDate)
.ToList();
to
IEnumerable<vwSDHOriginalMW> results =
PMCtx.Fetch<vwSDHOriginalMW>()
.ToList()
.Where(x => x.DT >= StartDate && x.DT < EndDate);
it is fast. But this isn't acceptable, because it seems like I would want my where clause to be passed to the database. In this case on a dev environment the view is only 180 rows, but it has potential to be millions, so I definitely don't want to return all my results into memory before I filter them.
After much digging and many headaches, I figured out that the view was referencing a view on a different database instance that referenced a table that was missing a non-clustered index. This caused the execution plan to get cached incorrectly. After adding the index on the other database:
USE [OTHERDATABASE]
GO
CREATE NONCLUSTERED INDEX [IX_ScheduleEnergyProfileJoin]
ON [dbo].[WTXS_ScheduleEnergyProfile] ([SEQSDR])
INCLUDE ([SEQSEPI],[StartDate],[EndDate])
GO
Then clearing the execution plan cache on the database with the view I'm using:
USE [MYDATABASE]
DBCC FREEPROCCACHE
DBCC DROPCLEANBUFFERS
The query is running quickly. So it turns out that the SQL that EF said it was using was probably not the sql that was getting sent to the database. Moral of the story is I should have gone through whatever hoops to get profiling permissions on this database instead of relying on the following to output the SQL that would actually be sent.
var sql = ((System.Data.Entity.Infrastructure.DbQuery<vwSDHOriginalMW>)results).ToString();