EF Framework converting to EF Core Syntax Error - entity-framework

Question

The following syntax when migrated to EF Core has the following error
InvalidOperationException: The LINQ expression 'DbSet()
.Join(
inner: DbSet(),
outerKeySelector: ij => ij.ImportDefinitionId,
innerKeySelector: id => id.ImportDefinitionId,
resultSelector: (ij, id) => new {
ij = ij,
id = id
})
.Join(
inner: DbSet(),
outerKeySelector: <>h__TransparentIdentifier0 => <>h__TransparentIdentifier0.id.ImportTypeId,
innerKeySelector: it => it.ImportTypeId,
resultSelector: (<>h__TransparentIdentifier0, it) => new {
<>h__TransparentIdentifier0 = <>h__TransparentIdentifier0,
it = it
})
.GroupJoin(
inner: DbSet(),
outerKeySelector: <>h__TransparentIdentifier1 => <>h__TransparentIdentifier1.<>h__TransparentIdentifier0.ij.ImportJobId,
innerKeySelector: ijp => ijp.ImportJobId,
resultSelector: (<>h__TransparentIdentifier1, ijpGroup) => new {
<>h__TransparentIdentifier1 = <>h__TransparentIdentifier1,
ijpGroup = ijpGroup
})' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly
by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList',
or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038
for more information.
(from ij in ImportJobs
join id in ImportDefinitions
on ij.ImportDefinitionId equals id.ImportDefinitionId
join it in ImportTypes
on id.ImportTypeId equals it.ImportTypeId
join ijp in ImportJobParameters
on ij.ImportJobId equals ijp.ImportJobId into ijpGroup
where ij.JobQueuedTimeUtc >= DateTime.Now.AddDays(-30)
orderby ij.JobQueuedTimeUtc descending
select
new
{
ImportDefinition = id,
ImportType = it,
LastImportJob = ij,
LastImportJobParameters = ijpGroup
}).ToList()
My attempt to change this is as follows
(from ij in ImportJobs
join id in ImportDefinitions
on ij.ImportDefinitionId equals id.ImportDefinitionId
join it in ImportTypes
on id.ImportTypeId equals it.ImportTypeId
from ijp in ImportJobParameters.Where(ijp => ij.ImportJobId == ijp.ImportJobId).DefaultIfEmpty()
where ij.JobQueuedTimeUtc >= DateTime.Now.AddDays(-60)
orderby ij.JobQueuedTimeUtc descending
select
new
{
ImportDefinition = id,
ImportType = it,
LastImportJob = ij,
LastImportJobParameter = ijp
}).ToList()
.GroupBy(i => new { i.ImportDefinition, i.ImportType, i.LastImportJob })
.Select(i => new { i.Key.ImportDefinition, i.Key.ImportType, i.Key.LastImportJob, LastImportJobParameters = i.Select(s => s.LastImportJobParameter) })
however this results in a IEnumerable of LastImportJobParameters having 1 item of null where previously there would be 0 items. Just wondering if there is an equivalent EF Core statement otherwise I will filter out once materialised.
** Classes simplified **
public class ImportJob
{
[Key]
public int? ImportJobId { get; set; }
[Required]
public Int16? ImportDefinitionId { get; set; }
[NotMapped]
public ImportDefinition ImportDefinition { get; set; }
public DateTime? JobQueuedTimeUtc { get; set; }
[NotMapped]
public List<ImportJobParameter> ImportJobParameters { get; set; }
}
public class ImportJobParameter
{
[Key]
public int? ImportJobParameterId { get; set; }
[Required]
public int? ImportJobId { get; set; }
[Required]
public short? ImportParameterId { get; set; }
public string ParameterName { get; set; }
public string ParameterValue { get; set; }
}
public class ImportDefinition
{
[Key]
public Int16? ImportDefinitionId
{
get;
set;
}
[Required]
[StringLength(255)]
public string Name
{
get;
set;
}
public ImportType ImportType
{
get;
set;
}
[Required]
public Int16? ImportTypeId
{
get;
set;
}
}
public class ImportType
{
[Key]
public Int16? ImportTypeId
{
get; set;
}
[Required]
[StringLength(100)]
public string Name
{
get;
set;
}
}

Answer 1

Do not use GroupJoin for eager loading, only for LEFT JOIN. EF Core team won't to fix this limitation. Make subquery for retrieveing detail data:
var query =
from ij in ImportJobs
join id in ImportDefinitions
on ij.ImportDefinitionId equals id.ImportDefinitionId
join it in ImportTypes
on id.ImportTypeId equals it.ImportTypeId
where ij.JobQueuedTimeUtc >= DateTime.Now.AddDays(-30)
orderby ij.JobQueuedTimeUtc descending
select new
{
ImportDefinition = id,
ImportType = it,
LastImportJob = ij,
LastImportJobParameters = ImportJobParameters
.Where(ijp => ij.ImportJobId == ijp.ImportJobId)
.ToList()
};

Answer 2

The real and probably faster solution is to fix the entity model and eliminate joins. In fact, it looks like all you have to do is remove [NotMapped] and write :
var flattened=context.Jobs
.Where(job=>job.JobQueuedTimeUtc >= date)
.Select(job=>new {
ImportDefinition = job.ImportDefinition ,
ImportType = job.ImportDefinition.ImportType,
LastImportJob = job,
LastImportJobParameter = job.ImportJobParameters
}).ToList()
What the original query does is a GroupJoin, a client-side operation with no equivalent in SQL. EF executes a LEFT JOIN and then regroups the right-hand rows in memory to reconstruct the Parameters collection. This is an expensive client-side operation that can load far more into memory than programmers realize, especially if they try to filter the right hand objects. EF Core doesn't support this
GroupJoin doesn't translate to the server in many cases. It requires you to get all of the data from the server to do GroupJoin without a special selector (first query below). But if the selector is limiting data being selected then fetching all of the data from the server may cause performance issues (second query below). That's why EF Core doesn't translate GroupJoin.
If the right-hand was an execution log with eg 10K executions per job, executing a GroupJoin to get the last 10 would result in all logs getting loaded and sorted in memory only for 99.9% of them to get rejected.
What the second query does is emulate a GroupJoin, by executing a LEFT JOIN, then grouping the objects in memory. Since this is a LEFT JOIN, nulls are expected on the right hand.
To get the result you want you'll have to filter the parameters, and then convert them to a list or array. Otherwise, every time you try to access LastImportJobParameters the LINQ subquery would run again :
.Select(i => new {
i.Key.ImportDefinition,
i.Key.ImportType,
i.Key.LastImportJob,
LastImportJobParameters = i.Where(s.LastImportJobParameter!=null)
.Select(s => s.LastImportJobParameter)
.ToList() })