My query with Include generates sql with Inner join instead Left. My FK is nullable, so I can't explain such behavior. With nullable FK I am expect normal Left join.
Have I missed something?
Linq query:
var projectEntity = await _context.Projects
// few more includes were omitted
.Include(p => p.Invoice)
.FirstOrDefaultAsync(c => c.ProjectId == id);
Classes:
[Table("InvoicedHistory")]
public class InvoiceHistory
{
[Key]
[Column("InvoicedHistory_ID")]
public int InvoicedHistoryId { get; set; }
// few properties omitted
[Column("Project_ID")]
public int? ProjectId { get; set; }
}
public class Project
{
public int ProjectId { get; set; }
// few properties were omitted
[ForeignKey(nameof(InvoiceHistory.ProjectId))]
public virtual InvoiceHistory Invoice { get; set; }
}
Project class also use fluent api:
modelBuilder.Entity<Project>(entity =>
{
entity.ToTable("Projects");
entity.HasKey(e => e.ProjectId)
.HasName("PK_Project_Project_ID_Project");
// few statements were omitted
});
Sql which was generated: (Was hard to clean up this query. It contains several joins to include data for properties I have omitted)
SELECT [t].[Project_ID], [t].[Project_Client], [t].[Project_IrsDate], [t].[Project_Name], [t].[Client_ID], [t].[Client_Name], [t].[InvoicedHistory_ID], [t].[DateSubmitted], [t].[Project_ID0], [t0].[Debitor_ID], [t0].[Project_ID], [t0].[Debitor_ID0], [t0].[Address_Number], [t0].[Alias], [t0].[Alpha_Name], [t0].[Co], [t0].[Country_ID], [t0].[Currency_ID], [t0].[Havi_YesOrNo]
FROM (
SELECT TOP(1) [p].[Project_ID], [p].[Project_Client], [p].[Project_IrsDate], [p].[Project_Name], [c].[Client_ID], [c].[Client_Name], [i].[InvoicedHistory_ID], [i].[DateSubmitted], [i].[Project_ID] AS [Project_ID0]
FROM [Projects] AS [p]
INNER JOIN [Clients] AS [c] ON [p].[Project_Client] = [c].[Client_ID]
INNER **<<<<<<<<(expect LEFT)** JOIN [InvoicedHistory] AS [i] ON [p].[Project_ID] = [i].[InvoicedHistory_ID]
WHERE [p].[Project_ID] = 19922
) AS [t]
LEFT JOIN (
SELECT [p0].[Debitor_ID], [p0].[Project_ID], [d].[Debitor_ID] AS [Debitor_ID0], [d].[Address_Number], [d].[Alias], [d].[Alpha_Name], [d].[Co], [d].[Country_ID], [d].[Currency_ID], [d].[Havi_YesOrNo]
FROM [ProjectDebitors] AS [p0]
INNER JOIN [Debitors] AS [d] ON [p0].[Debitor_ID] = [d].[Debitor_ID]
) AS [t0] ON [t].[Project_ID] = [t0].[Project_ID]
ORDER BY [t].[Project_ID], [t].[Client_ID], [t].[InvoicedHistory_ID], [t0].[Debitor_ID], [t0].[Project_ID], [t0].[Debitor_ID0]
Look at this line -
INNER <<<<<<<<(expect LEFT)<<<<<< JOIN [InvoicedHistory] AS [i] ON [p].[Project_ID] = [i].[InvoicedHistory_ID]
Inner join makes my query return nothing, because I have no invoice info. If I manually replace it with Left join, sql query will return me all necessary data.
I think you can use Fluent API to get your desired result:
modelBuilder.Entity<Project>()
.HasOne(p => p.Invoice)
.WithOne()
.HasForeignKey(ih => ih.ProjectId);
This should change it to a left join because we didn't specify .IsRequired()
As mentioned in the following SO Answer - Equivalent for .HasOptional in Entity Framework Core 1 (EF7)
You will not find an equivalent method in EF 7. By convention, a property whose CLR type can contain null will be configured as optional. So what decide if the relationship is optional or not is if the FK property is nullable or not respectively.
and
In case of your FK property is value type like int, you should declare it as nullable (int?).
Now most likely your problem with annotations is that the following is not doing what you think it is:
[ForeignKey(nameof(InvoiceHistory.ProjectId))]
//Does not resolve to:
[ForeignKey("InvoiceHistory.ProjectId")]
//Does resolve to:
[ForeignKey("ProjectId")]
Now even if that is what you are looking for, the order of operations for the ForeignKey detection is to check the parent type then the property type.
public class InvoiceHistory
{
public int? ProjectId { get; set; }
}
public class Project
{
public int ProjectId { get; set; }
// this is pointing to Project.ProjectId
// and Project.ProjectId is not nullable
// so the join becomes an inner join
// and really only works because they both have the same name
[ForeignKey(nameof(InvoiceHistory.ProjectId))]
public virtual InvoiceHistory Invoice { get; set; }
}
If you wanted this to work as pointing to the Property Type, you need to rename the InvoiceHistory name:
public class InvoiceHistory
{
public int? ProjectFk { get; set; }
}
public class Project
{
public int ProjectId { get; set; }
// this is pointing to InvoiceHistory.ProjectFk
// because there is no Project.ProjectFk
[ForeignKey(nameof(InvoiceHistory.ProjectFk))]
public virtual InvoiceHistory Invoice { get; set; }
}
EntityFramework Data Annotations
If you wanted to see it create bad SQL you could do this:
public class InvoiceHistory
{
public int? ProjectId { get; set; }
}
public class Project
{
public int ProjectFk { get; set; }
[ForeignKey("ProjectFk")]
public virtual InvoiceHistory Invoice { get; set; }
}
EF will then create:
INNER JOIN [InvoicedHistory] AS [i] ON [p].[Project_ID] = [i].[ProjectFk]
And will cause a SqlException with the message something like Invalid column name.
Related
The following syntax when migrated to EF Core has the following error
InvalidOperationException: The LINQ expression 'DbSet()
.Join(
inner: DbSet(),
outerKeySelector: ij => ij.ImportDefinitionId,
innerKeySelector: id => id.ImportDefinitionId,
resultSelector: (ij, id) => new {
ij = ij,
id = id
})
.Join(
inner: DbSet(),
outerKeySelector: <>h__TransparentIdentifier0 => <>h__TransparentIdentifier0.id.ImportTypeId,
innerKeySelector: it => it.ImportTypeId,
resultSelector: (<>h__TransparentIdentifier0, it) => new {
<>h__TransparentIdentifier0 = <>h__TransparentIdentifier0,
it = it
})
.GroupJoin(
inner: DbSet(),
outerKeySelector: <>h__TransparentIdentifier1 => <>h__TransparentIdentifier1.<>h__TransparentIdentifier0.ij.ImportJobId,
innerKeySelector: ijp => ijp.ImportJobId,
resultSelector: (<>h__TransparentIdentifier1, ijpGroup) => new {
<>h__TransparentIdentifier1 = <>h__TransparentIdentifier1,
ijpGroup = ijpGroup
})' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly
by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList',
or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038
for more information.
(from ij in ImportJobs
join id in ImportDefinitions
on ij.ImportDefinitionId equals id.ImportDefinitionId
join it in ImportTypes
on id.ImportTypeId equals it.ImportTypeId
join ijp in ImportJobParameters
on ij.ImportJobId equals ijp.ImportJobId into ijpGroup
where ij.JobQueuedTimeUtc >= DateTime.Now.AddDays(-30)
orderby ij.JobQueuedTimeUtc descending
select
new
{
ImportDefinition = id,
ImportType = it,
LastImportJob = ij,
LastImportJobParameters = ijpGroup
}).ToList()
My attempt to change this is as follows
(from ij in ImportJobs
join id in ImportDefinitions
on ij.ImportDefinitionId equals id.ImportDefinitionId
join it in ImportTypes
on id.ImportTypeId equals it.ImportTypeId
from ijp in ImportJobParameters.Where(ijp => ij.ImportJobId == ijp.ImportJobId).DefaultIfEmpty()
where ij.JobQueuedTimeUtc >= DateTime.Now.AddDays(-60)
orderby ij.JobQueuedTimeUtc descending
select
new
{
ImportDefinition = id,
ImportType = it,
LastImportJob = ij,
LastImportJobParameter = ijp
}).ToList()
.GroupBy(i => new { i.ImportDefinition, i.ImportType, i.LastImportJob })
.Select(i => new { i.Key.ImportDefinition, i.Key.ImportType, i.Key.LastImportJob, LastImportJobParameters = i.Select(s => s.LastImportJobParameter) })
however this results in a IEnumerable of LastImportJobParameters having 1 item of null where previously there would be 0 items. Just wondering if there is an equivalent EF Core statement otherwise I will filter out once materialised.
** Classes simplified **
public class ImportJob
{
[Key]
public int? ImportJobId { get; set; }
[Required]
public Int16? ImportDefinitionId { get; set; }
[NotMapped]
public ImportDefinition ImportDefinition { get; set; }
public DateTime? JobQueuedTimeUtc { get; set; }
[NotMapped]
public List<ImportJobParameter> ImportJobParameters { get; set; }
}
public class ImportJobParameter
{
[Key]
public int? ImportJobParameterId { get; set; }
[Required]
public int? ImportJobId { get; set; }
[Required]
public short? ImportParameterId { get; set; }
public string ParameterName { get; set; }
public string ParameterValue { get; set; }
}
public class ImportDefinition
{
[Key]
public Int16? ImportDefinitionId
{
get;
set;
}
[Required]
[StringLength(255)]
public string Name
{
get;
set;
}
public ImportType ImportType
{
get;
set;
}
[Required]
public Int16? ImportTypeId
{
get;
set;
}
}
public class ImportType
{
[Key]
public Int16? ImportTypeId
{
get; set;
}
[Required]
[StringLength(100)]
public string Name
{
get;
set;
}
}
Do not use GroupJoin for eager loading, only for LEFT JOIN. EF Core team won't to fix this limitation. Make subquery for retrieveing detail data:
var query =
from ij in ImportJobs
join id in ImportDefinitions
on ij.ImportDefinitionId equals id.ImportDefinitionId
join it in ImportTypes
on id.ImportTypeId equals it.ImportTypeId
where ij.JobQueuedTimeUtc >= DateTime.Now.AddDays(-30)
orderby ij.JobQueuedTimeUtc descending
select new
{
ImportDefinition = id,
ImportType = it,
LastImportJob = ij,
LastImportJobParameters = ImportJobParameters
.Where(ijp => ij.ImportJobId == ijp.ImportJobId)
.ToList()
};
The real and probably faster solution is to fix the entity model and eliminate joins. In fact, it looks like all you have to do is remove [NotMapped] and write :
var flattened=context.Jobs
.Where(job=>job.JobQueuedTimeUtc >= date)
.Select(job=>new {
ImportDefinition = job.ImportDefinition ,
ImportType = job.ImportDefinition.ImportType,
LastImportJob = job,
LastImportJobParameter = job.ImportJobParameters
}).ToList()
What the original query does is a GroupJoin, a client-side operation with no equivalent in SQL. EF executes a LEFT JOIN and then regroups the right-hand rows in memory to reconstruct the Parameters collection. This is an expensive client-side operation that can load far more into memory than programmers realize, especially if they try to filter the right hand objects. EF Core doesn't support this
GroupJoin doesn't translate to the server in many cases. It requires you to get all of the data from the server to do GroupJoin without a special selector (first query below). But if the selector is limiting data being selected then fetching all of the data from the server may cause performance issues (second query below). That's why EF Core doesn't translate GroupJoin.
If the right-hand was an execution log with eg 10K executions per job, executing a GroupJoin to get the last 10 would result in all logs getting loaded and sorted in memory only for 99.9% of them to get rejected.
What the second query does is emulate a GroupJoin, by executing a LEFT JOIN, then grouping the objects in memory. Since this is a LEFT JOIN, nulls are expected on the right hand.
To get the result you want you'll have to filter the parameters, and then convert them to a list or array. Otherwise, every time you try to access LastImportJobParameters the LINQ subquery would run again :
.Select(i => new {
i.Key.ImportDefinition,
i.Key.ImportType,
i.Key.LastImportJob,
LastImportJobParameters = i.Where(s.LastImportJobParameter!=null)
.Select(s => s.LastImportJobParameter)
.ToList() })
UPDATE: Issue fixed in current release https://github.com/AutoMapper/AutoMapper/issues/742
Using AutoMapper 3.3, QueryableExtensions and EF6
I have a user requirement to return a Count of other users created before the current user.
I have the following
public class User
{
public int Id {get;set;}
public string Name { get; set; }
public DateTime? DateActivated {get;set;}
}
public class UserViewModel
{
public int Id {get;set;}
public string Name { get; set; }
public DateTime? DateActivated {get;set;}
public int position {get;set;}
}
public class AutoMapperConfig
{
public static void ConfigAutoMapper() {
var db = new DB();
Mapper.CreateMap<User, UserViewModel>()
.ForMember(a => a.position, opt => opt.MapFrom(src => db.Users.Where(u => u.DateActivated < src.DateActivated).Count()));
Mapper.AssertConfigurationIsValid();
}
}
and finally the actual mapping:
user = db.Users.Project().To<T>(new { db = db }).FirstOrDefault(a => a.id == id);
db is a local DbContext variable and I'm using AutoMapper parameters to insert it into the mapper (https://github.com/AutoMapper/AutoMapper/wiki/Queryable-Extensions#parameterization)
So far so good, this compiles and runs, but the result for user.position is 0
I checked with sql profiler and here is the relevant section of the generated query:
CROSS JOIN (SELECT
COUNT(1) AS [A1]
FROM [dbo].[Users] AS [Extent4]
WHERE ([Extent4].[DateActivated] < [Extent4].[DateActivated]) ) AS [GroupBy1]
Notice how it refers to Extent4.DateActivated in both sides of the comparison, which will obviously yield 0 results.
So is what i'm doing just not possible? or did I do something wrong.
(and if I could do away with the parameterization and have automapper be able to refer to the current underlying db context that would be a bonus).
Thank you
EDIT
Just to make it clear, this count will be dynamic, since there are other criteria to filter prior users that I omitted from simplified the example.
I'm trying to self reference a table in my model to get a couple of details for my User entity
I have a class that looks like:
public class User
{
[Key]
[Column("recid")]
public int Id { get; set; }
[Column("givenname")]
public string FirstName { get; set; }
[Column("sn")]
public string LastName { get; set; }
[Column("mail")]
public string Email { get; set; }
[Column("managerEmail")]
public string LineManagerEmail { get; set; }
public string LineManagerFirstName { get; set; }
public string LineManagerLastName { get; set; }
}
How can I map this so that when returning a User the LineManagerFirstName and LineManagerLastName is retrieved from the same table, joined on LineManagerEmail?
for example, I anticipate the SQL to be something like:
SELECT
user.recid,
user.givenName,
user.sn,
user.mail,
user.managerEmail,
manager.givenName as lineManagerFirstName,
manager.givenName as lineManagerLastName,
FROM user
INNER JOIN user AS manager
ON user.managerEmail = manager.mail
In my Context class, I know I'll need to override the OnModelCreating method:
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
modelBuilder.Entity<User>().
}
...but being new to EF, this is where I'm getting a bit stuck!
If you use a view, you will probably not be able to update users as a view is readonly.
You should create entities according to tables and then query according to your needs and materialize a UserExtended with a query like
var q = from
u in Users
select
new UserExtended {
Id = u.Id,
/* .... */
LineManagerFirstName = u.Manager.FirstName
/* .... */
}
or a more elaborated query, in case of self join without the PK.
var q =
from u in Users
join m in Users on u.Email equals m.Email
select
new UserExtended {
Id = u.Id,
/* .... */
LineManagerFirstName = m.FirstName
/* .... */
}
will give you the following SQL (fields name do not match as I use one of my schema)
SELECT
[Extent1].[idUtilisateur] AS [idUtilisateur],
[Extent2].[Email] AS [Email]
FROM [dbo].[tableU] AS [Extent1]
INNER JOIN [dbo].[tableU] AS [Extent2] ON
([Extent1].[Email] = [Extent2].[Email])
OR (([Extent1].[Email] IS NULL) AND ([Extent2].[Email] IS NULL))
I am using Entity Framework to map two tables together using Entity Splitting as outlined here and here.
I have found that if I execute a .ToList() on an IQueryable<SplitEntity> then the results are from an Inner Join. However, If I take that same IQueryable and execute a .Count() it will return the number of records returned by a Full Join.
Here is a unit test that fails:
[TestMethod]
public void GetCustomerListTest()
{
// arrange
List<Customer> results;
int count;
// act
using (var context = new DataContext())
{
results = context.Customers.ToList();
count = context.Customers.Count();
}
// assert
Assert.IsNotNull(results); // succeeds
Assert.IsTrue(results.Count > 0); // succeeds. Has correct records from inner join
Assert.AreEqual(count, results.Count); // This line fails. Has incorrect count from full join.
}
This strikes me as very bad. How can I get the .Count() method to return the results from an Inner Join like the .ToList()?
Update - SQL
I was wrong about the full vs inner joins.
The .ToList() results in:
SELECT
[Extent1].[CustomerNumber] AS [CustomerNumber],
-- ...etc...
[Extent2].[CustomerName] AS [CustomerName],
-- ... etc...
FROM [dbo].[CustomerTable1] AS [Extent1]
INNER JOIN [dbo].[CustomerTable2] AS [Extent2] ON [Extent1].[CustomerNumber] = [Extent2].[CustomerNumber]
The .Count() results in:
SELECT
[GroupBy1].[A1] AS [C1]
FROM ( SELECT
COUNT(1) AS [A1]
FROM [dbo].[customerTable2] AS [Extent1]
) AS [GroupBy1]
Update - DataContext and entity code
The DataContext:
public class DataContext : DbContext
{
public DataContext() { Database.SetInitializer<DataContext>(null); }
public DbSet<Customer> Customers { get; set; }
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
base.OnModelCreating(modelBuilder);
modelBuilder.Configurations.Add(new CustomerMapping());
}
}
}
The Customer Mapping (FluentAPI):
public class CustomerMapping : EntityTypeConfiguration<Customer>
{
public CustomerMapping()
{
this.Map( m => {
m.Properties( x => new { x.CustomerNumber, /*...etc...*/});
m.ToTable("CustomerTable1");
})
.Map( m => {
m.Properties( x => new { x.CustomerName, /*...etc...*/});
m.ToTable("CustomerTable2");
});
}
}
The Customer entity:
public class Customer
{
[Key]
public string CustomerNumber { get; set; }
public string CustomerName { get; set; }
}
If the database and all records in CustomerTable1 and CustomerTable2 have been created by Entity Framework and SaveChanges calls in your application code this difference must not happen and you can go straight ahead and report this as a bug.
If you are mapping to an existing database or if other applications write records into the tables and you actually expect that not every record in CustomerTable1 has a corresponding record in CustomerTable2 and vice versa then Entity Splitting is the wrong mapping of your database schema.
Apparently the difference means that you can have Customers with a CustomerNumber (etc.), but without a CustomerName (etc.) - or the other way around. The better way to model this would be a one-to-one relationship where one side is required and the other side is optional. You will need an additional entity and a navigation property for this, for example like so:
[Table("CustomerTable1")]
public class Customer
{
[Key]
public string CustomerNumber { get; set; }
// + other properties belonging to CustomerTable1
public AdditionalCustomerData AdditionalCustomerData { get; set; }
}
[Table("CustomerTable2")]
public class AdditionalCustomerData
{
[Key]
public string CustomerNumber { get; set; }
public string CustomerName { get; set; }
// + other properties belonging to CustomerTable2
}
With this Fluent API mapping:
public class CustomerMapping : EntityTypeConfiguration<Customer>
{
public CustomerMapping()
{
this.HasOptional(c => c.AdditionalCustomerData)
.WithRequired()
.WillCascadeOnDelete(true);
}
}
I am querying a local table and I get the same count for both. I believe there is a problem with your context and that's why your results are inconsistent.
screenshot of essentially the same code just querying a simple dataset.
UPDATE:
I don't know why the SQL that is generated is different. You would think that they would be the same except for simply executing Count(*) instead of returning all the rows. That is obviously why you are getting a different counts. I just can't say why the SQL is different.
Maybe Jon Skeet or other genius will see this and answer! :)
I have a pair of simple classes generating a database in Code First in EF 4.1:
public class User
{
public int UserId { get; set; }
public string UserName { get; set; }
}
public class Purchase
{
public int PurchaseId { get; set; }
public int UserId { get; set; }
public int? SalespersonUserId { get; set; }
public User User { get; set; }
public User SalespersonUser { get; set; }
}
public class NewItemsDataContext : DbContext
{
public DbSet<User> Users { get; set; }
public DbSet<Purchase> Purchases { get; set; }
}
In my program, I create and write some data to it.
class Program
{
static void Main(string[] args)
{
Database.SetInitializer<NewItemsDataContext>(new DropCreateDatabaseIfModelChanges<NewItemsDataContext>());
using (NewItemsDataContext sadc = new NewItemsDataContext())
{
sadc.Users.Add(new User());
sadc.SaveChanges();
}
using (NewItemsDataContext sadc = new NewItemsDataContext())
{
sadc.Purchases.Add(new Purchase() { UserId = 1 });
sadc.SaveChanges();
}
using (NewItemsDataContext sadc = new NewItemsDataContext())
{
var sql = sadc.Purchases.Include(p => p.User);
foreach (Purchase purchase in sql)
Console.WriteLine(purchase.User.UserId.ToString());
}
}
}
Note, when I read the Purchase records back, I get an exception of purchase.User being null -- that is, the .Include did not pull anything in for the User. Now, if I ignore the salespersonUser navigation property in my OnModelCreating (or just comment it out):
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
modelBuilder.Entity<Purchase>().Ignore(p => p.SalespersonUser);
base.OnModelCreating(modelBuilder);
}
The code works, and the User is loaded in the .Include.
When the SalespersonUser nav property is ignored, the created database looks as you would expect it. The Purchase table has a PurchaseId, UserId, and SalespersonId. But once you add the SalespersonUser nav property back in (stop ignoring it), you end up with two more keys in the table: User_UserId and SalespersonUser_UserId (as well as the original UserId and SalespersonUserId).
Also, the SQL generated definitely shows where the problem arises.
Without the nav property:
{SELECT
[Extent1].[PurchaseId] AS [PurchaseId],
[Extent1].[UserId] AS [UserId],
[Extent1].[SalespersonUserId] AS [SalespersonUserId],
[Extent2].[UserId] AS [UserId1],
[Extent2].[UserName] AS [UserName]
FROM [Purchases] AS [Extent1]
INNER JOIN [Users] AS [Extent2] ON [Extent1].[UserId] = [Extent2].[UserId]}
and with the nav property:
{SELECT
[Extent1].[PurchaseId] AS [PurchaseId],
[Extent1].[UserId] AS [UserId],
[Extent1].[SalespersonUserId] AS [SalespersonUserId],
[Extent2].[UserId] AS [UserId1],
[Extent2].[UserName] AS [UserName],
[Extent1].[SalespersonUser_UserId] AS [SalespersonUser_UserId]
FROM [Purchases] AS [Extent1]
LEFT OUTER JOIN [Users] AS [Extent2] ON [Extent1].[User_UserId] = [Extent2].[UserId]}
Notice how it pulls the UserId, but joins with the User_UserId, and a left join, no less. The SalespersonUserId isn't even referenced in a join. Inside the database after writing the record, the UserId is set, but the User_UserID is null. Thus, nothing to join, and we get null for the Included User.
It would seem to me that this is a bug in EF, but it is possible there is a design reason for it. If so, can someone clear it up for me, and perhaps describe some fluent API that may work around it? I'm kind of partial to my nav properties.
EF is having problems w/ your navigation properties - thats why it's generating the extra FK's.
Try adding this fluent mapping, so you'r explicitly mapping the FK to nav property relationship:
modelBuilder.Entity<Purchase>()
.HasOptional(p => p.SalespersonUser)
.WithMany()
.HasForeignKey(p => p.SalespersonUserId);
modelBuilder.Entity<Purchase>()
.HasRequired(p => p.User)
.WithMany()
.HasForeignKey(p => p.UserId);