UPDATE: Issue fixed in current release https://github.com/AutoMapper/AutoMapper/issues/742
Using AutoMapper 3.3, QueryableExtensions and EF6
I have a user requirement to return a Count of other users created before the current user.
I have the following
public class User
{
public int Id {get;set;}
public string Name { get; set; }
public DateTime? DateActivated {get;set;}
}
public class UserViewModel
{
public int Id {get;set;}
public string Name { get; set; }
public DateTime? DateActivated {get;set;}
public int position {get;set;}
}
public class AutoMapperConfig
{
public static void ConfigAutoMapper() {
var db = new DB();
Mapper.CreateMap<User, UserViewModel>()
.ForMember(a => a.position, opt => opt.MapFrom(src => db.Users.Where(u => u.DateActivated < src.DateActivated).Count()));
Mapper.AssertConfigurationIsValid();
}
}
and finally the actual mapping:
user = db.Users.Project().To<T>(new { db = db }).FirstOrDefault(a => a.id == id);
db is a local DbContext variable and I'm using AutoMapper parameters to insert it into the mapper (https://github.com/AutoMapper/AutoMapper/wiki/Queryable-Extensions#parameterization)
So far so good, this compiles and runs, but the result for user.position is 0
I checked with sql profiler and here is the relevant section of the generated query:
CROSS JOIN (SELECT
COUNT(1) AS [A1]
FROM [dbo].[Users] AS [Extent4]
WHERE ([Extent4].[DateActivated] < [Extent4].[DateActivated]) ) AS [GroupBy1]
Notice how it refers to Extent4.DateActivated in both sides of the comparison, which will obviously yield 0 results.
So is what i'm doing just not possible? or did I do something wrong.
(and if I could do away with the parameterization and have automapper be able to refer to the current underlying db context that would be a bonus).
Thank you
EDIT
Just to make it clear, this count will be dynamic, since there are other criteria to filter prior users that I omitted from simplified the example.
Related
The following syntax when migrated to EF Core has the following error
InvalidOperationException: The LINQ expression 'DbSet()
.Join(
inner: DbSet(),
outerKeySelector: ij => ij.ImportDefinitionId,
innerKeySelector: id => id.ImportDefinitionId,
resultSelector: (ij, id) => new {
ij = ij,
id = id
})
.Join(
inner: DbSet(),
outerKeySelector: <>h__TransparentIdentifier0 => <>h__TransparentIdentifier0.id.ImportTypeId,
innerKeySelector: it => it.ImportTypeId,
resultSelector: (<>h__TransparentIdentifier0, it) => new {
<>h__TransparentIdentifier0 = <>h__TransparentIdentifier0,
it = it
})
.GroupJoin(
inner: DbSet(),
outerKeySelector: <>h__TransparentIdentifier1 => <>h__TransparentIdentifier1.<>h__TransparentIdentifier0.ij.ImportJobId,
innerKeySelector: ijp => ijp.ImportJobId,
resultSelector: (<>h__TransparentIdentifier1, ijpGroup) => new {
<>h__TransparentIdentifier1 = <>h__TransparentIdentifier1,
ijpGroup = ijpGroup
})' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly
by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList',
or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038
for more information.
(from ij in ImportJobs
join id in ImportDefinitions
on ij.ImportDefinitionId equals id.ImportDefinitionId
join it in ImportTypes
on id.ImportTypeId equals it.ImportTypeId
join ijp in ImportJobParameters
on ij.ImportJobId equals ijp.ImportJobId into ijpGroup
where ij.JobQueuedTimeUtc >= DateTime.Now.AddDays(-30)
orderby ij.JobQueuedTimeUtc descending
select
new
{
ImportDefinition = id,
ImportType = it,
LastImportJob = ij,
LastImportJobParameters = ijpGroup
}).ToList()
My attempt to change this is as follows
(from ij in ImportJobs
join id in ImportDefinitions
on ij.ImportDefinitionId equals id.ImportDefinitionId
join it in ImportTypes
on id.ImportTypeId equals it.ImportTypeId
from ijp in ImportJobParameters.Where(ijp => ij.ImportJobId == ijp.ImportJobId).DefaultIfEmpty()
where ij.JobQueuedTimeUtc >= DateTime.Now.AddDays(-60)
orderby ij.JobQueuedTimeUtc descending
select
new
{
ImportDefinition = id,
ImportType = it,
LastImportJob = ij,
LastImportJobParameter = ijp
}).ToList()
.GroupBy(i => new { i.ImportDefinition, i.ImportType, i.LastImportJob })
.Select(i => new { i.Key.ImportDefinition, i.Key.ImportType, i.Key.LastImportJob, LastImportJobParameters = i.Select(s => s.LastImportJobParameter) })
however this results in a IEnumerable of LastImportJobParameters having 1 item of null where previously there would be 0 items. Just wondering if there is an equivalent EF Core statement otherwise I will filter out once materialised.
** Classes simplified **
public class ImportJob
{
[Key]
public int? ImportJobId { get; set; }
[Required]
public Int16? ImportDefinitionId { get; set; }
[NotMapped]
public ImportDefinition ImportDefinition { get; set; }
public DateTime? JobQueuedTimeUtc { get; set; }
[NotMapped]
public List<ImportJobParameter> ImportJobParameters { get; set; }
}
public class ImportJobParameter
{
[Key]
public int? ImportJobParameterId { get; set; }
[Required]
public int? ImportJobId { get; set; }
[Required]
public short? ImportParameterId { get; set; }
public string ParameterName { get; set; }
public string ParameterValue { get; set; }
}
public class ImportDefinition
{
[Key]
public Int16? ImportDefinitionId
{
get;
set;
}
[Required]
[StringLength(255)]
public string Name
{
get;
set;
}
public ImportType ImportType
{
get;
set;
}
[Required]
public Int16? ImportTypeId
{
get;
set;
}
}
public class ImportType
{
[Key]
public Int16? ImportTypeId
{
get; set;
}
[Required]
[StringLength(100)]
public string Name
{
get;
set;
}
}
Do not use GroupJoin for eager loading, only for LEFT JOIN. EF Core team won't to fix this limitation. Make subquery for retrieveing detail data:
var query =
from ij in ImportJobs
join id in ImportDefinitions
on ij.ImportDefinitionId equals id.ImportDefinitionId
join it in ImportTypes
on id.ImportTypeId equals it.ImportTypeId
where ij.JobQueuedTimeUtc >= DateTime.Now.AddDays(-30)
orderby ij.JobQueuedTimeUtc descending
select new
{
ImportDefinition = id,
ImportType = it,
LastImportJob = ij,
LastImportJobParameters = ImportJobParameters
.Where(ijp => ij.ImportJobId == ijp.ImportJobId)
.ToList()
};
The real and probably faster solution is to fix the entity model and eliminate joins. In fact, it looks like all you have to do is remove [NotMapped] and write :
var flattened=context.Jobs
.Where(job=>job.JobQueuedTimeUtc >= date)
.Select(job=>new {
ImportDefinition = job.ImportDefinition ,
ImportType = job.ImportDefinition.ImportType,
LastImportJob = job,
LastImportJobParameter = job.ImportJobParameters
}).ToList()
What the original query does is a GroupJoin, a client-side operation with no equivalent in SQL. EF executes a LEFT JOIN and then regroups the right-hand rows in memory to reconstruct the Parameters collection. This is an expensive client-side operation that can load far more into memory than programmers realize, especially if they try to filter the right hand objects. EF Core doesn't support this
GroupJoin doesn't translate to the server in many cases. It requires you to get all of the data from the server to do GroupJoin without a special selector (first query below). But if the selector is limiting data being selected then fetching all of the data from the server may cause performance issues (second query below). That's why EF Core doesn't translate GroupJoin.
If the right-hand was an execution log with eg 10K executions per job, executing a GroupJoin to get the last 10 would result in all logs getting loaded and sorted in memory only for 99.9% of them to get rejected.
What the second query does is emulate a GroupJoin, by executing a LEFT JOIN, then grouping the objects in memory. Since this is a LEFT JOIN, nulls are expected on the right hand.
To get the result you want you'll have to filter the parameters, and then convert them to a list or array. Otherwise, every time you try to access LastImportJobParameters the LINQ subquery would run again :
.Select(i => new {
i.Key.ImportDefinition,
i.Key.ImportType,
i.Key.LastImportJob,
LastImportJobParameters = i.Where(s.LastImportJobParameter!=null)
.Select(s => s.LastImportJobParameter)
.ToList() })
My query with Include generates sql with Inner join instead Left. My FK is nullable, so I can't explain such behavior. With nullable FK I am expect normal Left join.
Have I missed something?
Linq query:
var projectEntity = await _context.Projects
// few more includes were omitted
.Include(p => p.Invoice)
.FirstOrDefaultAsync(c => c.ProjectId == id);
Classes:
[Table("InvoicedHistory")]
public class InvoiceHistory
{
[Key]
[Column("InvoicedHistory_ID")]
public int InvoicedHistoryId { get; set; }
// few properties omitted
[Column("Project_ID")]
public int? ProjectId { get; set; }
}
public class Project
{
public int ProjectId { get; set; }
// few properties were omitted
[ForeignKey(nameof(InvoiceHistory.ProjectId))]
public virtual InvoiceHistory Invoice { get; set; }
}
Project class also use fluent api:
modelBuilder.Entity<Project>(entity =>
{
entity.ToTable("Projects");
entity.HasKey(e => e.ProjectId)
.HasName("PK_Project_Project_ID_Project");
// few statements were omitted
});
Sql which was generated: (Was hard to clean up this query. It contains several joins to include data for properties I have omitted)
SELECT [t].[Project_ID], [t].[Project_Client], [t].[Project_IrsDate], [t].[Project_Name], [t].[Client_ID], [t].[Client_Name], [t].[InvoicedHistory_ID], [t].[DateSubmitted], [t].[Project_ID0], [t0].[Debitor_ID], [t0].[Project_ID], [t0].[Debitor_ID0], [t0].[Address_Number], [t0].[Alias], [t0].[Alpha_Name], [t0].[Co], [t0].[Country_ID], [t0].[Currency_ID], [t0].[Havi_YesOrNo]
FROM (
SELECT TOP(1) [p].[Project_ID], [p].[Project_Client], [p].[Project_IrsDate], [p].[Project_Name], [c].[Client_ID], [c].[Client_Name], [i].[InvoicedHistory_ID], [i].[DateSubmitted], [i].[Project_ID] AS [Project_ID0]
FROM [Projects] AS [p]
INNER JOIN [Clients] AS [c] ON [p].[Project_Client] = [c].[Client_ID]
INNER **<<<<<<<<(expect LEFT)** JOIN [InvoicedHistory] AS [i] ON [p].[Project_ID] = [i].[InvoicedHistory_ID]
WHERE [p].[Project_ID] = 19922
) AS [t]
LEFT JOIN (
SELECT [p0].[Debitor_ID], [p0].[Project_ID], [d].[Debitor_ID] AS [Debitor_ID0], [d].[Address_Number], [d].[Alias], [d].[Alpha_Name], [d].[Co], [d].[Country_ID], [d].[Currency_ID], [d].[Havi_YesOrNo]
FROM [ProjectDebitors] AS [p0]
INNER JOIN [Debitors] AS [d] ON [p0].[Debitor_ID] = [d].[Debitor_ID]
) AS [t0] ON [t].[Project_ID] = [t0].[Project_ID]
ORDER BY [t].[Project_ID], [t].[Client_ID], [t].[InvoicedHistory_ID], [t0].[Debitor_ID], [t0].[Project_ID], [t0].[Debitor_ID0]
Look at this line -
INNER <<<<<<<<(expect LEFT)<<<<<< JOIN [InvoicedHistory] AS [i] ON [p].[Project_ID] = [i].[InvoicedHistory_ID]
Inner join makes my query return nothing, because I have no invoice info. If I manually replace it with Left join, sql query will return me all necessary data.
I think you can use Fluent API to get your desired result:
modelBuilder.Entity<Project>()
.HasOne(p => p.Invoice)
.WithOne()
.HasForeignKey(ih => ih.ProjectId);
This should change it to a left join because we didn't specify .IsRequired()
As mentioned in the following SO Answer - Equivalent for .HasOptional in Entity Framework Core 1 (EF7)
You will not find an equivalent method in EF 7. By convention, a property whose CLR type can contain null will be configured as optional. So what decide if the relationship is optional or not is if the FK property is nullable or not respectively.
and
In case of your FK property is value type like int, you should declare it as nullable (int?).
Now most likely your problem with annotations is that the following is not doing what you think it is:
[ForeignKey(nameof(InvoiceHistory.ProjectId))]
//Does not resolve to:
[ForeignKey("InvoiceHistory.ProjectId")]
//Does resolve to:
[ForeignKey("ProjectId")]
Now even if that is what you are looking for, the order of operations for the ForeignKey detection is to check the parent type then the property type.
public class InvoiceHistory
{
public int? ProjectId { get; set; }
}
public class Project
{
public int ProjectId { get; set; }
// this is pointing to Project.ProjectId
// and Project.ProjectId is not nullable
// so the join becomes an inner join
// and really only works because they both have the same name
[ForeignKey(nameof(InvoiceHistory.ProjectId))]
public virtual InvoiceHistory Invoice { get; set; }
}
If you wanted this to work as pointing to the Property Type, you need to rename the InvoiceHistory name:
public class InvoiceHistory
{
public int? ProjectFk { get; set; }
}
public class Project
{
public int ProjectId { get; set; }
// this is pointing to InvoiceHistory.ProjectFk
// because there is no Project.ProjectFk
[ForeignKey(nameof(InvoiceHistory.ProjectFk))]
public virtual InvoiceHistory Invoice { get; set; }
}
EntityFramework Data Annotations
If you wanted to see it create bad SQL you could do this:
public class InvoiceHistory
{
public int? ProjectId { get; set; }
}
public class Project
{
public int ProjectFk { get; set; }
[ForeignKey("ProjectFk")]
public virtual InvoiceHistory Invoice { get; set; }
}
EF will then create:
INNER JOIN [InvoicedHistory] AS [i] ON [p].[Project_ID] = [i].[ProjectFk]
And will cause a SqlException with the message something like Invalid column name.
I've seen an answer to a similar problem (Entity framework and Exists clause), but not using lambda and I would like to understand what is wrong,
whether it is my mapping or if is the way to use. I'm using Entity Framework 5.
Here is my mapping:
public class Attribute : EntityWithGuid
{
... Other mappings ....
[InverseProperty("Attribute")]
public virtual ICollection<CategoryAttribute> CategoryAttributes { get; set; }
}
public class Category : EntityWithGuid
{
... Other mappings ....
[InverseProperty("Category")]
public virtual ICollection<CategoryAttribute> CategoryAttributes { get; set; }
}
public class CategoryAttribute : EntityWithGuid
{
... Other mappings ....
[ForeignKey("Category_Id")]
public virtual Category Category { get; set; }
public string Category_Id { get; set; }
[ForeignKey("Attribute_Id")]
public virtual Attribute Attribute { get; set; }
public string Attribute_Id { get; set; }
}
If I run the command below, it works fine and the result SQL is
var query = Attribute.Where(x => !x.CategoryAttributes.Any());
SELECT
Extent1.Id,
Extent1.Name,
Extent1.Type,
Extent1.Active
FROM
Attribute AS Extent1
WHERE
NOT EXISTS( SELECT
1 AS C1
FROM
CategoryAttribute AS Extent2
WHERE
Extent1.Id = Extent2.Attribute_Id)
But if I put one more clause appears a Project2 alias for Attribute instead Extent1 and gives error because inside the exists clause within the remains Extent1.Id
var query = Attribute.Where(x => !x.CategoryAttributes.Any(y=>y.Category_Id == idcategory));
SELECT
Project2.Id,
Project2.Name,
Project2.Type,
Project2.Active
FROM
Attribute AS Project2
WHERE
NOT EXISTS( SELECT
1 AS C1
FROM
CategoryAttribute AS Extent2
WHERE
(Extent1.Id = Extent2.Attribute_Id)
AND (Extent2.Category_Id = #p__linq__0))
The exception is
Unknown column 'Extent1.Id' in 'where clause'
Found the problem.
After performing a simple query and receive an error similar thought something was wrong. Below the query, SQL and the undesirable alias Project
User.Where(x => x.Name.Contains(partialName))
SELECT
Project1.Id,
Project1.Name
FROM
User AS Project1
WHERE
(LOCATE('Marcelo', Extent1.Name)) > 0
ORDER BY Project1.Name ASC
Recently I upgraded MySql and with it came the new Net connector, version 6.7.4. I went back to the previous Net connector version (6.5.6) and everything is working again.
I am using Entity Framework to map two tables together using Entity Splitting as outlined here and here.
I have found that if I execute a .ToList() on an IQueryable<SplitEntity> then the results are from an Inner Join. However, If I take that same IQueryable and execute a .Count() it will return the number of records returned by a Full Join.
Here is a unit test that fails:
[TestMethod]
public void GetCustomerListTest()
{
// arrange
List<Customer> results;
int count;
// act
using (var context = new DataContext())
{
results = context.Customers.ToList();
count = context.Customers.Count();
}
// assert
Assert.IsNotNull(results); // succeeds
Assert.IsTrue(results.Count > 0); // succeeds. Has correct records from inner join
Assert.AreEqual(count, results.Count); // This line fails. Has incorrect count from full join.
}
This strikes me as very bad. How can I get the .Count() method to return the results from an Inner Join like the .ToList()?
Update - SQL
I was wrong about the full vs inner joins.
The .ToList() results in:
SELECT
[Extent1].[CustomerNumber] AS [CustomerNumber],
-- ...etc...
[Extent2].[CustomerName] AS [CustomerName],
-- ... etc...
FROM [dbo].[CustomerTable1] AS [Extent1]
INNER JOIN [dbo].[CustomerTable2] AS [Extent2] ON [Extent1].[CustomerNumber] = [Extent2].[CustomerNumber]
The .Count() results in:
SELECT
[GroupBy1].[A1] AS [C1]
FROM ( SELECT
COUNT(1) AS [A1]
FROM [dbo].[customerTable2] AS [Extent1]
) AS [GroupBy1]
Update - DataContext and entity code
The DataContext:
public class DataContext : DbContext
{
public DataContext() { Database.SetInitializer<DataContext>(null); }
public DbSet<Customer> Customers { get; set; }
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
base.OnModelCreating(modelBuilder);
modelBuilder.Configurations.Add(new CustomerMapping());
}
}
}
The Customer Mapping (FluentAPI):
public class CustomerMapping : EntityTypeConfiguration<Customer>
{
public CustomerMapping()
{
this.Map( m => {
m.Properties( x => new { x.CustomerNumber, /*...etc...*/});
m.ToTable("CustomerTable1");
})
.Map( m => {
m.Properties( x => new { x.CustomerName, /*...etc...*/});
m.ToTable("CustomerTable2");
});
}
}
The Customer entity:
public class Customer
{
[Key]
public string CustomerNumber { get; set; }
public string CustomerName { get; set; }
}
If the database and all records in CustomerTable1 and CustomerTable2 have been created by Entity Framework and SaveChanges calls in your application code this difference must not happen and you can go straight ahead and report this as a bug.
If you are mapping to an existing database or if other applications write records into the tables and you actually expect that not every record in CustomerTable1 has a corresponding record in CustomerTable2 and vice versa then Entity Splitting is the wrong mapping of your database schema.
Apparently the difference means that you can have Customers with a CustomerNumber (etc.), but without a CustomerName (etc.) - or the other way around. The better way to model this would be a one-to-one relationship where one side is required and the other side is optional. You will need an additional entity and a navigation property for this, for example like so:
[Table("CustomerTable1")]
public class Customer
{
[Key]
public string CustomerNumber { get; set; }
// + other properties belonging to CustomerTable1
public AdditionalCustomerData AdditionalCustomerData { get; set; }
}
[Table("CustomerTable2")]
public class AdditionalCustomerData
{
[Key]
public string CustomerNumber { get; set; }
public string CustomerName { get; set; }
// + other properties belonging to CustomerTable2
}
With this Fluent API mapping:
public class CustomerMapping : EntityTypeConfiguration<Customer>
{
public CustomerMapping()
{
this.HasOptional(c => c.AdditionalCustomerData)
.WithRequired()
.WillCascadeOnDelete(true);
}
}
I am querying a local table and I get the same count for both. I believe there is a problem with your context and that's why your results are inconsistent.
screenshot of essentially the same code just querying a simple dataset.
UPDATE:
I don't know why the SQL that is generated is different. You would think that they would be the same except for simply executing Count(*) instead of returning all the rows. That is obviously why you are getting a different counts. I just can't say why the SQL is different.
Maybe Jon Skeet or other genius will see this and answer! :)
I have a project where I'm using EF5, I made a custom Guid Generator and I have an override of the SaveChanges method to assign the ids of my entities.
Everything is working fine except in one case: when the ID of one entity is a FK to another ID of another entity.
A little bit of code to explain the problem:
I have two entities I cannot change:
public class FixedEntityA
{
public Guid Id { get; set;}
public string SomeText { get; set; }
}
public class FixedEntityB
{
public Guid Id { get; set;}
public int OneInt { get; set; }
}
In my project I have an entity defined like this:
public class ComposedEntity
{
public Guid Id { get; set;}
public FixedEntityA FixedA { get; set; }
public FixedEntityB FixedB { get; set; }
public double OneDouble { get; set; }
}
The relationships are:
ComposedEntity may have 0 or 1 FixedEntityA
ComposedEntity may have 0 or 1 FixedEntityB
The constraints on the id are:
The Id of FixedEntityA is a FK pointing to the Id of ComposedEntity
The Id of FixedEntityB is a FK pointing to the Id of ComposedEntity
The mapping class are:
public ComposedEntity(): EntityTypeConfiguration<ComposedEntity>
{
HasOptional(fea => fea.FixedA).WithRequired();
HasOptional(feb => feb.FixedB).WithRequired();
}
Here is my SaveChanges override:
foreach (var entry in ChangeTracker.Entries<IEntity>().Where(e => e.State == EntityState.Added))
{
Type t = entry.Entity.GetType();
List<DatabaseGeneratedAttribute> info = t.GetProperty("Id")
.GetCustomAttributes(typeof (DatabaseGeneratedAttribute), true)
.Cast<DatabaseGeneratedAttribute>().ToList();
if (!info.Any() || info.Single().DatabaseGeneratedOption != DatabaseGeneratedOption.Identity)
{
if (entry.Entity.Id == Guid.Empty)
entry.Entity.Id = (Guid) _idGenerator.Generate();
}
}
return base.SaveChanges();
This code works fine everywhere for all kind of relationships except in this case, I am missing a test to make sure I'am not setting an id on id that are foreign keys, and I have no clue on how to check if an Id is a FK...
Here is a sample object where this code fails:
var fea = new FixedEntityA();
var feb = new FixedEntityB();
var composedEntity = new ComposedEntity();
composedEntity.FixedA = fea;
composedEntity.FixedB = feb;
If you insert the whole graph, all three objects are marked as Added and all Ids are default.
The problem is, with the current SaveChanges method, I will go through all object with the Added state in the change tracker and I will assign an Id to all entity with a default Guid and break my FK constraints.
Thanks in advance guys!
Here is some code that will get the FK properties for a given type (it's horrible I know). Should be simple enough to plug this into your code.
var typeName = "Category";
var fkProperties = ((IObjectContextAdapter)db)
.ObjectContext
.MetadataWorkspace
.GetItems<AssociationType>(DataSpace.CSpace)
.Where(a => a.IsForeignKey)
.Select(a => a.ReferentialConstraints.Single())
.Where(c => c.FromRole.GetEntityType().Name == typeName)
.SelectMany(c => c.FromProperties)
.Select(p => p.Name);